I need to implement the chaining methods and using functions into what I do, much easier to use and read. Great video as always.

@robmulla

Жыл бұрын

Totally. Just those two things alone are huge! Glad you enjoyed the video.

Usually these videos address REALLY nooby mistakes that any general programmer already avoids. THIS video however ACTUALLY addresses library FUNCTIONALITY and discusses the tools that a programmer may be unaware of to increase readability and efficiency. Rob, my good sir, you just earned a sub.

@robmulla

Жыл бұрын

So happy to have you as a sub. Even happier to read such a kind comment. Looking forward to more videos like this in the future.

@murphygreen8484

Жыл бұрын

I couldn't have said that better myself. I am self taught and definitely learned new tricks here. You have also earned my sub!

@javiercmh

Жыл бұрын

Same! I had no idea many of these even existed!!

@xinaesthetic

Жыл бұрын

@@robmulla I am a fairly experienced programmer, not so much with Python, but I have a few things I might want to use Pandas for at some point and this has given me a bit of a taste for features that I look forward to trying.

Please keep doing this. No additional jargon, crisp, straight to the point explanations are what are required. No body needs a 10 hour tutorial. Thank you for this.

@robmulla

Жыл бұрын

I'll try my best! I do like trying to cram a ton of information into a short format, but these videos take a while to create. I totally copied the format from mmcoding (check out the channel if you haven't already)

00:18 #1. Writing into csv with unnecessary index 00:53 #2. Using column names which include spaces 01:25 #3. Filter dataset like a PRO with QUERY method 01:44 #4. query strings with(@ symbol) to easily reach variables 02:07 #5. "inplace" method could be removed in future versions, better explicitly overwrite modifications 02:35 #6. better Vectorization instead of iteration 03:01 #7. Vectorization method are preferable than Apply method 03:30 #8. df.copy() method 04:08 #9. chaining formulas is better than creating many intermediate dataframes 04:28 #10. properly set column dtypes 05:01 #11. using Boolean instead of Strings 05:25 #12. pandas plot method instead of matplotlib import 05:45 #13. pandas str.upper() instead apply and etc 06:10 #14. use data pipeline once instead of repeating many times 06:41 #15. learn proper way of renaming columns 06:59 #16. learn proper way of grouping values 07:31 #17. proper way of complex grouping values 08:01 #18. percent_change or difference now could be implemend with function 08:25 #19. save time and space with large datasets with pickle,parquet,feather formats 08:58 #20. conditional format in pandas(like in Microsoft Excel) 09:22 #21. use suffixes while merging TWO dataframes 09:48 #22. check merging is success with validation 10:13 #23. wrapping expression so they are readable 10:33 #24. categorical datatypes use less space 10:55 #25. duplicating columns after concatenating, code snippet

@robmulla

Жыл бұрын

Thanks for making this!

@akmalmir8531

Жыл бұрын

@@robmulla i wish i commented better as English is not my native language, Thank You for bringing us Valuable Tutorials that saves us our time and energy! I wish i helped and learned from you more

@kongson14

Жыл бұрын

egg bro

@PGhai

Жыл бұрын

thanks, I like no 4

@bonumonu5534

Жыл бұрын

This needs to be pinned

Matt Harrison's "Effective Pandas: Patterns for Data Manipulation" is one of the best resources I've read on idiomatic pandas.

@robmulla

Жыл бұрын

I really need to get myself a copy! He knows his stuff for sure.

@MrEo89

Жыл бұрын

He has a great video (series?) on effective pandas also!

@lordvoldemort1985

Ай бұрын

ty i will look into this book

I have been working 2 years now with pandas and I can strongly affirm that I have made like 70% of those bad practices, appreciate a lot your video!

@robmulla

Жыл бұрын

Thanks for commenting. Honestly I still make many of them to this day.

I thought I was pretty good in Pandas, but you gave me so many new things to improve. HUGE thank you!

@robmulla

Жыл бұрын

Glad I could help! I'm constantly learning better ways to do things in pandas myself.

@olyaagapova989

Жыл бұрын

I was thinking that I was pretty bad, but surprisingly I usually only make 2 mistakes from the video (which is a cool chance to improve). I just love such videos because not only they help to improve your skills, but also to be realistic about your expectations and ambitions. Thanks for the video, Rob!

The pandas query function does not outperform the loc method. In fact, it is sometimes much slower when your query/data is so big. We industry users will utilize the loc method for quick EDA. Query might be useful when you have a scheduled cron

@robmulla

Жыл бұрын

Yea. Query isn’t for speed of processing but speed of writing the code.

Rob, amazing video and intuitive. Happy to subscribe!

I'm currently working on my first major pandas project and I reckon that I may have done around 15/25 of these 'mistakes'. Looks like I have some optimisation to do over the coming days!

@robmulla

Жыл бұрын

We all have to start somewhere. I didn't learn many of these until I had been using pandas for years.

oh wow the quality and clarity is worth subscribing! thank you !

Rob, thank you for all the time and energy you have put in for us. Would appreciate an updated video on "Exploratory Data Analysis" may be expanding on your year old one. Thank you again!

I started to watch your videos recently, and from now on I'm doing the chaining and putting each function in "one row" to make the data cleaner, and also, the query method, so powerful and simple, I was used to replicate the dataframe with the column and value searched to filter my df. You are boosting my studies! Thanks for that!

This video is amazing, I am using pandas for a long time now and still learned so many new good practices thank you

I can't believe how good this video is. I love your no-nonsense delivery; I don't have time at work to watch a 4-hour "intro" video. Keep it up!

This was great! Just what I needed :)

I can't believe I watched this whole video and only 2 of them were things I didn't know about! Thank you for sharing!

Great video. Thank you for being so direct and giving us valuable tips ☺

@robmulla

Жыл бұрын

Glad you liked it! Thanks for giving feedback. Share the video with anyone else you think might also like it.

Rob, as always, fantastic video. I have to admit, i get caught on some of those mistakes so it is great to have you point out and make suggestions on how to correct them. Thanks for sharing. Much appreciated.

@robmulla

Жыл бұрын

I fall into these a lot too! We can all get better, glad you found the video helpful.

Another awesome, useful video, Rob. Thank you.

@robmulla

Жыл бұрын

Thanks for watching Deepak!

Very useful! Thank you for sharing in such an easy and agile way.

@robmulla

Жыл бұрын

Hey! Glad you learned something. Appreciate the feedback!

Love it. Thank you!

Great video. Very helpful. Please keep making more like this

@robmulla

Жыл бұрын

Appreciate that. I plan to!

Thanks, great tips! I've been using pandas for years, and I've only recently started using some of these (particularly query, and didn't know about the @ operator)

@robmulla

Жыл бұрын

Glad it was helpful! The @ operator is really useful. You can also do stuff like min() or or apply operations between columns within the query.

Great insights, thanks for these important tips

@robmulla

Жыл бұрын

Glad you found them helpful. Share it somewhere on social you think people might learn from!

This is awesome, I’ve been wanting to know what are the better ways to write my code and why. Please continue to make these videos.

@robmulla

Жыл бұрын

Wow! Thanks so much Emily. Really apprecaite the feedback and super thanks!

Excellent points! Learned new stuff that a lot of tutorials don't explicitly teach.

@robmulla

Жыл бұрын

Glad it was helpful! Thanks for watching and please share with others.

lots of good info! thank you!

@robmulla

Жыл бұрын

Glad you learned from it!

Thanks for this great tutorial!!

Very illuminating video! I learned a lot quickly.

@robmulla

Жыл бұрын

Thanks for the feedback Daniel!

This can be, some of my first times commenting in youtube after years of usage. This video was INCREDIBLY USEFUL! There's a lot of my previous team members did on scripts and sometimes are complicated to maintain or create new ones following the same logic. This covers exactly what they used and what is the best option to rewrite it and make it more understandable. Thank you so much for this godly information.

@robmulla

Жыл бұрын

You're very welcome! I really appreciate the positive feedback. I’ll try to keep making helpful videos like this. Share with your friends in the meantime!

Very useful video, thank you for making this !

@robmulla

Жыл бұрын

Glad it was helpful! Share it with anyone you think might also benefit.

Wow dude! You are single handedly responsible for my data science growth. PLEASE keep making more of these videos I really appreciate it.

@robmulla

Жыл бұрын

Wow! I love hearing feedback like this. I'll keep making videos if you all keep watching! :D

I feel personally attacked. Thanks so much for releasing this. I knew my code was bad, but not THIS bad.

@robmulla

Жыл бұрын

Haha. With coding we all are learning and getting better every day. Me included. Thanks for watching!

@alberttu8120

Жыл бұрын

These are fantastic refactoring suggestions.

Learned tons with this. Short and succinct. New subscriber.

@robmulla

Жыл бұрын

Thanks for subscribing!

Really enjoyed how fast this content came. I felt like it was a great speed to keep me engaged. I usually find these types of videos boring.

One of the best videos I've seen on Pandas! So glad someone prominent enough is advocating for method chaining and pandas methods!

@iReaperYo

11 ай бұрын

The 'Query' method in particular is relatively unknown. In conjunction with not using 'snake case' this leads to beginners being very inefficient at code due to not being able to use dot syntax I am just an intermediate level so I can relate to many of these mistakes. It goes as deep as university however. They do not teach clean, efficient code at all!

@robmulla

11 ай бұрын

Glad you enjoyed it! I confess I don't use chaining nearly as much as I should.

Super useful! Thanks a lot, mate!

@robmulla

Жыл бұрын

Thanks for watching. Please share with someone you think might also like it.

I didn't know about suffixes. Amazing!

@robmulla

Жыл бұрын

Thanks Ken, glad I you were able to learn something new! Love your videos.

Learned more about Pandas in this video than a whole many videos worth hours combined. Seriously, thank you.

Hey, Rob! Super video this one. I myself am Sr. DS working each day intensively with pandas, I will implement many of the tips you show! Thanks a million :)

@robmulla

Жыл бұрын

Awesome to hear! I'm still learning new tricks with pandas every day.

I'm an experienced developer looking to get familiar with Pandas. I found this video very valuable.

This is amazing! Thanks a lot.

@robmulla

Жыл бұрын

Glad you like it!

Thank you for creating such an amazing video on pandas. It has even been really helpful for me as a pandas new bee. Leanrt a lot! 🎉

@robmulla

8 ай бұрын

Love it!

1:28 Before I discovered your videos, I'd never considered using the query method. The examples I've previously seen online made it look like a me-too add-on for seasoned SQL users. Using conditionals to mask off rows seemed just as easy and more pythonic. Also, at work, I typically filter with a script when I pull down the data, so by the time I get the data into pandas, I just need to tweak. But, you've shown me the light. Thanks!

@robmulla

Жыл бұрын

I totally understand where you are coming from. Its important to keep in mind query can be slower, but for quick filtering it can be really quick and clean way to filter data. It really depends on what I'm doing. Glad I showed you something new though!

Oh man, that guide is pro! Thanks, gonna apply all of that when refactoring my project!

@robmulla

Жыл бұрын

Glad it helped! Tell a friend!

Awesome stuff. I've been using pandas for over 4 years, but it never occurred me to start using the query method instead of loc (despite me finding it tiresome to keep repeating "df" all over the place when using loc). I also appreciate the quick format. You see KZreadrs taking too long to say nothing at all, so congrats on actually going through 25 tips in 10 minutes. You got yourself a sub!

@janusztarnowski115

Жыл бұрын

Simple Application Enter your details kzread.info/dash/bejne/douil49-mpu4XbA.html

Amazing Tips, Many thanks.

I used the Pandas lib more then 2 years, but today I learned something new! Thank you, man!

@robmulla

Жыл бұрын

Glad you learned something new! Share with anyone else you think might appreciate it!

The space need to be avoid part is so true! But wait a second, every time I face the space but not underscore is from others data, so I think what we actually need is how to deal with the space condition.(Which is a pain of journey)

@Saareem

Жыл бұрын

Maybe rename all the columns with versions without a space. Like, you replace all the spaces with an underscore. df.rename can take dictionaries or even a mapper function so this is easy to do. Using a dictionary is preferable as you can just reverse map it, if you want to use the columns with spaces in them in the end.

@robmulla

Жыл бұрын

Good point. In most cases to can be done with a list comprehension one liner!

This video made me realize i have still a long road ahead in Pandas. Thanks! Just subscribed ;D

@robmulla

Жыл бұрын

Thanks for the sub! We all start somewhere, but you'll pick it up quickly in no time.

This is really useful, thank you!

@robmulla

Жыл бұрын

Glad you found it useful, Juan!

I'm new to Pandas and all tips from this video are gold for me, thank you a lot!

@robmulla

Жыл бұрын

Glad you learned something new. Welcome to the world of pandas!

@garyfritz4709

Жыл бұрын

+1000. I’m brand new to Pandas and still trying to grok the idiom. This video is GOLD.

This video rocked me. I've been using python for a few months and watching this video made me bust out my laptop so I could try all of these items out. Thank you for this.

@robmulla

Жыл бұрын

So glad you found it helpful. Share with a friend!

I loved this to s be to my students. You did a great job in a short video!

@robmulla

Жыл бұрын

Thank you so much! It's hard to make it short but is worth it in the end.

Really helpful tips, thanks for video

@robmulla

Жыл бұрын

Thanks for watching!

Great video, thank you!!

@robmulla

Жыл бұрын

Glad you liked it! Share witha friend!

found your channels few days ago and man you have some epic content . The noob mistakes here are the exact way most tutorials teach you..just wondering why the hell the non noob ways are not taught as they are easier and shorter and the syntax makes more sense... thank you for this video

@robmulla

Жыл бұрын

Glad you like them! I’m trying to continue to make more stuff like this so keep watching!

OMG! I had to rest after first 10. So huge dose of information. Thanks.

Awesome video! I work with Pandas for +3 years and learned a lot here! Thanks

@robmulla

Жыл бұрын

Happy to hear it. Tell your friends!

Great video. Lots of operations and procedures that are helpful for effective coding. Would be really helpful to have a cheat sheet linked for easy reference.

Great video as always. I will start exploring query method more. Rob, Can you please make a video on how feature engineering, especially how to create new features using aggregation etc. Thank you

@robmulla

Жыл бұрын

Glad you enjoyed the video. Feature engineering would be a good topic for a future video. I'll add it to the list!

Dude, Amazing video apparently clear the concept.

@robmulla

Жыл бұрын

Glad you think so! Share with your friends!

Thank you. This video was helpful.

9:13 another pet peeve, though this one is more important than the last one. Do not use backslashes. ever. well, not _never_, use them when writing a `with` statement with more than two context managers. But otherwise, don't. I'll quote the `Black` (the formatter) documentation: Backslashes and multiline strings are one of the two places in the Python grammar that break significant indentation. You never need backslashes, they are used to force the grammar to accept breaks that would otherwise be parse errors. That makes them confusing to look at and brittle to modify. This is why Black always gets rid of them.

@robmulla

Жыл бұрын

Good point. Then backlashes are and old habit I’ve been trying to stop use. We all are learning constantly!

some great tips here. i usually chain with \ and i didn't know a query method exists!! guess you learn everything new all the time!

@robmulla

Жыл бұрын

Glad you learned something new! Cheers.

I really think this should be written up in a medium blog article. Would be awesome to refer to.

@robmulla

Жыл бұрын

That’s a good idea. I really want to make blogs for all my videos but I don’t have the time. Maybe someday

Thanks, lots of good info.

@robmulla

Жыл бұрын

Glad it was helpful!

Dear Rob, I'm a total beginner in Python and Pandas. From what I understand, the warning at 3:30 is not about making a copy of sliced data, but rather about not using the .loc method and using "direct assignment" for columns (or whatever it's called). I could be wrong, but this is what I've gathered from reading the documentation and encountering a similar warning in my code. Thanks for your valuable content. It has been a great help

Great video! I also like the jazz bass behind you, I also play bass :)

@robmulla

Жыл бұрын

Awesome! I’m more of a guitar player but I also enjoy playing bass.

Found lots of favorite annoyances and learned a few new tricks! I'll add a shout-out to the ".pipe()" method to allow for wrapping all your transforms in a single statement when a single .method can't cover the required transform. An added bonus of "pipe()" - since it's using user defined functions to do the transforms, you can add decorators to automatically print out metadata on the resulting transform steps to get a quick insight into potential bugs.

@robmulla

Жыл бұрын

Oh. Great one. I forgot to add pipe and assign in this video but wish I did.

Hi, I love your videos!!! Can you please make a video on how to handle missing values and outliers?

@robmulla

Жыл бұрын

Great suggestion! I did have a whole video on this topic on Abhishek Thakur's channel. Check it out here: kzread.info/dash/bejne/d42ttbCDhZvXgpM.html

Great tips!

Thank you! The .diff method is a lifesaver when computing velocities. The advice on not using inplace is excellent i got into various troubles because of it but i thought that's what the "experienced guys" do.

@robmulla

Жыл бұрын

Thanks for watching. inplace is very tricky. Diff method is really powerful, and there are parameters you can use within it depending on your use case.

Great video. Subscribed

@robmulla

Жыл бұрын

Thanks for the sub!

When you mention the slice warning sometimes you don't care about the original data frame so it doesn't matter if you modified it

@robmulla

Жыл бұрын

That’s true. But I don’t like seeing the warnings. And if you don’t need the rest of the data you can just overwrite it with the slice?

Wonderful, this video is super helpful

@robmulla

Жыл бұрын

Glad you think so!

@slapfighting

Жыл бұрын

@@robmulla videos like these actually helps a data scientist’s sword to be sharper. Thank you for sharing it, hope to look for more advanced videos like these in future.

Oh god. I clicked on this video just to confirm that this is one more overly exaggerated self-confident dude trying to teach newbies of 2 weeks experience. After watching this, this is god damn life changing. As an engineer focusing on fluid dynamics and floater response, I use pandas daily basis. Out of 25, I didn’t know approximately 20. Every single person who has any plan to use pandas must watch this. Awesome!

So good. Glad I subscribed.

@robmulla

Жыл бұрын

Glad you watched!

Thanks, really helpful

@robmulla

Жыл бұрын

Thanks!

Nice video! I have been using pandas for years and still run into these issues :)

@robmulla

Жыл бұрын

Thanks! Glad you enjoyed the video. I really enjoy your videos too.

Dude I've worked with pandas for 7 years and learned some new tricks, thanks a lot!

@robmulla

Жыл бұрын

Great to hear! You've been working with it longer than I have. Please share my channel with any friends you think might also learn from it.

Wow, very useful - a true "tour de force" for better Pandas code. THX for this !

@robmulla

Жыл бұрын

Glad it was helpful! Please consider sharing it with anyone else you think would benefit from watching.

the last 5 were cool! thank you

@robmulla

Жыл бұрын

Glad you found them helpful.

Great video!!!

@robmulla

Жыл бұрын

Glad you liked it!

Regarding the 'inplace' comment at 02:07 there's a very valid and very useful reason to prefer that and it's memory usage. `df = df.reset_index()` or anything similar creates an entire copy of the dataset before replacing it with the original and for extremely big data that is a problem, it may get over the physical memory available and have the OS kill the script.

@robmulla

Жыл бұрын

Interesting. I’ve heard this but then also thought it was debunked. I think the fact that the pandas core developers want to remove inplace gives good reason to try and avoid using it.

@SamusUy

Жыл бұрын

@@robmulla I guess it's more "functional style" to do it like they want but I recently had this problem with the memory when creating copies and I solved it by using 'inplace' (Python 3.7 and Pandas 1.3.5 if it matters)

@robmulla

Жыл бұрын

@@SamusUy good to know!

Great vid thanks!

@robmulla

Жыл бұрын

Glad you liked it!

Great video for new users not knowing tips and tricks.. Wish you shared the code also to keep it handy for reference

@robmulla

Жыл бұрын

Thanks for watching. I don’t think I kept the code unfortunately

wow this was very helpful thanks!

This is excellent thanks

@robmulla

Жыл бұрын

You're very welcome

At 6:23 (#14) you're returning the dataframe, but you're also modifying it in place. Having a return there gives the impression that the original dataframe isn't modified, specially if you also assign it to itself later. It ties back to #5.

So good. Thank you.

@robmulla

Жыл бұрын

Thanks for watching!

Oi! There were several of those I didn't know. I wouldn't have thought I was a noob, but I guess we all have a bit of that in us. Thanks for the video!

@robmulla

Жыл бұрын

Glad you learned something new. I find I’m always learning something new with python and data science. That’s why I love it so much.

did you ever tried to use np.vectorize function to apply transformations over a df column? that one is along with my favorites. amazing video btw, subscribed!

@robmulla

Жыл бұрын

Yes! I've used it before and had some good results. Thanks for watching!

Thanks for the tips

@robmulla

Жыл бұрын

Thanks Levy for watching.

As a beginner this video made me learn some basic concept about pandas. thanks

Very helpful !

@robmulla

Жыл бұрын

Glad you found it helpful.

Merge validator! Excellent thanks!

@robmulla

Жыл бұрын

👍

Extremely underrated channel Extremely helpful

@robmulla

Жыл бұрын

Thanks Nikhhilil!

I'm so guilty of number 8! Thank you for this!

@robmulla

Жыл бұрын

I’ve made every one of these mistakes at some point so I know how you feel. Thanks for watching!