My top 25 pandas tricks
You're about to learn 25 tricks that will help you to work faster, write better pandas code, and impress your friends. These are the BEST tricks I've learned from 5 years of teaching Python's pandas library.
Don't miss the BONUS at the end of this video!
TRICKS:
0:00 Introduction
0:43 1. Show installed versions
1:20 2. Create an example DataFrame
2:22 3. Rename columns
3:47 4. Reverse row order
4:36 5. Reverse column order
5:01 6. Select columns by data type
5:40 7. Convert strings to numbers
6:55 8. Reduce DataFrame size
8:15 9. Build a DataFrame from multiple files (row-wise)
10:00 10. Build a DataFrame from multiple files (column-wise)
10:45 11. Create a DataFrame from the clipboard
11:50 12. Split a DataFrame into two random subsets
12:57 13. Filter a DataFrame by multiple categories
13:52 14. Filter a DataFrame by largest categories
14:42 15. Handle missing values
15:57 16. Split a string into multiple columns
16:59 17. Expand a Series of lists into a DataFrame
17:39 18. Aggregate by multiple functions
18:41 19. Combine the output of an aggregation with a DataFrame
19:56 20. Select a slice of rows and columns
20:52 21. Reshape a MultiIndexed Series
22:04 22. Create a pivot table
23:01 23. Convert continuous data into categorical data
23:56 24. Change display options
24:47 25. Style a DataFrame
26:14 Bonus. Profile a DataFrame
DOWNLOAD the Jupyter notebook:
github.com/justmarkham/pandas...
WATCH my introductory series, Data Analysis with pandas:
• Data analysis in Pytho...
JOIN the "Data School Insiders" community:
/ dataschool
LET'S CONNECT!
- Email Newsletter: www.dataschool.io/subscribe/
- LinkedIn: / justmarkham
- Twitter: / justmarkham
- Facebook: / datascienceschool
- KZread: kzread.info?su...
Пікірлер: 658
THANKS for watching! 🙌 Which trick are you most excited to start using? CLICK REPLY and let me know!
@TarmoSillajoeGoogle
5 жыл бұрын
Thanks. Trick nr 9- building a dataframe from multiple files is the one i've been searching for. Comment to trick nr. 3. Sometimes modifying column names works by using Python's list comprehension. Say we want to capitalize all column names. Just write: df.columns = [my_col.capitalize() for my_col in df.columns]
@dataschool
5 жыл бұрын
Glad you like trick 9, that is one of my favorites! As for capitalizing column names, a list comprehension works, but a string method is even simpler: df.columns = df.columns.str.capitalize()
@juanbetancourt5106
5 жыл бұрын
Thanks, those tricks are really amazing, I wonder if I can take this magic to the web, is it a good choice to combine pandas power with django?
@mritunjaykumar970
5 жыл бұрын
The last trick will significantly reduce the amount of work we do understand the data through eda.that trick was amazing.thanks kevin
@sreenivas7739
5 жыл бұрын
Thanks for a great video. Bonus trick is the most useful.
Video should be called 'Magic with Pandas'. Thanks for this really useful resource!
@dataschool
4 жыл бұрын
Thank you! 🙏
The long awited video is out now. Thank you so much mate. Great learning with you.
@dataschool
5 жыл бұрын
You're very welcome!
Kevin, love your content and it is amplified by your speech cadence, enunciation, and tone. My favorite tricks are 10 and 11 (Building a dataframe from multiple files rows/columns). Thank you for sharing your knowledge, mad respect!
@dataschool
4 жыл бұрын
Thank you so much! 🙏
One of my favorite videos on KZread, thank you so much!
@dataschool
4 жыл бұрын
Wow! Thank you so much for your kind words! :)
The bonus content totally blew my mind off! Thank you very much Kevin! You're simply the best! Looking forward to purchase your ML series!
@dataschool
2 жыл бұрын
Thank you so much for your kind words!
Man, you are just so awesome! Giving all these valuable tricks for free on your channel. God bless you!!!
@dataschool
5 жыл бұрын
Thank you so much! Glad you like them 👍
Fantastic! Been using pandas since a year now, You have opened new possibilities for me to keep looking and learning.!! Please keep doing the great work!
@dataschool
4 жыл бұрын
Thank you!
Well done Kevin! Thanks for sharing this great material. My best regards!
@dataschool
5 жыл бұрын
You're very welcome, Hector!
Brilliant as usual! From beginning to end! The bonus was also mind blowing :)
@dataschool
5 жыл бұрын
Thank you so much, Roberto!
Kevin Markham, you by far one of the best Data Science Instructor, I came across. Thank you for providing us with such high standard content at no cost.
@dataschool
4 жыл бұрын
Thank you so much! 🙏
Wow, that last tip! What a phenomenal overview and diagnostic tool. I've learned so much from your videos. So grateful for your content.
@dataschool
3 жыл бұрын
Thanks very much for your kind words!
you just saved my life, i was collecting these things over google wasted my good time but now i know where to look for. keep sharing , God bless you. Thanks
@dataschool
3 жыл бұрын
You are welcome! 😄
Excited to try ALL of these! Thanks!
@dataschool
2 жыл бұрын
You're welcome!
Superb ! Excellent ! introduction for Beginner (myself) to get introduced and master these Pandas at the same time thereby saving me a lot of time. Thank you so much Kevin !!! You Rock !!!
@dataschool
5 жыл бұрын
That's great to hear! Thanks for watching 👍
Fantastic tips, thank you!! Super valuable and with clear, practical examples. Once again, thank you!
@dataschool
4 жыл бұрын
Thank you so much! 🙏
It is one of the best videos on Pandas. I really love some of the tricks e.g., the Clipboard', and 'The Excel copy & paste ones.
@dataschool
Жыл бұрын
Glad it was helpful! Thanks for your kind words!
Best Pandas trips ever. I’m going to make myself some flash cards for each of these. Thank you.
@dataschool
4 жыл бұрын
You're very welcome! Glad you like them!
Thank you, for giving me shout out, and thank you for including my trick in your video (pd.profiling) :)
Thank you Kevin, You are really a good instructor. Also, I really loved how you shout out people your learnt tricks from, that's very awesome!
@dataschool
3 жыл бұрын
Thank you!
Kevin, you are a fantastic instructor. I'll sign up for anything you offer. My 2 favorites: clipboards, and, what made my mouth drop - the dataframe profiling.
@dataschool
3 жыл бұрын
Thank you so much for your kind words Valerie! I have a lot of resources that might interest you here: www.dataschool.io/start/ And here is my email newsletter: www.dataschool.io/subscribe/ Thanks again!
@giangtu6000
3 жыл бұрын
I love you! KEVIN!
First time ever I honestly wish I could upvote this more than once! Great tips, great explanation! One short addition to trick 19: I enjoyed getting to know the transform method, but as most of the times in pandas, there is another way with (for me at least) more often used functions: Just take the aggregated data-frame and merge it with the old one (left merge) instead of concatenating. I don't know if there are any runtime benefits of any of these methods, but to me it's one less command to remember - even though now I will look into uses of the transform method! :-)
@dataschool
5 жыл бұрын
Thank you so much! 🙏 As for trick 19, you are totally right that merge could accomplish the same thing. I like the transform code much better, but I also encourage people to learn fewer functions overall, so there's nothing wrong with just using merge! Anyway, thanks again for your kind words 😊
Awesome pandas tricks!!! Thanks for this great resource
@dataschool
4 ай бұрын
You're very welcome!
Kevin, you are simply amazing. Thanks for sharing these wonderful tricks!
@dataschool
3 жыл бұрын
Glad you like them!
This is a jewel video! All tricks left me without words! thanks
@dataschool
5 жыл бұрын
You're very welcome! 💎
This is one of the most useful coolest videos about Pandas I have ever watched. Thank you sooooo much for compiling all these tips and tricks! Absolutely amazing!!!!!!
@dataschool
Жыл бұрын
You're very welcome! 🙌
Great stuff. Thanks for the tips.
@dataschool
2 жыл бұрын
You're welcome!
Last Bonus tip, Best in 2020 till date. Thank you so much. You have made my life easy!!!
@dataschool
3 жыл бұрын
Thank you!
These are really great tricks.
Great video. I didn't know about the cut function and it will make a portion of my daily data processing tasks a lot easier. Thanks!
@dataschool
4 жыл бұрын
That's awesome to hear!
Yeah, 25 tricks!! I would like to thank you a lot for sharing these tips.
@dataschool
5 жыл бұрын
You're very welcome!
Wow... Veryyyy cool tricks. Saves a ton of time.
@dataschool
4 жыл бұрын
Thanks Sanjay! 🙌
All the tricks were mind blowing. But the bonus one was out of the world. It is surely going to help those who struggles Exploratory Data Analysis. Many thanks for sharing this.
@dataschool
5 жыл бұрын
Awesome! So glad it was helpful to you! 🙌
Super duper useful video. Amazing!!
@dataschool
3 жыл бұрын
Thank you!
Great video. The bonus tip is very useful.
@dataschool
5 жыл бұрын
Thanks!
Thank you for all these tricks.
@dataschool
2 жыл бұрын
You're welcome!
No other trick mentioned in the video is comparable to the Bonus Trick (Profiling).. This is simply outstanding !!!!
@dataschool
4 жыл бұрын
Thanks!
Fantastic content as always. Thank you Kevin!
@dataschool
5 жыл бұрын
You're very welcome Jerome! Glad you liked it 😊
I had some trouble with my dataframe. So I was stumbling over the internet to find solution,and your video came. Although I didn't find solution in your video, but one of the trick gave me a logical idea of how to resolve the issue. It told me where I was wrong. And as far as tricks are concerned, they are amazing. Some went over my head but that's okay. This whole panda python thing is new to me. I'd prefer to bite slow instead of gulping down entire meal in one go
@dataschool
2 жыл бұрын
Thanks for your kind words, and glad I could be helpful!
The last 5 tricks were fantastic.!!
@dataschool
2 жыл бұрын
Thank you!
This is incredibly useful! And explained perfectly! Thank you! It would be awsome to see from you some content on advanced plotting, data plotting tricks :)
@amandal8170
3 жыл бұрын
Agree, if we can see the plotting ticks will be fan jolly tastic. :p
@dataschool
2 жыл бұрын
You're very welcome, and thanks for your suggestions!
Thank you for the video, really useful stuff!
@dataschool
2 жыл бұрын
Glad it was helpful!
Really informative Kevin, appreciate your effort in creating these video's. Such a valuable resource. Trick 19 was probably my favourite.
@dataschool
4 жыл бұрын
Thanks for your kind words, and thanks for sharing! 🙌
It helps me a lot in Data Wrangling. I really love your videos. Keep doing!
@dataschool
4 жыл бұрын
Great to hear!
Best pandas tutorial !! Thanks
@dataschool
3 жыл бұрын
You're welcome!
I can’t have enough of this!!
@dataschool
7 ай бұрын
Thank you!
Awesome video! Thanks for reminding me of the transform function to combine aggregates back onto a df. Also, the df profiling is a great idea that I've never heard of.
@dataschool
4 жыл бұрын
Thank you!
Wow the last one was fantastic. styling the dataframe was also pretty neat. Thanks for this 👍
@dataschool
4 жыл бұрын
You're very welcome!
Your contents are to the point, your teaching style is exceptional. I am learning so much from your videos. Thank you!
@dataschool
Жыл бұрын
Wow, thank you so much!
Very helpful, Thanks !
@dataschool
4 жыл бұрын
You're welcome!
Useful is the word! Thank you
@dataschool
3 жыл бұрын
You're welcome!
thank you this is amazing. will be saving this for future reference!
@dataschool
5 жыл бұрын
Awesome! Glad it was helpful to you 👍
This was amazing video to sum up basic Pandas usage. Thanks for this useful resources!
@dataschool
3 жыл бұрын
Thank you for the kind comment!
Holy Moly What a master piece this is. I liked the clipboard trick a lot man like that literally saves a lot of code and time. And the bonus blew me off. It looks literally like a final report LOL. Can't wait to implement these tricks in my analytics work. Thanks a lot Data School
@dataschool
2 жыл бұрын
You're very welcome! Thanks for your kind words!
Best pandas video ever!
@dataschool
Жыл бұрын
Thank you! 🙌
Thanks for this consolidated and a very crisp video on Pandas....
@dataschool
4 жыл бұрын
You're welcome!
Fantastic! It really helped me escaping from a lot of itchy situations!
@dataschool
2 жыл бұрын
Great to hear!
Amazing! Thank you!
@dataschool
Жыл бұрын
You're welcome!
Needless to say. Thank you once again for the excellent video, like always very neat explanation and clean deployment of ideas.
@dataschool
5 жыл бұрын
Thank you so much! I'm glad the video was helpful to you!
Awesome!! TQVM!
@dataschool
3 жыл бұрын
Thanks!
Thank you so much. I learned a lot of good things. Thank you.
@dataschool
3 жыл бұрын
You are so welcome!
Amazing tips!!!......keep sharing more such videos
@dataschool
5 жыл бұрын
Glad you like them! 😄
Very cool! Thank you!
@dataschool
3 жыл бұрын
You're welcome!
Want even MORE pandas tricks? I share a new trick here every weekday 👉 www.dataschool.io/python-pandas-tips-and-tricks/ 👈
@Dara-lj8rk
4 жыл бұрын
Please make a video on time series analysis in Pandas
@dataschool
4 жыл бұрын
Thanks for your suggestion!
Great Pandas resource! Difficult to say which trick most 'excites' me, but will go with using transform to add aggregation functions to the data frame. Thanks!!!
@dataschool
5 жыл бұрын
I like that trick as well... I only learned about it recently!
The bonus is cherry on the cake!
@dataschool
4 жыл бұрын
🍒
You are amazing!!!
it is amazing because every time I watch this video i can learn sth new! thank u!
@dataschool
3 жыл бұрын
That's awesome to hear!
This is a nice vid! When I was learning clojure someone had compiled a '100 most used functions' and boy would I love that for pandas functions or methods: the docs are too extensive but tutorials are typically too brief.
@dataschool
Жыл бұрын
That's an excellent idea!
The bonus trick is so amazing
@dataschool
3 жыл бұрын
Thank you!
Pandas_profiling is BEST Can't wait to use it! Thanks Kevin for doing these
@dataschool
5 жыл бұрын
You're very welcome! 😄
Great job Kevin. Learned a lot.
@dataschool
5 жыл бұрын
Thanks! Great to hear 👍
the best guy in this business!!!
@dataschool
2 жыл бұрын
Thank you so much! 🙏
I was revisiting dataschool video's. In my view these are still one of the best resources out there. Thanks a lot ...
@dataschool
Жыл бұрын
Thank you!
Thanks for the tricks, specially for the pivot_table() function and pandas_profiling . For me this is a nice summary of pandas. Thanks
@dataschool
4 жыл бұрын
Awesome!
Thanks a lot. This is super useful! I'll try to use as many of these tricks as I can whenever I can :D
@dataschool
2 жыл бұрын
Great to hear!
Thanks Kevin! I actually put tip #23 into use immediately after watching.
@dataschool
5 жыл бұрын
Awesome! Glad you are putting it into use Adam! 🙌
Thank you so much. This is very helpful.
@dataschool
3 жыл бұрын
Great to hear!
The best! Thanks so much!
@dataschool
2 жыл бұрын
You're welcome!
This video was awesome.. Very Brief, to the point. It will be very easy to revise the concept/code by just revisiting them.. Thanks a lot Sir.....
@dataschool
2 жыл бұрын
You're very welcome!
Gr8, thanks for the video, it will be a great time saver and food for thought.
@dataschool
5 жыл бұрын
You are so very welcome, Murali! And thank you for all of your support over the years 🙌
Nice compilation of usefull pandas hacks.... I watched your video to find help on the .merge() method, to combine two dataframes in a sql manner..
Tnx. The .style.format(dict) is a game changer. I used to convert my parameters to ensure a fixed number of parameters would show up well, i.e convert a fraction to a percentage. Not sure why this had never been mentioned in the many videos I have watched before.
@dataschool
2 жыл бұрын
You're very welcome!
Very helpful and useful tricks for day to day Data operations
@dataschool
3 жыл бұрын
Thank you!
Thank you, it's very useful!!!! I have subscribed your video, thank you for your summary.
@dataschool
4 жыл бұрын
You're welcome!
Pandas profiling is a new thing for me. It is good to learn new thing everyday. Subbed !!
@dataschool
3 жыл бұрын
Thanks!
This video helped me so much. Thanks a lot :))))
@dataschool
Жыл бұрын
Great to hear!
Thanks for your help. It's very clear and easy to understand.
@dataschool
4 жыл бұрын
You're very welcome!
all tricks superb, continue good work🎉
@dataschool
Жыл бұрын
Thank you so much 😀
what the awesome tricks!! thank you!!
@dataschool
3 жыл бұрын
You're very welcome!
Excellent content!!!
@dataschool
Жыл бұрын
Much appreciated!
Thanks for compiling it at one place
@dataschool
3 жыл бұрын
You're welcome!
Thanks for the great tips Kevin. Very useful 🙌🏼🙌🏼 Still going through the video, but loved all the ones so far. Will let you know my favorites after I finished 😊
@dataschool
5 жыл бұрын
Awesome! Glad to hear you're finding them useful!
@BabakKeyvani0
5 жыл бұрын
Wow, so hard to pick favorites. I have used most of these in my projects, and was planning to use the styling functionality on some dataframes. certainly didn't know about the bonus trick! Looking forward to trying that. Thanks again for sharing all these 🙌🏼🙌🏼👍🏼👍🏼
@dataschool
5 жыл бұрын
Love the styling functionality! You're very welcome 🙏
AAAAaaaaa sweet tips! Thank you so much!
@dataschool
3 жыл бұрын
You're very welcome!
Absolutely brilliant!
@dataschool
4 жыл бұрын
Thank you!
simply awesome.. helpful, well layed out..Thanks
@dataschool
3 жыл бұрын
Thank you! 🙏
Totally awesome!!! Thanks!!!
@dataschool
5 жыл бұрын
You're very welcome!
it's actually quite helpful, thx a lot
@dataschool
3 жыл бұрын
Great to hear!
Pandas Profiling! I'm so excited to use it!
@dataschool
5 жыл бұрын
Nice! Let me know how it goes 🙌
Awesome video!
@dataschool
3 жыл бұрын
Thank you!