How I Use Python as a Data Engineer

Join Python for Data Engineering Course: datavidhya.com/courses
➡️ Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?pl...
USE CODE: COMBO50 for 50% off
In this video, I have talked about how I use python as a Data Engineer and shared my learnings and best practices I have learned in the last 5 years.
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🎯Twitter - / parmardarshil07
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
Timestamps
0:00 Intro
0:38 Building Data Pipeline using Python
0:51 Data Sources and APIs
1:43 Data Extraction Code
2:14 Data Transformation
2:56 What is Data Pipeline
4:15 Data Storage
4:41 Data Pipeline Deployment
6:44 Course - Python for Data Engineering
7:04 Free Roadmap - Python for Data Engineering
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Hashtags ✨
#dataengineering #python #airflow

Пікірлер: 106

  • @DarshilParmar
    @DarshilParmar Жыл бұрын

    Tried new way of editing, let me know if you like this format. Also new project is coming soon on GCP so don't forget to hit LIKE button

  • @joelluis4938

    @joelluis4938

    Жыл бұрын

    Hi. It will be included on your DE course ? or it will be a new one?

  • @TomKanay

    @TomKanay

    Жыл бұрын

    Appreciate your good work. Keep doing. Can you please share more details about your editing software. Which software you are using for video editing?

  • @My2Cents_

    @My2Cents_

    Жыл бұрын

    9 Min of Super packed and incredible Video. Thank you so much for sharing the Knowledge.

  • @markslima1557

    @markslima1557

    Жыл бұрын

    Lots of great information here - however sometimes you put up these slides and graphics with good information, but you pull away from them really quickly, I have to pause video to ingest all the information I'm seeing, perhaps be mindful not to cut away too fast. This is not a Michael Bay movie! Otherwise excellent video!

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    @@markslima1557 This is just an overview video, I have indepth video on each topics Search 1. Twitter Data Pipeline Darshil 2. KZread data analytics 3. Uber Data Analytics You will learn everything in depth

  • @samiiiiii2413
    @samiiiiii241310 ай бұрын

    This was really great short and completely on point. Thank you.

  • @moeen007
    @moeen007 Жыл бұрын

    Thank you for the clear and crisp summary.

  • @leogodofnone2706
    @leogodofnone2706 Жыл бұрын

    Exactly what was looking for 👍🏻

  • @craftymehreen1847
    @craftymehreen1847 Жыл бұрын

    Brother ... learning a lot from you .. Dedication determination and focus ...Thank you so much :)

  • @BillusTinnus
    @BillusTinnus Жыл бұрын

    Great video!

  • @livetoday5849
    @livetoday58495 ай бұрын

    Thank you for good explaination!

  • @technicalsuranii
    @technicalsuranii11 ай бұрын

    Thank you Darshil🎉

  • @prasadbbd3525
    @prasadbbd3525Ай бұрын

    Good one, covering the overview of DE

  • @kammaridurgacharan431
    @kammaridurgacharan431 Жыл бұрын

    Crisp and clear ❤

  • @mohitbhatia140
    @mohitbhatia140 Жыл бұрын

    I haven’t added any comment on any video on youtube till now, I’m following you and your videos since more than a year, your content for Data Engineering is tooo amazing among all the other data engineers channels. Keep up the good work buddy!

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Awesome! Thank you!

  • @karangupta_DE
    @karangupta_DE Жыл бұрын

    Great video. Recently I have seen a lot of companies encouraging graphql to query their API.

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Thank you Karan

  • @jerbear97
    @jerbear9710 ай бұрын

    hey darshil, how does python (as you described your process in the video) fit in a cloud platform, such as aws/gcp/azure

  • @rajulsahu9218
    @rajulsahu92183 ай бұрын

    Very informative video. Lots of information covered in simple terms.

  • @krish0804
    @krish0804 Жыл бұрын

    Very valuable information

  • @playwithvihaan7601
    @playwithvihaan7601 Жыл бұрын

    Amazing video.May I know which editing software are you using?

  • @bamidelak
    @bamidelak Жыл бұрын

    I think Im loving the way you lecture. Just followed you on twitter and have subscribed to your channel. I want to verge on Data engineer path. Python doesn't stick to my brain or let me say I am being impatient. Any link to learn python fast and understand better? Atlas I can be doing some data ingestion, Minning and visualisation like you

  • @viral.jhgggg669
    @viral.jhgggg669 Жыл бұрын

    I am learning AWS Refshift cluster from pluralsight but I don't understand anything. I need to learn redshift clusters in depth for a data engineering job. so please give me some solution.and please reply me 🙏

  • @nagulmeerashaik5336
    @nagulmeerashaik5336 Жыл бұрын

    Hiii.. Can you do the azure data engineering projects end to end for one plssss

  • @nanayaw7159
    @nanayaw7159 Жыл бұрын

    You are a genius...

  • @payalbhatia5244
    @payalbhatia5244 Жыл бұрын

    This is great , Darshil. This video gives some really wise nugget of knowledge , which is at times hard to get at workplace even and also in any other courses. You have explained it so simply. I am quite experienced but took a long route to understand. You are doing it for free,. Great Job

  • @Ravi_Teja_Padala_tAlKs
    @Ravi_Teja_Padala_tAlKs Жыл бұрын

    I enjoy each and every video, learnt something new from you and keep going on your efforts @darshil and Thanks

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Thank you Ravi

  • @Chennaiahch63

    @Chennaiahch63

    Жыл бұрын

    kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg

  • @looklook6075
    @looklook60757 ай бұрын

    I’m interested in signing up your data engineering class. Will pyspark and airflow be learned in the course? Will the projects involve pyspark or airflow?

  • @praveenbhandari2516
    @praveenbhandari251611 ай бұрын

    What is the best way to make a logic building in python tell me some steps

  • @innovativewaves1847
    @innovativewaves1847 Жыл бұрын

    can you please suggest some budget laptops for student studding data engineering.

  • @jeffrey6124
    @jeffrey612411 сағат бұрын

    Great detailed intro ... Do you have a detailed video of the things that were mentioned here? That would really be a great follow up 🤓 Thank you!

  • @DarshilParmar

    @DarshilParmar

    10 сағат бұрын

    I have a course, please check - datavidhya.com/combo-course-info

  • @ranjansrivastava9256
    @ranjansrivastava925611 ай бұрын

    Please share your Python training video for Spark.

  • @kiransalagala6249
    @kiransalagala62498 ай бұрын

    bro, I want to learn about data engineering for Redshift. But I see that you have course available for snowflake. My work belongs to manage redshift as of now. What should I do? Please help me.

  • @moizk8223
    @moizk8223 Жыл бұрын

    Your best video!

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Thank you

  • @Sharmasurajlive
    @Sharmasurajlive Жыл бұрын

    Mast content hai re Bhai 👍🏻

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Thank you

  • @AmarNath-zh8cv
    @AmarNath-zh8cv Жыл бұрын

    Tnq sir

  • @rayyanabdulwajid7681
    @rayyanabdulwajid7681 Жыл бұрын

    Hi Darshil. Is Pandas syntax necessary to remember for interviews? I'm asking because pandas also supports SQL syntax.

  • @markslima1557

    @markslima1557

    Жыл бұрын

    Pandas is such a useful and perfect library for working with dataframes in Python, I would definitely make it a priority to have at least the basics down. It isn't difficult, and I think rather fun.

  • @RaNd0MiZeD
    @RaNd0MiZeD Жыл бұрын

    Thanks

  • @devrohan9679
    @devrohan9679 Жыл бұрын

    where else you can find all of this!Its all here

  • @syedhashir5014
    @syedhashir5014 Жыл бұрын

    make video on docker in mac

  • @aniketbhalla1521
    @aniketbhalla15215 күн бұрын

    Wow!!

  • @ghostareagaming6895
    @ghostareagaming6895 Жыл бұрын

    Bhaiya kya aap bata sakte ho ki data engeneer k liye konsi stream leni sahi hogi aur konsi bachelor's degree leni sahi rahegi ?? Please reply

  • @shashankemani1609
    @shashankemani1609 Жыл бұрын

    Please make a data engineering project on Rust using polars library.

  • @Chennaiahch63

    @Chennaiahch63

    Жыл бұрын

    kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg

  • @syedansar4681
    @syedansar4681 Жыл бұрын

    Hello bro...Iam having 8+ yrs experience in Sap basis administrator.. Willing to move to Data science will that be good choice? If yes what is best institute you suggest me to train.

  • @ajayram7973
    @ajayram7973 Жыл бұрын

    Sir you really are my mentor, I followed all you instructions as a Data Engineer intern. I saw all you Content. I just want to know, how one can do freelance as a Data Engineer. Are we allow to build ETL tools or the system (ETL+Database design+ APIs) and sell ???? Please make a video this 🙏

  • @Chennaiahch63

    @Chennaiahch63

    Жыл бұрын

    kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg

  • @HeshamNasser0
    @HeshamNasser09 ай бұрын

    Genius

  • @ashishchauhan8328
    @ashishchauhan8328 Жыл бұрын

    @darshil parmar Azure users do this python course And how much use poration of this course

  • @FacebookInc.
    @FacebookInc. Жыл бұрын

    bhaiya mujhe data science seekhna hai , lekin usse pahle mujhe basic se python advance tak seekhni hai toh kya agar mein aapkaa course purchase karun toh kam se kam beginner to intermediate tak toh pahuch hee jaunga naa🤔

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Haa, basic se advance plus project

  • @user-yv1ue2ml9s
    @user-yv1ue2ml9s Жыл бұрын

    Hi Darshil, your python course got English subtitles? cheers

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    No subtitles but content is in english

  • @sravankumar1767
    @sravankumar1767 Жыл бұрын

    Nice explanation bro can you please tell me how to extract data from Api using python code . Can you please send me. In our project we have to write this code in function apps. Please share

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Check out my video Twitter Data Pipeline with Airflow

  • @govardhan9418
    @govardhan9418 Жыл бұрын

    I have 2+ years into the DBA, am I eligible to land to Data engineer????

  • @Chennaiahch63

    @Chennaiahch63

    Жыл бұрын

    kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg

  • @diwakarnrp2092
    @diwakarnrp2092 Жыл бұрын

    Hii darshil pls add the some projects in azure

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Coming soon

  • @zeeshanu6
    @zeeshanu66 ай бұрын

    Why you have not discussed about dealing with databases in your course: Python for Data Engineering?

  • @DarshilParmar

    @DarshilParmar

    6 ай бұрын

    It's a part of SQL DE, in order to deal with database you need basic SQL understanding

  • @sachinnambiar
    @sachinnambiar Жыл бұрын

    Please make a course of sql for data engineering as well

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    It's next in the pipeline

  • @sachinnambiar

    @sachinnambiar

    Жыл бұрын

    @@DarshilParmar looking forward to it 😁

  • @jcmc8220

    @jcmc8220

    Жыл бұрын

    Yeah! 🎉

  • @Chennaiahch63

    @Chennaiahch63

    Жыл бұрын

    kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg

  • @user-ke2uw9kx2o
    @user-ke2uw9kx2o11 ай бұрын

    Make Series on Snowflake with Matillon ETL

  • @SlicerJi
    @SlicerJi Жыл бұрын

    Can i do it with PYspark

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Yes

  • @mohammadmauwaz984
    @mohammadmauwaz984 Жыл бұрын

    Why not azure 🥺

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    It will come in future

  • @mohammadmauwaz984

    @mohammadmauwaz984

    Жыл бұрын

    @@DarshilParmar Can you provide a good source to go through for Azure Data Factory not like Udemy.

  • @moizk8223

    @moizk8223

    Жыл бұрын

    @@mohammadmauwaz984 www.youtube.com/@WafaStudies

  • @FacebookInc.
    @FacebookInc. Жыл бұрын

    Bhaiya aap hamesha English hee bolte ho kyaa aap thoda saa simple Hindi mein sikhaa dijiye toh jyada better rahega aapka content accha hai lekin aap Hindi bhi bola kijiye sikhate time ya fir Hindi mein ek course launch kar dijiye

  • @ntuthukophlybeatzmakhathin898
    @ntuthukophlybeatzmakhathin8988 ай бұрын

    As a beginner, just couldn't follow, too fast

  • @DarshilParmar

    @DarshilParmar

    8 ай бұрын

    There are end-to-end projects that go in-depth Just search "Darshil Parmar Projects" you'll get hour-long tutorial

  • @Funnyanimalstalkk
    @Funnyanimalstalkk Жыл бұрын

    brother make it free please people will give you more links and subscribers maybe some people can't afford like me

  • @sumitrawat0001
    @sumitrawat0001 Жыл бұрын

    Bhai thodhi Hindi bolo lo yaar

  • @shantanutripathi
    @shantanutripathi Жыл бұрын

    As a viewer, I've to ask, is that thumbnail makes your content more engaging? I don't understand. For me, it's kind of irritating as everyone opens their mouth...or puts hand on face or smthng

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    I understand even I don’t like it but when new viewers gets recommended this video reactions make it more clickable, the way KZread algo works and I have to do this way

  • @shantanutripathi

    @shantanutripathi

    Жыл бұрын

    @@DarshilParmar It's okay because your content is good. That's why I see you daily in my feed. No offence on the your thumbnail but I personally think that an standard way of thumbnail (ex. bytebytego, ibm, Hussain Nasser) would work much better in long run and create more value to viewers

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Definitely, I’ll look into this

  • @sarahkhan-hg9fo
    @sarahkhan-hg9fo Жыл бұрын

    I think you have lost creativity . This Twitter api is just a repetition of one of your earlier video. Put some effort while making content or u can take a break as well to get back the creativity

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    I knew this might come but think about this there are many people who never watched my content and this introduces them to my channel, not everyone can watch 1 hr long video but this summary can give me path to watch entire video

  • @deelovely7565

    @deelovely7565

    Жыл бұрын

    Right Darshil. This is really helpful for me who has been looking for a video to learn about extracting data using APIs. Thank you.

  • @ajayram7973

    @ajayram7973

    Жыл бұрын

    ​@@deelovely7565 Do you need Data Engineer+ developer?

  • @deelovely7565

    @deelovely7565

    Жыл бұрын

    @@ajayram7973 I am learning data engineering

  • @ajayram7973

    @ajayram7973

    Жыл бұрын

    @@deelovely7565 All the best 👍

  • @johannest.mensah6516
    @johannest.mensah6516 Жыл бұрын

    Hi @DarshilParmar I bought the course linked in this video but I’m unable to interact with any videos. Can’t pause, skip, etc. I have emailed you since then but no response yet. Kindly respond to my email or issue a refund. Thank you

  • @DarshilParmar

    @DarshilParmar

    Жыл бұрын

    Hey, Let me check and will get back to you

  • @johannest.mensah6516

    @johannest.mensah6516

    Жыл бұрын

    Thanks so much for the quick resolution! All good now

  • @mandarjadhav3405

    @mandarjadhav3405

    9 ай бұрын

    Hey Johannes....I would like to know your review about this course, before I purchase it. Thanks you.

  • @user-lk2dz3mw9h
    @user-lk2dz3mw9h5 ай бұрын

    excellent demo Darshil , Please share your contact number - i would like to get a course from you in Data Engineering