How I Use Python as a Data Engineer
Join Python for Data Engineering Course: datavidhya.com/courses
➡️ Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?pl...
USE CODE: COMBO50 for 50% off
In this video, I have talked about how I use python as a Data Engineer and shared my learnings and best practices I have learned in the last 5 years.
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🎯Twitter - / parmardarshil07
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
Timestamps
0:00 Intro
0:38 Building Data Pipeline using Python
0:51 Data Sources and APIs
1:43 Data Extraction Code
2:14 Data Transformation
2:56 What is Data Pipeline
4:15 Data Storage
4:41 Data Pipeline Deployment
6:44 Course - Python for Data Engineering
7:04 Free Roadmap - Python for Data Engineering
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Hashtags ✨
#dataengineering #python #airflow
Пікірлер: 106
Tried new way of editing, let me know if you like this format. Also new project is coming soon on GCP so don't forget to hit LIKE button
@joelluis4938
Жыл бұрын
Hi. It will be included on your DE course ? or it will be a new one?
@TomKanay
Жыл бұрын
Appreciate your good work. Keep doing. Can you please share more details about your editing software. Which software you are using for video editing?
@My2Cents_
Жыл бұрын
9 Min of Super packed and incredible Video. Thank you so much for sharing the Knowledge.
@markslima1557
Жыл бұрын
Lots of great information here - however sometimes you put up these slides and graphics with good information, but you pull away from them really quickly, I have to pause video to ingest all the information I'm seeing, perhaps be mindful not to cut away too fast. This is not a Michael Bay movie! Otherwise excellent video!
@DarshilParmar
Жыл бұрын
@@markslima1557 This is just an overview video, I have indepth video on each topics Search 1. Twitter Data Pipeline Darshil 2. KZread data analytics 3. Uber Data Analytics You will learn everything in depth
This was really great short and completely on point. Thank you.
Thank you for the clear and crisp summary.
Exactly what was looking for 👍🏻
Brother ... learning a lot from you .. Dedication determination and focus ...Thank you so much :)
Great video!
Thank you for good explaination!
Thank you Darshil🎉
Good one, covering the overview of DE
Crisp and clear ❤
I haven’t added any comment on any video on youtube till now, I’m following you and your videos since more than a year, your content for Data Engineering is tooo amazing among all the other data engineers channels. Keep up the good work buddy!
@DarshilParmar
Жыл бұрын
Awesome! Thank you!
Great video. Recently I have seen a lot of companies encouraging graphql to query their API.
@DarshilParmar
Жыл бұрын
Thank you Karan
hey darshil, how does python (as you described your process in the video) fit in a cloud platform, such as aws/gcp/azure
Very informative video. Lots of information covered in simple terms.
Very valuable information
Amazing video.May I know which editing software are you using?
I think Im loving the way you lecture. Just followed you on twitter and have subscribed to your channel. I want to verge on Data engineer path. Python doesn't stick to my brain or let me say I am being impatient. Any link to learn python fast and understand better? Atlas I can be doing some data ingestion, Minning and visualisation like you
I am learning AWS Refshift cluster from pluralsight but I don't understand anything. I need to learn redshift clusters in depth for a data engineering job. so please give me some solution.and please reply me 🙏
Hiii.. Can you do the azure data engineering projects end to end for one plssss
You are a genius...
This is great , Darshil. This video gives some really wise nugget of knowledge , which is at times hard to get at workplace even and also in any other courses. You have explained it so simply. I am quite experienced but took a long route to understand. You are doing it for free,. Great Job
I enjoy each and every video, learnt something new from you and keep going on your efforts @darshil and Thanks
@DarshilParmar
Жыл бұрын
Thank you Ravi
@Chennaiahch63
Жыл бұрын
kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg
I’m interested in signing up your data engineering class. Will pyspark and airflow be learned in the course? Will the projects involve pyspark or airflow?
What is the best way to make a logic building in python tell me some steps
can you please suggest some budget laptops for student studding data engineering.
Great detailed intro ... Do you have a detailed video of the things that were mentioned here? That would really be a great follow up 🤓 Thank you!
@DarshilParmar
10 сағат бұрын
I have a course, please check - datavidhya.com/combo-course-info
Please share your Python training video for Spark.
bro, I want to learn about data engineering for Redshift. But I see that you have course available for snowflake. My work belongs to manage redshift as of now. What should I do? Please help me.
Your best video!
@DarshilParmar
Жыл бұрын
Thank you
Mast content hai re Bhai 👍🏻
@DarshilParmar
Жыл бұрын
Thank you
Tnq sir
Hi Darshil. Is Pandas syntax necessary to remember for interviews? I'm asking because pandas also supports SQL syntax.
@markslima1557
Жыл бұрын
Pandas is such a useful and perfect library for working with dataframes in Python, I would definitely make it a priority to have at least the basics down. It isn't difficult, and I think rather fun.
Thanks
where else you can find all of this!Its all here
make video on docker in mac
Wow!!
Bhaiya kya aap bata sakte ho ki data engeneer k liye konsi stream leni sahi hogi aur konsi bachelor's degree leni sahi rahegi ?? Please reply
Please make a data engineering project on Rust using polars library.
@Chennaiahch63
Жыл бұрын
kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg
Hello bro...Iam having 8+ yrs experience in Sap basis administrator.. Willing to move to Data science will that be good choice? If yes what is best institute you suggest me to train.
Sir you really are my mentor, I followed all you instructions as a Data Engineer intern. I saw all you Content. I just want to know, how one can do freelance as a Data Engineer. Are we allow to build ETL tools or the system (ETL+Database design+ APIs) and sell ???? Please make a video this 🙏
@Chennaiahch63
Жыл бұрын
kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg
Genius
@darshil parmar Azure users do this python course And how much use poration of this course
bhaiya mujhe data science seekhna hai , lekin usse pahle mujhe basic se python advance tak seekhni hai toh kya agar mein aapkaa course purchase karun toh kam se kam beginner to intermediate tak toh pahuch hee jaunga naa🤔
@DarshilParmar
Жыл бұрын
Haa, basic se advance plus project
Hi Darshil, your python course got English subtitles? cheers
@DarshilParmar
Жыл бұрын
No subtitles but content is in english
Nice explanation bro can you please tell me how to extract data from Api using python code . Can you please send me. In our project we have to write this code in function apps. Please share
@DarshilParmar
Жыл бұрын
Check out my video Twitter Data Pipeline with Airflow
I have 2+ years into the DBA, am I eligible to land to Data engineer????
@Chennaiahch63
Жыл бұрын
kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg
Hii darshil pls add the some projects in azure
@DarshilParmar
Жыл бұрын
Coming soon
Why you have not discussed about dealing with databases in your course: Python for Data Engineering?
@DarshilParmar
6 ай бұрын
It's a part of SQL DE, in order to deal with database you need basic SQL understanding
Please make a course of sql for data engineering as well
@DarshilParmar
Жыл бұрын
It's next in the pipeline
@sachinnambiar
Жыл бұрын
@@DarshilParmar looking forward to it 😁
@jcmc8220
Жыл бұрын
Yeah! 🎉
@Chennaiahch63
Жыл бұрын
kzread.info/head/PLTEjme3I6BCgOc6mmIozEA6jWtLqSGCPg
Make Series on Snowflake with Matillon ETL
Can i do it with PYspark
@DarshilParmar
Жыл бұрын
Yes
Why not azure 🥺
@DarshilParmar
Жыл бұрын
It will come in future
@mohammadmauwaz984
Жыл бұрын
@@DarshilParmar Can you provide a good source to go through for Azure Data Factory not like Udemy.
@moizk8223
Жыл бұрын
@@mohammadmauwaz984 www.youtube.com/@WafaStudies
Bhaiya aap hamesha English hee bolte ho kyaa aap thoda saa simple Hindi mein sikhaa dijiye toh jyada better rahega aapka content accha hai lekin aap Hindi bhi bola kijiye sikhate time ya fir Hindi mein ek course launch kar dijiye
As a beginner, just couldn't follow, too fast
@DarshilParmar
8 ай бұрын
There are end-to-end projects that go in-depth Just search "Darshil Parmar Projects" you'll get hour-long tutorial
brother make it free please people will give you more links and subscribers maybe some people can't afford like me
Bhai thodhi Hindi bolo lo yaar
As a viewer, I've to ask, is that thumbnail makes your content more engaging? I don't understand. For me, it's kind of irritating as everyone opens their mouth...or puts hand on face or smthng
@DarshilParmar
Жыл бұрын
I understand even I don’t like it but when new viewers gets recommended this video reactions make it more clickable, the way KZread algo works and I have to do this way
@shantanutripathi
Жыл бұрын
@@DarshilParmar It's okay because your content is good. That's why I see you daily in my feed. No offence on the your thumbnail but I personally think that an standard way of thumbnail (ex. bytebytego, ibm, Hussain Nasser) would work much better in long run and create more value to viewers
@DarshilParmar
Жыл бұрын
Definitely, I’ll look into this
I think you have lost creativity . This Twitter api is just a repetition of one of your earlier video. Put some effort while making content or u can take a break as well to get back the creativity
@DarshilParmar
Жыл бұрын
I knew this might come but think about this there are many people who never watched my content and this introduces them to my channel, not everyone can watch 1 hr long video but this summary can give me path to watch entire video
@deelovely7565
Жыл бұрын
Right Darshil. This is really helpful for me who has been looking for a video to learn about extracting data using APIs. Thank you.
@ajayram7973
Жыл бұрын
@@deelovely7565 Do you need Data Engineer+ developer?
@deelovely7565
Жыл бұрын
@@ajayram7973 I am learning data engineering
@ajayram7973
Жыл бұрын
@@deelovely7565 All the best 👍
Hi @DarshilParmar I bought the course linked in this video but I’m unable to interact with any videos. Can’t pause, skip, etc. I have emailed you since then but no response yet. Kindly respond to my email or issue a refund. Thank you
@DarshilParmar
Жыл бұрын
Hey, Let me check and will get back to you
@johannest.mensah6516
Жыл бұрын
Thanks so much for the quick resolution! All good now
@mandarjadhav3405
9 ай бұрын
Hey Johannes....I would like to know your review about this course, before I purchase it. Thanks you.
excellent demo Darshil , Please share your contact number - i would like to get a course from you in Data Engineering