Python Fundamentals For Data Engineering: Create your first ETL Pipeline
Ғылым және технология
▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
0:00 - Intro
0:25 - Extract Transform Load Example
1:05 - Importing the right packages
1:55 - Extract
2:41 - Transform
4:56 - Load
6:00 - Running first ETL pipeline
8:00 - Outro
Intro
- In the field of data engineering, you are frequently required to read and manipulate data.
- Python is the tool of choice to achieve that.
- In this video, we will cover the fundamentals of Python you need to become a data engineer
- If you are a beginner at coding, there is no need to get overwhelmed with complexities, the idea is to start small and over time as you use more in your daily workflow, the better you will get
- So for Data Engineering you should treat Python as a language to provide a means to an end, don’t need to master it, but you will use this a lot.
In this video, I will cover a sample project to build your first ETL pipeline
GitHub Link: github.com/syalanuj/youtube/b...
FOLLOW ME ON
MEDIUM: / syal.anuj
INSTAGRAM: /
TWITTER: / anuj_syal
GITHUB: github.com/syalanuj
WEBSITE: anujsyal.com
#python #dataengineering #bigdata #etl #extract #transform #load #tutorial
Пікірлер: 26
This is really awesome
Excellent video, very clear
Great content Anuj!
Super helpful, thanks :-)
Thank you Anuj!
That was a very good explanation
excellent video
Very helpful
great video and content!!, i want to ask how to make a two database one is database AdventureWorks database, one for Stagging database, and python code of initial ETL and incremental ETL on the phase of Extraction. thank you
@AnujSyal
6 ай бұрын
See what pip packages support these databases. I would start from psycppg2. As a last resort you can also check apis by these databases if any and then connect to these using python
Nice video what editor plug-in are you using I love it
@AnujSyal
5 ай бұрын
I am using iMovie
@brookster7772
5 ай бұрын
@@AnujSyal thanks man I’m new to python and I was talking about your notebook file in visual studio code. Thanks man.
@AnujSyal
5 ай бұрын
@@brookster7772 That comes in built with Vscode. Use the following syntax to create a cell # %%
Great work! when I try to run import requests, it says No module named requests - what do I need to import to be able to import requests?
@savichopra9083
15 күн бұрын
pip install requests
This is cool, only one thing missingg and its how to orchestate this, how to setup automation
@AnujSyal
11 ай бұрын
Thanks for the suggestion, I do have a video on docker and airflow, maybe that one can give you some idea in terms of automation
Hi Anuj, When I try to code this in google Colab. I'm getting error that "name 'transform' is not defined"
@nakul--
Жыл бұрын
make sure you have imported it correctly at the beginning of your code. import via : from sklearn.preprocessing import transform
@AnujSyal
Жыл бұрын
Yeah Nakul is right, it looks like an import error
how did you open dataviewer to see the actual dataframe?
@AnujSyal
9 ай бұрын
When you configure running jupyter notebook it comes with that. You can see Variables as part of the jupyter environment, and go into each dataframe to check in the values in memory. It is really similar to spyder
SQL on Steroids is pandas, that was a Good one 😅
haha pandas is like excel on steriods! lolllll
All good except the fake accent