I'm a freelancer and also self-taught programmer who wrote a lot of Data Scraping scripts using Python for my clients. I have a loyal client who requires me to extract data for them every month. I repeated this manual process for about 3 months until I realized the scraped data was getting bigger in size and dimension. I thought it was not a data scraping job anymore, it was a data engineering job. After doing some research online, I figured out some useful tools I can use to manage all data collected by the data scraper. I start by storing all the data in an SQL database and automating the process using Apache Airflow. That was the first time I thought Data Engineering was the right direction for my career path as a developer. Thanks for your advice, Jay.

@raffymcfee9846

4 ай бұрын

how long did you train the skills of Data Scraping and Python to make money out of it?

@ajdinpipo08

3 ай бұрын

wow, thanks for sharing this, it's really inspiring me to learn more about data engineering

@vuxuanhuy9079

Ай бұрын

​@@ajdinpipo08 You can sharing resource you learn DE ?

A bit of advice. If you want to work as a data engineer but don't have a computer science background (CS degree or work experience as a software engineer), then there will be a lot of gravity pulling you towards becoming an analyst, i.e. becoming a data user instead of a data builder. If you want to avoid this, you have to be really intentional about educating yourself in the technical aspects of data infrastructure, and pursuing job opportunities where you apply that knowledge. It's easy to forget that moving data around, not looking at the data itself, is what a data engineer does. But businesses often don't understand that, they just lump all the "data people" together. If you're not careful, you'll find yourself creating Tableau dashboards, wrangling data in Pandas, or maintaining Excel spreadsheets, simply because you're the "data guy". Also, if you want a successful career as a data engineer, get comfortable with being uncomfortable. New frameworks come out all the time, and you'll be constantly learning. It's not as crazy as web development, but it still moves relatively fast. If you start feeling comfortable at a job, it's time to ask for a transfer or change jobs, or you'll waste away.

@krisnaaprileo4575

9 ай бұрын

Thanks for advice sir. I'm an accounting student but I'm not comfortable with my majority 😂.

@mairios521

8 ай бұрын

Wow!! Great advice!! I really appreciate it! Thank you a lot!

@skeletor4115

8 ай бұрын

DE since 2016, I agree completely

@md.ikbalhossain4839

7 ай бұрын

That’s a great advice.

@SumanthLazarus

7 ай бұрын

Well said. In a service-heavy IT market like India, most jobs are skewed towards dashboarding and analysis. Data Engineers have to scour over the internet and build projects credibly prior or while interviewing for relevant Data Engineer roles which are few and far between.

I love the way you split the career to basic and advanced skills. I’ve been keep doing that to become the data engineer. Thanks for sharing!

Beginning my path here, great video with no filler!

I wish I would have seen this sooner! Great and crisp content.

really great and clear explanation ! please keep this way. I would recommend this video to anybody wanting for learn about data engineering.

Best Roadmap ever. Just get started with it. Thanks a lot!

Loved your video, it surprised me how you started the second section getting us into the book. It felt like already starting :) thank you!

Very useful tips for those who are just starting out. Great content!

I recently signed up for the data engineering course from Datacamp and I have covered pretty much most SQL courses but I'd love to go back on a few of them again, I work as a data analyst but I feel a career in Engineering would be great.

@holmesketrick7

Жыл бұрын

We are in the same boat

@Xavierpng

Жыл бұрын

​@@holmesketrick7mee to

@kaizen9554

Жыл бұрын

Exactly. SQL can be voluminous to learn. It takes time and practice to let it sink in

@claudiobarrera3186

Жыл бұрын

Hi, I’ve done the complete python track of data analyst, mostly of the sql courses of DataCamp. I’m from Chile, Latin America, do you have any advice for me to find remote jobs as data analyst in the US?

@korands

11 ай бұрын

hi may I ask, does datacamp teach you how to normalize column tables in excel format to tables in sql / other database formats? thankyou

Hands done, the best video I have seen on the subject. It's no nonsense, practical and easy to digest (of course after you put in the work). I am just starting out in my journey to become a Azure Data Engineer so I thank you Jay. Thanks for sharing and being so transparent. I'll keep you posted on my many adventures. For what it's worth, I'm 70 years young and still dedicated to life long learning.

@jayzern

8 ай бұрын

That's amazing! Love the attitude on life long learning

@aidylweiss.

7 ай бұрын

Wow, that is so encouraging! Godspeed!

This is super super SUPER practical and insightful and helpful. Looking forward to your future content and successes!

@jayzern

Жыл бұрын

Thanks for the support Derek

Thank you. Very kind of you to put out this valuable easy to digest and to follow content.

Thanks Jay for posting such an informative and insightful video!!!

What a gem. I am currently pursuing AI/ML. Getting data scientist job as a fresher without experience is almost impossible so had to start with either as data analyst or data engineer. Since now everyone is becoming data analyst so their is demand and supply issue in the job market. Thinking of expanding my domain knowledge to data engineering as well but as you stated “got lost” with the available resources. This video helped me to make a roadmap to pursue data engineering.

@jayzern

Жыл бұрын

Thanks for sharing. I agree it's super difficult to get a DS job fresh out of college. Branching out to DE is a great way to "stand out" from the crowd.

Very good explanation. Love it for the conclusion with the complete motivation.

Amazing video with to the point information. Thank you Jay!

Thanks for the useful tips. Short to the points yet very effective !

Powerful tips! Thanks a lot for lightning the path!

I am transitioning from web development to data. Fortunately for me, I learnt development skills from Learnenough. I have gone through their Commandline course and now on Python and Python for Data Science. I am excited to be doing this. Thank you for this content.

Outstanding video! This video was remarkably well-explained and detailed. It is also structured very well and was easy to follow along with your reasoning. It was a pleasure to watch. I am a Senior iOS Engineer looking to dive deeper into ETL and related topics. This helped me organize what I want to learn. Definitely subscribing.

Great work on explaining the concepts 👍🏽

Thank you for such an informative video, I'm a junior data engineer and your channel is an invaluable resource!

thanks for delivering a well produced video centered around DE

I have been working with python so I think a few very important things 1) pyspark 2) pick AWS and/or Azure to learn their data engineering stack 3) very solid SQL skills.

nice and well structured information, liking this!

Holy f, brother this is an amazing and quick overview. You def got potential in this, keep it up!

@jayzern

Жыл бұрын

Thanks man! So much more to come

This was really helpful. Thanks for sharing ❤️

Hey Jay thanks for sharing all those valuable information with us. I hope to you see soon in NYC! :D thanks again brother!

Amazing and informative!!

Great information. Looking for more videos from you. Keep doing .

Thank you for this video, it created more of a path for me.

Best roadmap for Data Engineering on KZread. New sub Here 😊. Greetings from Argentina 🇦🇷

Hello broo, your video was great! I'm an aspiring Data Engineer and what you said about being overwhelm with many ambigous word on the first year was true, sometimes I feel burn out too. Kinda sad but I know this career is great so I keep it up. This video helps me regain my motivation :))

It was amazing introduction and guide line. Thanks

Nice briefing,waiting for more knowledge sharing on data engineering

I am a fresh graduate in Information Systems and my internship had me working with the team's data lead using Python, Apache Airflow, and PostgreSQL. I was glad and became more confident from the experience but once I was job hunting, I kept finding data engineering job postings demanding tools like Azure, AWS, Spark, Hadoop, etc. I was discouraged as I only had the three I mentioned. But I was a bit encouraged when you talked about tools as getting more specific on what the task demands, and that foundation through concepts can be better.

Love the content bro, hope you keep growing. Its cool to see another Malaysian in the US whose also in the field of Data while making KZread videos on the side.

@jayzern

Жыл бұрын

Thanks bro! Awesome to see other Malaysians too

Excellent information man

i'm a junior data scientist transitioning to data engineering/MLOps. i'm also based in nyc! i'm recently discovering my interest in filmmaking too! looks like i stumbled upon your content at just the right time. just subscribed!!!

@jayzern

Жыл бұрын

No way! data engineering/MLOps and filmmaking is such a great niche

Thanks for this video! I always loved SQL and manipulating data but was put in the software Eng camp. I hated programming so gave it up and moved to IT/admin now I realize data engineering may have been a fun career choice and this video has given me a lot of interest on things to get back into!

Thank you for sharing insights

Thank you, thank you a lot, Jay!! I recently graduated as a Data Scientist but when I see the options of a career path I think Data Engineer fits better with me than others. This video is helpful!! I'm a new subs!

Hugely useful! Subbed!

i was pretty much nervous and scared how to start, thank you for clearing things up :)

Well developed and presented informative video. Thanks a lot 🙏👍

Great job with your career man, mixing technical and artistic paths, great balance.

@jayzern

Ай бұрын

Thanks for the kind words man!

Very insightful video 👏👏👏

Is an excellent introduction I’m far away from being close to a data engineer. I’m just starting off in learning data analytics, but I enjoyed very much this introduction. I felt like it was well explained, and definitely sets the tone moving forward in my learning.

Awesome video Jay! I will personally be looking through the resources that you've recommended.

@jayzern

Жыл бұрын

Thanks oliver, u the best

This is a super high quality video! Cant wait to see this channel grow!

@jayzern

Жыл бұрын

Thanks man!

Eye opening, thanks

Thank you for this great video

Hello bro ,i completed my undergraduate in actuarial but failed miserable and i have been down,filled with regrets and lack motivation ever since but your has given something to believe and get me to regain my confidence . Today begin a new journey,my journey into teaching myself to become a data engineer hope to come back to this post after a uear to share my progress.

@mondlistyleman9389

Жыл бұрын

you’ll do well, being a DE is thee best. and you’ll easily work worldwide

@RNSS_0

11 ай бұрын

I am going back to your comment next year to check you. keep up 🫡

@Odin.x.

7 ай бұрын

How has your journey been so far?

@MajorDougx

4 ай бұрын

​@@Odin.x.He came bck to reality

Thank you so much!

No long ass intro no stupid music I love this video

Useful information thank you

This is hands down the best high level introduction to data engineering I’ve come across. Thank you for making this amazing resource guide as well. I work as a BI analyst now and wanting to expand my skill set so I can truly become a data engineer. I appreciate your time and hard work!

@jayzern

Жыл бұрын

Wow that means a lot! Keep working on expanding your skillsets. Learning is a non-stop journey

@anishnehete

11 ай бұрын

​@@jayzernI'm a college student , I am currently doing the az900 and then I shall do the data bricks certification, are these enough to guarantee an entry level job in data engineering?

I've been a data analyst for a few years and I am just starting to learn the material to transition into the DE space.

This video helps me learn how to learn ML and process data.

Man, this was an awesome video. I'm currently a data analyst and my company has just offered me a junior data engineer role. I have a lot of learning to do and the path you have provided is helpful!

@jayzern

3 ай бұрын

Amazing man, good luck in your new role!

I started / enrolled in courses on DataCamp on path to be Data Analyst then will take courses for Data Engineering. I’m a super excited to learn about Data and the career opportunities

thank for information and good video

Thank you for a great introduction! I have been working in data science for quite some time, but over the years, I realized that the analysis part is much less interesting for me than the data part.

Starting Data Technologies course on September, for now I don't think about money, I just think how to survive and not become mad during education because of huge amount of information

Hi Jay! Thank you so much for this video. Can you please make a video about DE vs Cloud Engineer. It seems that CE make more money but require more skills like AWS/Google Cloud; but a lot of skills overlap as well.

Thanks for the mindset part! I got a BS in DS, but shifted into DE because I enjoyed the backend of getting data for the analysis more than the analysis itself. Being new in the field, it’s definitely overwhelming with all the tools. However, Ill try to take your advice and start small by reading into a tool and doing instead of learning. I guess the best way to learn is to build and try? I always overcomplicate stuff and end up not following through because I try to tackle all these tools at once but will tackle it in smaller parts. From there, keep adding on to my project. Excited as this will be my first “DE” project. Thanks!

@jayzern

10 ай бұрын

Amazing! I started out with a Bachelors in DS too

@updateserver

10 ай бұрын

I do the same thing. It's a blessing and a curse. Scale back. Focus on one tool. Do a project. Join everything together like one complete Lego set. Blog about it. Reach out to recruiters. Wait for the 🎣 & snag a job

Thankss very helpful..

Great content, keep up the good work!

@jayzern

Жыл бұрын

Thank you!

Great content.. the screen you have online with the contents where are u reading this from? The book or from a website?

you earned a subscription because of simplicity and being efficient on teaching!

Thank you Jayzern

I appreciate this video. At 3:00 he mentions that "the objective is to learn about data concepts, not computer science." I'm literally a total novice (but getting deeper into this), and I always thought that computer science was partly focused on data concepts. Or, maybe he means that data concepts is more specifically focused, in contrast to computer science which is extremely broad.

@krisnaaprileo4575

9 ай бұрын

Do I have opportunity to learn together with you mate? I also a total novice but getting deeper 😂

in the project youve mentioned in the core data skills section. are the resources that are involved free? as far as I know, airflow is open source. I have worked with databricks and the whole azure ecosystem, but it was my dads companies subscription.

Hey, First of all thank you for making this video, Is there any way you could provide the doc you are referring to in the video? That will be so helpful.

About to dive into SQL this weekend. I'm just starting my journey into data analytics though. I don't think I have what it would take for an engineer 😅 but I still like watching videos like this cause I like looking at different things

@jayzern

Жыл бұрын

Amazing! Good luck on your data analytics journey.

@DanielleBaylor

Жыл бұрын

@@jayzern thank you! It's a little intimidating, but so far I'm hopeful ❤️. Kind of wish I'd known about this when I was younger though lol

@taishamikell1438

Жыл бұрын

Sis!!! 🗣️YOU HAVE WHAT IT TAKES. Commit to putting in the work and you can become an engineer. Don’t let self-doubt keep you from reaching your potential. You got this!!

@LisaEveShow

9 ай бұрын

How are you doing on this journey?

@DanielleBaylor

9 ай бұрын

@@LisaEveShow considering I have ZERO technical background, it's going ok lol. I'm starting to get the basics of SQL and about to start Python. I kind of wish I'd spent more time on the hard skills this whole time

This is awesome, thank you for sharing your knowledge. I'm a Software Engineer studying for my Master of Science in Data Analytics, maybe becoming a Data Engineer is in my future? Right now, focusing on my SWE job and learning SQL and Python through school have been my ways of learning and doing and while gaining experience (as well as coding projects on the side when I can).

@jayzern

Жыл бұрын

Thanks for sharing Aaron. A lot of people are in the same boat and can totally relate. Enjoy the ride and good luck in your Masters degree.

@atyt22

Жыл бұрын

@@jayzern thanks I subbed, will be tuning in for more videos!

Great information Thanks Jayzern . Could you make a video on HOW I WOULD LEARN SNOWFLAKE DEVELOPER [if i could start over ]

I'm currently trying to sharpen my DE skills and make the transition as well, this video was incredibly helpful. I have an MS in Data Science, an undergrad in Data Analytics & Finance, so I'd love to add these skills to my toolbelt. Perhaps I'll look into Analytics Engineering roles!

Hi, I am looking into this. I started learning analytics but I want to expand my skillset now to also try for DE jobs. I have two doubts, would appreciate if you have time 1. Can someone make a career transition to data engineer from a non tech background(not having a STEM or CS degree)? Do companies accept? 2. I browsed through some jobs on LinkedIn and almost every job asked for some kind of experience in data or Software engineering. So, my doubt is if there are jobs for an entry level role like Junior Data Engineer or Associate Data Engineer?

Hi Jay, is it recommended to have a good overview of certain areas of mathematics before starting to learn the individual stages of data engineering?? Like linear algebra, discrete math, and the like?

Thank you, its a good video

@jayzern

2 ай бұрын

Thanks for the support!

much appreciated

I'm currently transitioning from Data Analytics to Data Engineering. I think DE is only going to grow over the next decade as AI continues to blow up. Thanks for the amazing video jayzern!

@Opq01939

11 ай бұрын

Why u transitioning from data analysis To data engineering? Is there any problem with data analysis?

Designing data intensive applications is another great book by O'Reilly

@jayzern

10 ай бұрын

One of my fav books!

Hi, how do you suggest building and presenting a portfolio?

Thank you can you suggest demanding portfolio or personal project that helps in placing over best mnc

its a very useful video thankyou

Nice Video. May be you can suggest a couple of courses for each section of the video, bcos that is where people have a hard time choosing and taking action.

@Deiniol_1

Жыл бұрын

Exactly

@jayzern

Жыл бұрын

Ooo thanks for the suggestion, will include that in future videos

This is super helpful. More of these please

Could someone please explain how does apache airflow and apache spark differ from Google BigQuery?

Hi thank you very much for this explanatory and helpful video, Please I keep trying the link to the notes from notion but it's not working, can you please help sort it out. I'll be very grateful 🙏. Thank again

Started my journey in learning, I'm currently a systems engineer and it suck lmao, requirements this, requirements that.... anyways, I feel like I found a good solid book to read and some videos on you tube to follow along with for postgreSQL

Hi, Jay. I've been a senior data analyst for 3 years and it's time to move on to DE. This video is so much useful to my career growth! I use SQL(MS sql server and oracle), python(mainly clean data and upload data to database) and tableau a lot in my job, so I'm wondering what's my next step to start learning the skills you mentioned in the video. Any recommendation would be appreaciated. Thank you.

that is the video I am looking to

many thanks, i have confuse 2 things. Learning tools or Building concept . after watch your video. i have more specific look . kee sharing !

Hi Jay, i have a question for data visualization. Do you advise power bi, for data visualization?

thank you

Wow top video! could you give us a list of youtube channels + blogs for Data engineers? Take care!

6:04 that throat clear was intense 😂