Sign up for the totally free tier of Prefect Cloud here: prefec.tv/doug-mercer

Appreciate the effort you put in your videos. Thanks for sharing. keep it up!

@dougmercer

4 ай бұрын

Thanks so much! Will do =]

Could I say this video changed my life? I have gone on a prefect binge. And wow. Already put it to use, and wished I had it all these years instead of cron jobs.

@dougmercer

4 ай бұрын

That's so awesome! It was a game changer for us to. Thanks so much for sharing-- your comment made my morning =]

@LeveragedAlpha

4 ай бұрын

@@dougmercer No - thank you, Doug! So glad I had the privilege of discovering your youtube channel. It's already my favorite technical channels (and among my favorites, overall). You go deep into providing actionable advice for solving problems I face, with incredibly high-quality content, but without the puffery and cringe thumbnails etc. It has already helped me improve as a developer, and I look forward to continuing by watching (and implementing) more videos.

Such a good video! TY for the content. I would 100% take a more indepth course on using Prefect.

@dougmercer

2 ай бұрын

Thanks Adam! I may do another Prefect video in the future, but in the meantime you can maybe check out prefect.io for their blog/docs or their KZread channel @PrefectIO. They have a ton of cool integrations with other software that I barely scratched the surface of

Thank you for the sublime content! I'm curious about the font in your text editor (1:57).

@dougmercer

4 ай бұрын

Hey Max! That's Anonymous Pro. It's available for free on Google Fonts, fonts.google.com/specimen/Anonymous+Pro

Fantastic stuff!

@dougmercer

4 ай бұрын

Thanks! I appreciate it =]

Your videos are incredible! Would you share your editing stack? Cheers!

@dougmercer

2 ай бұрын

Hey Lucas! Thanks so much =] I currently edit on... Davinci Resolve 18.1 Krokodove plugin for Fusion (which is preventing me from updating to latest Davinci Resolve, because BlackMagic is dragging their feet on releasing an updated SDK 😡) A `pygments` formatter I wrote for converting code text to formatted text plus node for my code, which I then animate in Fusion in Resolve. (Which is not automatic, and a total pain in the butt) Epidemic sound for music Storyblocks/envato for other stock media In the future, I'd like to better automate my code animations. Previously I have dumped a ton of hours into trying to make a `manim` (python animation library) extension for animating the difference in code, but manim's Code mobject is really poorly implemented and requires too many workarounds to deal with. I've considered using reveal.js for code animations, but I think the result would be worse for what I do.

The combo prefect+dbt is amazing

Nice channel. Hope it grows.

@dougmercer

11 күн бұрын

Thanks Mauro! I hope so too🤞

This looks really exciting, doesnt seem to do much new, bit really easy to use and all in one place. I can think of some places to use it at work already and am definitely keeping prefect in mind. Ignore the trolls! 😂

@dougmercer

4 ай бұрын

Hah! thanks 😂

I used to run an Apache Airflow installation on an EC2 from AWS, to me its look a lot of an Airflow alternative, is that correct? Can you tell about advantages (and disadvantages) between they, if you know of course?

@dougmercer

4 ай бұрын

Sure! Quick caveat-- I've only used Airflow a bit, but I was so frustrated with the on-boarding and user experience that I gave up on it and have used Prefect ever since. My take is that Prefect is a more Python-native library. Writing Prefect flows/tasks feels a lot more like just writing Python than Airflow, which feels like I'm writing config files with a thin Python wrapper. There's actually quite a few breakdowns comparing the two on the web. - www.prefect.io/prefect-vs-airflow - neptune.ai/blog/argo-vs-airflow-vs-prefect-differences Beyond that, Prefect's blog has an interesting deep dive point out some frustrating "Airflow-isms" and explaining how Airflow can break down at scale-- www.prefect.io/blog/implications-of-scaling-airflow Some pros/cons that were most relevant to me... Prefect Pros: - more modern, user friendly API - really easy to get started (pip install one library, add some decorators, run a terminal command to get a server running for monitoring) - supports dynamic workflows (rather just static DAGs) Prefect Cons: - Some really cool features are Prefect Cloud exclusive. Airflow Pros: - Oldest workflow orchestration out there, so wide user community/documentation (i.e. Mature). Airflow Cons: - I hate their API. - Static DAGs

@glaucojunior

4 ай бұрын

@@dougmercer wow, thank so much for the information, just knwoed your channel and loved it, cheers from Brazil!

@dougmercer

4 ай бұрын

Cheers!

@fffklan3986

22 күн бұрын

pros of airflow: - Mature, tried and tested, runs on 100% open source tech. - multiple ways of writing DAGS (they have a newer taskflow api) - If you're ever looking into using it in a production environment, entry level config of AWS managed airflow can be as little as $400/mo. - do cool stuff, learn data engineering, fun fun fun cons: - if you don't know linux/docker well, good luck. - xcom can be a painpoint (limits the amount of data that can be passed between tasks, meaning that large amounts of data need to be stored between tasks). - you have to use your brainpower and thoughts to accomplish things. pros of prefect: - easy cons: - paid version starts at $1,850. - can't really put it on a resume as a skill (if ur trying to get a job) - sponsoring youtubers for publicity in conclusion: if u want something done quick, use plain old cron. if you want a trusted, robust orchestration system, use airflow.

@martimlobao

16 күн бұрын

I’ve used both Airflow and Prefect for work, and can say that setting up and running Airflow is a million times more painful that Prefect. OTOH, Prefect can become extremely expensive with even just a few users, whereas Airflow is dirt cheap, which is why we ended up switching from Prefect to Airflow 😢

Airflow user here 😂

@dougmercer

24 күн бұрын

Airflow is totally fine 😂! It's been the standard for a while for a reason... That said, when I tried to use it, I felt like I was basically creating a config file through Python kwargs instead of writing Python code. Wasn't the right choice for me

@doofyguck6186

18 күн бұрын

@@dougmercer the way I tend to use Airflow (due to my company's architecture) is composing and extending the operators to fit our needs. So in our main airflow DAG file it's simply a config file but it uses our custom made extended operators. That's where we write most of our logic.

How is this different from Apache Airflow?

@dougmercer

Ай бұрын

I left a pretty detailed reply on @glaucojunior's comment, so maybe check that out High level gist is-- Prefect is a bit more modern, feels like you're writing Python, and has more dynamic features. Airflow is, uh, mature, but feels like you're basically just writing config files laundered. Here's a third party comparison of the two neptune.ai/blog/argo-vs-airflow-vs-prefect-differences

This is how promotional videos should look like!

@dougmercer

Ай бұрын

Thanks! It helps that I had already used their library for years, so I knew what I wanted to make before they reached out.

Looks like a python only alternative to Kestra - and still no sight of the software engineering practice to maintain build and ETL pipelines next to code in a CICD fashion… 😮‍💨

meh, there are alternatives like Flyte, Vertex AI, Kubeflow, etc. Or one can use Kedro to write the pipeline and export it to run in the cloud

@dougmercer

28 күн бұрын

Oh neat, I'll have to look into them a bit more. I've seen kubeflow but not the other three

@BenHammond

7 күн бұрын

…tech people are weird in their level of reflex dismissiveness (I say this as a tech person).

Thanks!

@dougmercer

2 ай бұрын

Wow! Thanks Adam, you made my whole week =]

Why not just use Cron?

@dougmercer

4 ай бұрын

Good question! Basically-- scheduling is only the first of five problems (and is the only problem that CRON would solve). Later in the video, you'll see why. Our code had flaky upstream processes that needed retries, our team had no idea what was going on mid-script run (where the real script took multiple days), we weren't scaling/distributing work effectively, and we didn't have a good system for keeping track of our pipeline's results. CRON alone wouldn't be enough without writing our own... well.. workflow orchestration system. Prefect gave me a ton of features out of the box, and I was able to turn around a refactored script in an afternoon that blew my team's minds.

@dougmercer

4 ай бұрын

@ordinarygg As someone who has written their own general purpose pipelining library (because I wanted to) and multiple web dashboards (using Flask, FastAPI, Streamlit, Panel, etc.), I can promise you that it's way easier to pip install a dependency, add a few decorators, and run a few terminal commands than it is to replicate all the useful functionality that Prefect provides. Check out my comment I left above in response to @marlan__ to get a better idea where I'm coming from.

@Redditard

15 күн бұрын

Me: This all can be fixed using cron and internal error handling.

@AniaKovas

13 күн бұрын

Because it's about more than that, Cron is a crude tool anyway, but also you use the tools you know and are willing to learn. He's using Cron anyhow, and there's, just like I said, more going on. That's the point. I teach this stuff, this is a good lesson. Didn't know about prefect, and this is a good place for my students to look.

Nice video, just wish the music was mute/gone.

@dougmercer

25 күн бұрын

Sorry about that. I saw you posted a similar comment on several videos. I agree that it'd be awesome if KZread supported multichannel audio that allowed the viewer to entirely mute music. In my past videos, I've definitely mixed the audio too loud. I really tried to keep it quiet in the 1brc, and will continue to be more mindful about future videos. However, I probably will never have a video without music. In my personal opinion, music has the potential to make videos more engaging. Admittedly, sometimes my mixing and song selection misses the mark. So, thanks for at least attempting to watch my videos, but sorry about the VO/music mix

how do I make my application more reliable? rely on some dumb cloud ai startup. these all feel like already solved problems, this is just a new set of tooling the developer has to learn, except this time there's probably next to no documentation

@dougmercer

19 күн бұрын

Prefect's open source library is useful in and of itself-- no cloud required. Also, their documentation is good ¯\_(ツ)_/¯

How is this better than airflow?

@dougmercer

10 күн бұрын

"better" is probably a matter of taste. I prefer it because it feels more like writing Python and less like writing a config file. There are several comparisons out there that dive into the differences. Here's a third party link comparing them + another approach (Argo) neptune.ai/blog/argo-vs-airflow-vs-prefect-differences

Novice to automation workflows: why not jenkins?

@dougmercer

29 күн бұрын

Typically Jenkins is used for managing CI/CD pipelines (automating build, test, deployment). Though, I suppose you could use it for data pipelining tasks 🤔 Prefect is a bit more specifically tailored for data engineering, ML ops, or just generally writing pipelines with Python Code (and would not be well suited for administering CI/CD pipelines out of the box). To my mind, similar technologies to Jenkins include GitHub Actions, Gitlab CI, or bamboo. Similar technologies to Prefect include Airflow, Argo, Luigi, Dagster.

@ardenthebibliophile

28 күн бұрын

@@dougmercer appreciate it. The company I work for, while very large, is relatively new to the data engineering space. We've had to learn a lot as regular engineers and scientists as we had a bit of huxterism a couple years ago where worlds were promised and delivery was thin (at best). Add in our technology groups are limited in what is approved for use and it's a complicated scenario that Jenkins, somehow, made it through. Will check out airflow and Argo too!

@dougmercer

28 күн бұрын

My day job sounds really similar to yours. Hopefully you can find something that's approved!

@ardenthebibliophile

28 күн бұрын

@@dougmercer pharmaceutical industry is what I work in

@dougmercer

28 күн бұрын

I work with the government. Getting software approved is a whole process...

Talented but definitely need a senior engineer or a tech lead. There are a lot of ways to solve all of these problems (khm airflow, mlflow😂) Was this a real situation or just made up to help illustrate why this is in fact a great library?

@dougmercer

15 күн бұрын

Yeah, there are a lot of ways to solve these sort of problems. We chose this and it worked well for us ¯\_(ツ)_/¯

This feels like a 15 minute ad. I dislike this video greatly.

@dougmercer

3 ай бұрын

Sorry you feel that way... Although the video is sponsored, this is a true story from my day job, and the script doesn't make any claims I don't stand by. That said, feedback taken. Thanks for commenting.

@kc12394

2 ай бұрын

Dude he literally said it's sponsored in the first 90 seconds of this video and you're still complaining it "feels" like an ad. No one is forcing you to watch a video. You can turn off the video as soon as you hear that it's sponsored you know...