Managed Apache Airflow With Astro! Goodbye MWAA and GCC
Ғылым және технология
Manage Airflow With Astro! Goodbye, MWAA and GCC
Astro is the new way of managing Apache Airflow in the cloud!
It supercharges Airflow by providing all you need to author, run, manage, and monitor your data pipelines at scale without the usual hassles such as no local development environments, no auto-scaling or in-place upgrades, poor CI/CD experience, being stuck in a proprietary ecosystem, etc.
At the end of this video, you will be able to:
1️⃣ Create Airflow deployments for your dev and prod environments
2️⃣ Set up a local development environment with the Astro CLI
3️⃣ Easily and quickly create data pipelines and tasks with Cloud IDE
4️⃣ Define a CI/CD pipeline to deploy in dev and prod automatically
5️⃣ Monitor your pipelines and get alerted if something goes wrong
Try Astro for FREE: bit.ly/43YD1tS
00:00 - Intro
00:37 - Why Astro?
00:55 - Sign up
01:17 - Welcome to Astro
01:19 - Create the DEV Airflow deployment
02:48 - Astro Deployment Features
03:25 - The Data Pipeline you will create
03:52 - Prerequisites
04:15 - Set up your local development environment with the Astro CLI
05:52 - Create the DEV branch
06:03 - Create your data pipeline with Cloud IDE
06:45 - Task 1: Fetch API data
07:14 - Create an Airflow Variable
07:29 - Install Python dependencies
07:47 - Make global imports
07:57 - Test the task
08:10 - Task 2: Print the random activity
08:59 - Commit your DAG with Cloud IDE
10:03 - Create and configure the CI/CD pipeline
10:30 - Create the DEV API key
11:26 - Create the PROD Airflow deployment
11:59 - Create the PROD API key
12:31 - Deploy on DEV with the CICD pipeline
13:17 - Deploy on PROD with the CICD pipeline
13:54 - Create the API variable in the deployments
14:18 - Trigger the DAG
14:26 - Monitoring your deployments and pipelines
14:50 - Get alerted if a DAG run fails
15:19 - Congratulations!
15:47 - User management and more
Enjoy ❤️
#dataengineering #data #airflow
Пікірлер: 20
I do not know if we will be able to use this soon, but if we can, a lot of time and headache will reduce. You people are doing awesome work out there.! Thanks Marc for this detailed demonstration. I will surely try it out.
@MarcLamberti
11 ай бұрын
Whenever you're ready try here bit.ly/43YD1tS and let me know how it goes 🫶
What amazing video. I've learned a few things watching it!
@MarcLamberti
11 ай бұрын
Happy to help 🙏
Why you added API variable in the deployments?
NICE will see the full video when the kids sleep 😂
@MarcLamberti
11 ай бұрын
Hope you gonna enjoy it!
How can I set up Hue, spark, etc in this project? Will I change airflow_settings.yaml?
@MarcLamberti
8 ай бұрын
Create another file docker-compose.override.yml and define you services in there. astro dev start will take it.
Can we install custom packages on airflow instances?
@MarcLamberti
11 ай бұрын
Yes, I show how to do it in the video :)
@mcpiatkowski
11 ай бұрын
@@MarcLamberti Sorry if I missed something but does that include some custom not open sourced packages? For example company data access libraries? And by the way the IDE looks great! It must speed up the prototyping and development tenfold.
When I tried adding organization from astronomer, it reported below error. An error occurred server error
@MarcLamberti
11 ай бұрын
Make sure the organization is unique and let me know if it works now
@nitinkansal
11 ай бұрын
@@MarcLamberti i tried so many names which are unique but still same error
is this video also uploaded to your udemy course ?
@MarcLamberti
11 ай бұрын
wasn't planning to. You want?
@helloworld4553
10 ай бұрын
@@MarcLambertiyes. That would help. Thanks!
Goodbye MWAA/GCC, and also goodbye budget. Astronomer is extremely expensive.
@MarcLamberti
11 ай бұрын
I don’t know if it is “extremely” expensive. I guess it comes down to your use cases and the features you get from Astro than others don’t have. You can find the pricing here www.astronomer.io/pricing/. Let me know if that helps