The ExternalPythonOperator: No more dependency conflicts in Apache Airflow
Ғылым және технология
The ExternalPythonOperator: No more dependency conflicts in Apache Airflow
🏆 BECOME A PRO WITH AIRFLOW: www.udemy.com/course/the-comp...
Airflow has many options to avoid dependency conflicts.
If you have a Kubernetes cluster, you can use the KubernetesPodOperator to run a task in a Kubernetes Pod.
If you have Docker, then the DockerOperator helps to run a task in a Docker Container.
An easier, though, is the ExternalPythonOperator which requires a pre-created Python virtual environment.
Пікірлер: 6
Hello Marc, I have learned a lot from you since I started to watch your videos! Thank you! Isn't a good practice to always import your modules inside the definition of your task? Shouldn't you have imported yfinance inside "extract_stock_old_yfinance" function also?
@MarcLamberti
8 ай бұрын
Hi 👋 Thank you so much 😊 that’s a good point. I would say yes. If the import is used in one task then make it inside the task. I could have done that indeed 👌
Is airflow running in docker? I have many problems using PythonVirtualenvOperator or ExternalPythonOperator inside docker because you must include system site packages as True (it creates conflicts between venv and base python libraries) or otherwise you will get "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv"
Hello. What if my external project is in another folder? I mean, NOT inside the Docker container. More precisely, I already have 4 projects that perform some ETL tasks, and I would like to orchestrate them using Airflow. How can I achieve that? I see that it's pretty easy to use Airflow with Docker, but how can I execute something that is outside the Docker container?
@DSRajaduraiP
5 ай бұрын
We want to rebuild our docker image