Here buddy you dropped this: 👑

This is really good, Marc, thnx.

@MarcLamberti

3 жыл бұрын

Glad you like it!

@loudsilence5181

3 жыл бұрын

@@MarcLamberti do you cover more in depth customization in your udemy courses? Like, how does one figure out what variables are needed for this thing to work as it should?

very nice article, thanks alot it work for me.

@MarcLamberti

3 жыл бұрын

You're welcome!

hey @Marc! Is there maybe a plan to upgrade this useful material to be relevant for Airflow 2? Thanks

Hi Mark - question on providing credentials/api keys to a specific DAG. Say I need Airlfow to run my python script that reads data from a database, and then pushes something to AWS. I need to store 1) creds for database and 2) for aws (separate from airflow logs bucket). How do you achieve this in this setup you presented?

Hi mark - Even port forwarded to 8080 exactly like in the video but unable see the Airflow UI.

Quick reminder, after running your port forwarding command, open a new shell, even backgrounding port forwarding process will prohibit you from reaching page in browser.

Hey Marc. It was a great tutorial but I always failed installing the helm release. It says: timed out waiting for the condition. Is there any work around I can do to solve this? Thank you

@tomke92z

Жыл бұрын

Same here... I enabled debug and I get the following error: "Error: INSTALLATION FAILED: failed post-install: warning: Hook post-install airflow/templates/create-user-job.yaml failed: jobs.batch "airflow-create-user" already exists helm.go:84: [debug] failed post-install: warning: Hook post-install airflow/templates/create-user-job.yaml failed: jobs.batch "airflow-create-user" already exists"

@antoniusbonni3601

Жыл бұрын

Same. I always got installation failed, tried several times create another yaml file override.yaml but always getting error.

Hey, is there any way that I could use selenium with apache airflow? I am doing some web scraping using selenium. I dont find any ways to integrate it with airflow. One article I found using docker. Problem is I downloaded airflow using PIP. Can you help me?

@MarcLamberti

3 жыл бұрын

Never used Selenium but try your tuto with Docker :) It will be easier btw

Do you have rebuild the values.yaml each time you want to deploy a new DAG as an image, I noticed you absolutely mapped the default repository to just one image in the values.yaml. How is this scalable in an environment where you need to constantly deploy multiple dags.

@Marc when I run the docker build command, I get this error Step 1/2 : FROM apache/airflow:1.10.10.1-alpha2-python3.7 manifest for apache/airflow:1.10.10.1-alpha2-python3 not found: manifest unknown: manifest unknown Can you please help? what could be the issue?

hey marc, lets say I have two machine A and B ...on machine A Kubernetes is running and on B Airflow...so is there any way to send task from airflow to remote cluster lies on A. I know I've to use kubernetesExecutor..but how do I configure it

@MarcLamberti

4 жыл бұрын

You can do this either with the Kubernetes Executor or the KubernetesPodOperator. Take a look at the last one ;)

Why did you not use the helm/stable/airflow chart? Just curious becuase I've tried with that one

@MarcLamberti

3 жыл бұрын

Cause the current stable airflow chart is not the official one :)

@alexkuznicki845

3 жыл бұрын

@@MarcLamberti Haha got it! Thanks for providing the tutorials by the way very helpful. I will ask one more question in a new comment for other uses who may have the same question.

@Marc when I try to login using the username as "admin' as well as the password 'admin' , it gives me error and says, Invalid login. Please try again. Can you please help?

can you share steps in github repository??

How can I access the deployed Airflow on local kubernetes cluster through a local network using IpAddress?

@FuzzPiAndFacial

3 жыл бұрын

You should be able to access via ip double check the port you have set

Hi MarcLamberti, Thanks for creating this video. I couldn't find the materials from the video in the link provided. Can you please upload the repository used in this video?

@MarcLamberti

3 жыл бұрын

I just checked, here is the link: marclamberti.com/youtube-materials/

@rfrost4708

3 жыл бұрын

@@MarcLamberti I can't seem to find any resources on that page. Just some text. Can you confirm once again please.

I get an error saying: Error: failed to download "airflow/airflow" when I run helm install -f values.yml --kube-context kind-airflow-cluster airflow airflow/airflow I have run helm repo update but that doesn't help. Has anyone else experienced this?

@vipulkandivalikar4885

3 жыл бұрын

Same here. I get the same issue too :(

@vipulkandivalikar4885

3 жыл бұрын

@Mohsin Rafi I was never able to pull an image from my local registry from a K8 pod. However when I uploaded it to a public docker hub registry my pods were able to use that image and run without issues.

@cesarvigario

2 жыл бұрын

I think the image does not exist. Go to values.yml and change the parameter "defaultAirflowTag" to something like 1.10.10-2-python3.7.

I get this error Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition When entering "helm install -f values.yml --kube-context kind-airflow-cluster airflow airflow/airflow"

Windows version please

Hey, I Have followed the tutorial and was able to get it working with the default image that you have used (apache/airflow:1.10.10.1-alpha2-python3.7) I am trying to get it working with a private registry (nexus). I'd like to know how do we pass the imagepulsecrets in values.yml file. my values.yml looks as follows. # Default airflow repository -- overrides all the specific images below defaultAirflowRepository: myregistry:5100/image # Default airflow tag to deploy defaultAirflowTag: latest registry: myregistry:5100 secretName: secret_created_in_cluster

@vinaykumararigala1728

4 жыл бұрын

with the above values.yml i get imagepullbacoff error

@kushagraverma7855

4 жыл бұрын

@@vinaykumararigala1728 check pod logs. There should be some errors. Also am forgetting, there are some changes to be made in values.yaml to enable KubernetesExecutor.

@vinaykumararigala1728

4 жыл бұрын

@@kushagraverma7855 the pod is not even coming up actually, I just get a image pull backoff error

Hé I found that the link to your chart is not working any more nor do you have any materials in your "link to the materials". You mention that it is based on the "official chart" and that that is not "stable/airflow", but I can't find that chart anywhere either.. Could you provide a reference.

@MarcLamberti

3 жыл бұрын

Yes, the official chart is in the repository of apache/airflow under the folder chart. Here: github.com/apache/airflow/tree/master/chart

@daanluttik454

3 жыл бұрын

@@MarcLamberti Thanks.

helm install --debug -f values.yml --kube-context kind-airflow-cluster airflow airflow/airflow install.go:172: [debug] Original chart version: "" install.go:189: [debug] CHART PATH: /home/tensorflow/.cache/helm/repository/airflow-1.0.0.tgz client.go:255: [debug] Starting delete for "airflow-broker-url" Secret client.go:109: [debug] creating 1 resource(s) client.go:255: [debug] Starting delete for "airflow-fernet-key" Secret client.go:109: [debug] creating 1 resource(s) client.go:255: [debug] Starting delete for "airflow-redis-password" Secret client.go:109: [debug] creating 1 resource(s) client.go:109: [debug] creating 17 resource(s) client.go:109: [debug] creating 1 resource(s) client.go:464: [debug] Watching for changes to Job airflow-create-user with timeout of 5m0s client.go:492: [debug] Add/Modify event for airflow-create-user: ADDED client.go:531: [debug] airflow-create-user: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 Error: failed post-install: timed out waiting for the condition helm.go:81: [debug] failed post-install: timed out waiting for the condition

@marcusaureliusanonymous

3 жыл бұрын

Can Some help me in order to resolve this issue?

@marcusaureliusanonymous

3 жыл бұрын

Delete the cluster and followed your tutorial again and it worked now but not sure why it happened at first place!!!

This tutorial is broken, the Helm chart is no longer available