How to Run a Python Docker Image on AWS Lambda
Learn how to run a Python Docker image on AWS Lambda using Lambda container images.
Using Docker containers for your Python app on AWS Lambda means you can create a special environment for your app with its own tools and settings, making sure it works everywhere, and also making it easy to include all the things (libraries and packages)your app needs to run correctly.
🛠️ Project Code: github.com/pixegami/aws-lambd...
🔗 Links
Lambda Base Images: docs.aws.amazon.com/lambda/la...
AWS CLI: aws.amazon.com/cli/
AWS CDK: docs.aws.amazon.com/cdk/v2/gu...
Docker: www.docker.com/
📚 Chapters
00:00 Why run Docker on AWS Lambda?
01:04 Create CDK Project
01:45 Create a Python handler app
04:07 Create Dockerfile for Lambda
06:23 Test the Docker image locally
07:11 Create AWS Docker Lambda with CDK
09:44 Deploying to AWS
11:59 Wrapping up
#pixegami #aws
Пікірлер: 103
You are brilliant!! Following your steps, I deployed several Python apps to AWS Lambda. Your instructions were clear, code was perfect (worked fine without any changes) and explanations were precise. Very rare in free KZread tutorials. Thank you.
@pixegami
5 ай бұрын
That's awesome feedback. Thanks for sharing that! Glad it was useful for you.
Dude, I cannot tell you how AMAZING you are and how AMAZING this video is. I could literally hug you through the screen. Thank you so much for your video man. It was a PERFECT explanation.
@pixegami
4 ай бұрын
Thank you! I'm glad it was helpful.
Fantastic video. It covered so much (CDK, Docker, Lambda) in just 13 minutes. And it actually all works really easily. My first interaction with lambda was a few years ago and decided I would never, ever touch it. But after this video, with CDK and the other progress since then, I am reconsidering moving my workflows over to lambda instead of FastAPI. Pretty much all because of watching this video.
@pixegami
3 ай бұрын
Thank you! I really appreciate this kind of feedback, and honestly it's hearing things like this that motivate me to make this content. I hope that project all goes well for you!
Thank you for sharing this! It saves me a lot of time trying to create a bunch of layers to execute some heavy deps functions on AWS Lambda 🤯🙏
@pixegami
5 ай бұрын
You're welcome! I'm glad to hear it helped :)
How do you even know what we want? Short simple to the point explanation. Thank you so much. Please keep creating such amazing videos!
@pixegami
2 ай бұрын
Thank you, I appreciate your comment, and I'm glad you enjoyed it!
This is was so helpful! Thank you so much!
I'm very thankful for this video. I really struggle with GUI tools like websites. This command line interface solution really helped me to ship and urgent work I was struggling to configure on aws web interface. Thank you for your work.
@pixegami
8 ай бұрын
Glad it helped!
omg exactly what i needed thanks so much!!
@pixegami
11 ай бұрын
You're welcome!
Perfect! This video is brilliant, If anyone need to Run a Python Docker Image on AWS Lambda. Check this, You will definitely able to solve your issue
@pixegami
Ай бұрын
Glad it was useful!
Thank you for putting so much effort into this video, the quality is just magnificent. Your channel is super underrated, I hope you get the recognition that you deserve soon.
@pixegami
Ай бұрын
Thank you! Really glad to hear you enjoyed this :)
Great video! Thanks :)
This is perfect! Great work, thanks!!!
@pixegami
3 ай бұрын
Glad it was useful, thank you!
wow! Excelent video. Thanks so much!
@pixegami
8 ай бұрын
You're welcome!
New skill added to my resume, thank you so much :D
@pixegami
8 ай бұрын
Congrats! Well done :)
You've earned a sub my man.
I absolutely needed this bro. Thank you so much. +1 follow
@pixegami
7 ай бұрын
Thank you! Glad it helped :)
Thank you, gained a sub.
@pixegami
6 ай бұрын
Thank you :)
Great. thank you
@pixegami
5 ай бұрын
No worries!
THANKS
@pixegami
2 ай бұрын
You're welcome!
hello thank you for your video , i have a small question , in my lamda function i have some api keys ,how to do the .env config in aws with docker
What if we want to use env variables? Where would we place them and how can we retrieve them in the py file?
Thanks a lot ! Nice tutorial. Do you plan to make a video tutorial for a larger project, such as building a full-stack website or something similar ?
@dchenk
11 ай бұрын
Just use docker-compose for this.
@pixegami
11 ай бұрын
Thanks! I do plan some in the coming months, but right now I'm just trying to build a base of smaller videos (like this one) to cover all the basic tools first :)
@ueiwqoak
8 ай бұрын
@@pixegami bump on @davidcasanova2951 's request. Fantastic video
Great video! When creating the CDK project why do you choose typescript as your language and not python? When I choose python as the language the lib folder is created in the venv folder.
@pixegami
11 ай бұрын
I tried Python before and I'm not sure I like the experience as much as Typescript. CDK is natively developed in Typescript, so I feel it makes the quality of code hinting better, error messages easier to read (without all the JSII stuff they used to translate it to Python) and most importantly, it has the largest example base online and in tools like GPT.
king
Error saying invalid entry point i am using arm64?? Im using m2 any fixes?
I don't quite get the part on LAMBA_TASK_ROOT? is it an environment variable that u need to set up when u first create the lambda function? the video doesn't seem to cover it..
Would this setup also work to deploy a streamlit app? Or is it only for request-response type apps?
@pixegami
8 ай бұрын
Ah, I don't think it works well for Streamlit. Streamlit relies on a session and an active connection with the user, but Lambda doesn't really do that. But stay tuned, I'm also working on a tutorial for deploying Streamlit with Docker to AWS as well.
Thanks. It could be more helpful if you walk through the permission setting in IAM, as I end up spending an hour configuring that during bootstrapping as a newbie.
@pixegami
7 ай бұрын
Ah I see, thanks for sharing that. IAM is one of those things that I often forget can sometimes be very challenging unless you've used it before. Noted for next time!
@hamishrobertson946
7 ай бұрын
fyi for me these issues popped up around the 943 mark (Deploy to AWS). Assuming you are running as a managed user you'll need to add the appropriate policies to your IAM group as 'Permission denied' starts popping up each time you run cdk deploy/bootstrap. I had to add stuff like AmazonS3FullAccess and AmazonSSMFullAccess
Is there anything extra I have to do to get environment variables working with this setup?
@pixegami
2 ай бұрын
You should be able to apply environment variables via the CDK construct for Lambda. See the `environment` parameter: docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html
thanks for tutorial. What is your terminal that suggest commands?
@pixegami
8 ай бұрын
I use fish terminal. See this video for my Mac terminal setup: kzread.info/dash/bejne/p5urz6qYqN3Rc6Q.html
Great explanation. What are ways you can pass parameters to the container?
@pixegami
5 ай бұрын
Thank you! Well, it depends on what kind of parameters you'd like to pass. Environment variables are probably an obvious and easy way to get started. You could also potentially modify/override the "command" or the "entry point" it is executed with.
im trying to create a docker image for a python lambda from an amplify app, how would this work for that setup?
@pixegami
11 ай бұрын
I haven't used Amplify, but from what I understand it creates and manages the Lambda runtime for you. I'm not sure if you can extend it to support images. Let me know if you find out otherwise. That's why for more fine-grained control Iike to use AWS CDK for the infrastructure.
Does the cost of lambdas change with the use of ECR? Or is there any additional cost generated by using ECR?
@pixegami
4 ай бұрын
ECR cost would be billed separately to Lambda. I think ECR also has a free tier? aws.amazon.com/ecr/pricing/
Fantastic video! However, how do we handle logic that takes more than 3 seconds to execute, considering Discord messages have a 3-second limit?
what is the terminal autocomplete feature you are using it's nice how it autocompletes for you
@jeronimoahumada7817
9 ай бұрын
Looks like zsh-autosuggestions
@pixegami
8 ай бұрын
Yup! Or you can use `fish` terminal and get it out of the box. I have a video on the terminal setup here: kzread.info/dash/bejne/p5urz6qYqN3Rc6Q.html
Hi, this setup spike my billing very high, The setup was to build lambda function to read the latest file from the s3 dir and make transformation then finally to s3 target dir, So this all setup with the python script has to run once the s3 notification to lambda function that an file just came to s3. But it went into a loop and made the s3 and lambda billing spike Let me knew what is the issue in my setup that i didn't noticed at first while running this python script in lambda
@pixegami
Ай бұрын
Uh oh. It sounds like you have a recursive Lambda function. I guess after you update the S3 file, it probably triggers another event (which triggers the function) in a loop. Normally, if you want to trigger Lambda from an S3 event, I wouldn't recommend to update/write back into the same S3 - maybe just use a different bucket for the output. For the billing spike, you can create a ticket to AWS support to explain the mistake, and see if they'll be able to give you a refund on the cost (it usually works if it's the first time you are making an honest mistake).
Amazing. I'm facing an issue in Dockerfile in "RUN pip install -r requirements.txt". when I run cdk deploy, I get Failed to build asset error. In local, it is running correctly. Any idea how to resolve this? meanwhile I commented 'RUN pip install -r requirements.txt' and returned just dummy message without numpy. This time Cdk deploy worked and the url was giving me the json response
@pixegami
5 ай бұрын
Hmm, I think we'll need to see what the actual error is - the build could fail for a variety of different reasons.
I follow your steps and faild to build on the RUN pip install ... I commented it out and can build the image. What it take to get pip command to work? I use Windows 10 and curl command (in PowerShell) does not work with the URL. What's the ../2015-03-03/... for?
@pixegami
10 ай бұрын
Hmm, what error do you see when you try to run pip install?
@mohamedabdo-xg6tw
9 ай бұрын
@@pixegami I built my image, it runs with no errors but when I do Curl nothing happens. so, as saml asked, what ../2015/... stands for? or where it comes from?
@kepitingtiara
5 ай бұрын
@@mohamedabdo-xg6tw , I did have same questions, here the GPT help me The 2015-03-31 represents the version of the AWS Lambda API. When you send an HTTP POST request to this endpoint, it triggers the Lambda function handler specified in your Docker container.
I came here after trying to install my libraries to my python lambda function and it give up so I am trying to upload it as a docker image
@pixegami
3 ай бұрын
Yup, uploading a library with Python *without* Docker is surprisingly difficult, especially if you want to use libraries that have an OS-dependent library under the hood (which includes AI/ML libraries, numpy, and more). That's why the Docker way is really good to know :)
Hii can you make a new video for Docker Python API deployment?
Is it possible to deploy docker image using terraform? If so, can you do a video on this please?
@pixegami
3 ай бұрын
I'm gonna guess it's possible, but I haven't used Terraform before. I'll add it to my list to check out!
Trying your tutorial on windows, prompted the error: 'curl: (52) Empty reply from server'. Any help will be apreciated!
@pixegami
5 ай бұрын
Is this for when you are testing it locally? What about the Lambda docker output itself? In the terminal where you are running the image locally, do you see any messages?
does not work for me, my curl command gets stuck without any response
@pixegami
8 ай бұрын
Have you run the container successfully? Do you see any output from the container terminal?
Solution : on M chip Mac : change credstore from desktop to osxkeychain , use sudo nano ~/.docker/config.json Thanks for sharing but I’m getting the error ERROR: failed to solve: public.ecr.aws/lambda/python:3.11: error getting credentials - err: exit status 1, out: ``. Can you please advise a solution ?
@pixegami
5 ай бұрын
Hmm, sounds like a permissions issue. I recommend checking how you are setting up your AWS credentials for this. AWS goes through a series of steps to try and determine your credentials: docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html#credentialProviderChain
07:02 it's interesting that I've done the same with Postman and it didn't work. CURL works, though
@pixegami
7 ай бұрын
Hmm, I wonder why. Does it throw an error?
On the pre-requisites. How to install cdk on aws 2023 linux image. sudo yum install nodejs npm sudo npm install -g aws-cdk sudo npm install -g npm@10.2.5 sudo npm install -g aws-cdk
I wish real world senerios are this simple lol, I am currently using node and canvas and that's needed a lot of Os level dependies like gcc-c++ cairo-devel etc, how to cache them because everytime I make changes I need to build docker image again and that is taking too much time
@pixegami
5 ай бұрын
Well, tutorials and examples are always going to be a simpler cleaner version of real world problems which does have its own challenges :) But your problem does have a solution. Docker builds images in layers, so anything you want to update/change, just put them in a later layer (e.g. lower in the Dockerfile) so it doesn't have to rebuild everything up until that point. That way it'll just re-build your changes into the image without having to go through the dependencies installation again. If you want to take it a step further, check out using GitHub actions to automate your workflow and deployment (it also has caching): kzread.info/dash/bejne/a6mBxbBvZtDMp6g.html
Dumb question, what terminal are you using?
Nice video. Thanks @pixegami
@pixegami
8 ай бұрын
Thank you!
On the pre-requisites. How to instal Docker on aws 2023 linux image. sudo yum update -y sudo yum install -y docker sudo systemctl start docker sudo systemctl enable docker sudo groupadd docker sudo usermod -aG docker $USER docker version
@pixegami
7 ай бұрын
Thanks for sharing this!
@aacesta
7 ай бұрын
After the above you may also have to restart your machine before commands such as docker ps or similar can succeed in connecting to the daemon.
It's not python it's typescript
@pixegami
7 ай бұрын
The Docker image is a Python app. If you mean the CDK - yes that's Typescript. You can use Python for it too, but I generally go with Typescript since it's the 'native' language for CDK.
Great video, thank you! Wondering how this would work if using poetry instead of requirements.txt? I've seen a lot of different answers on SO and yotube.