Configure GitLab CI Runner with Docker executor using AWS EC2

In this tutorial, I will show you how to Configure your own GitLab Runner with a Docker Executor using AWS EC2.
🔻 Click on SHOW MORE 🔻
This is useful if you are trying to use GitLab.com and are running run out of minutes or simply need more power.
I will show you how you can run your own GitLab Runner on AWS and cover:
- creating a new virtual machine (VM) on AWS using EC2
- installing the GitLab Runner on the VM
- configuring the GitLab Runner to work with GitLab.com
--
⭐️ Video contents ⭐️
⌨️ 00:00 - Overview
⌨️ 01:33 - Create a VM on AWS EC2
⌨️ 05:28 - Install dependencies (Git, Docker)
⌨️ 08:46 - Download & install the GitLab Runner
⌨️ 09:40 - Register the GitLab Runner with GitLab
⌨️ 15:42 - Test the GitLab Runner
⌨️ 18:32 - Test Docker build and push
⌨️ 21:28 - Enable the privileged mode
⌨️ 27:07 - Conclusion & next steps
⭐️ Commands & resources mentioned ⭐️
A Brief Guide to GitLab CI Runners and Executors
/ a81b9b8bf24e
A list of the most important commands used in this tutorial:
Update dependencies:
sudo yum -y update
Install Git:
sudo yum install -y git
Install Docker:
sudo amazon-linux-extras install -y docker
Add the ec2-user user to the docker group:
sudo usermod -aG docker ec2-user
Ensure the Docker daemon starts automatically:
sudo systemctl enable docker
Restart your EC2 instance:
sudo reboot
Download the GitLab Runner installation package:
curl -LJO "gitlab-runner-downloads.s3.am..."
Install the GitLab Runner:
sudo rpm -i gitlab-runner_amd64.rpm
Register the GitLab Runner:
sudo gitlab-runner register
Edit the config.toml file:
sudo vim /etc/gitlab-runner/config.toml
The GitLab CI pipeline config used to test the runner:
gitlab.com/-/snippets/2300614
⭐️ 🎓 GitLab CI/CD ONLINE COURSE ⭐️
Want to learn more about CI/CD pipelines? Check my GitLab CI online course.
Get it at a special price and help support this KZread channel:
www.vdespa.com/courses
--
⭐️ 💬 I have a question. ⭐️
I do my best to answer all comments here on KZread but I cannot make any guarantees.
If you are asking a question, make sure to include as many details as possible.
--
⭐️ 💡 I have a video idea ⭐️
Do you want me to create a video on a specific topic? Just fill out the form below:
forms.gle/uWEzXFQ2viJtZtvZ7
--
⭐️ 📺 Learn GitLab on KZread ⭐️
Collection of video tutorials:
▸ Learn GitLab CI:
• Gitlab
___
⭐️ 📕 Free resources ⭐️
📌 Official Gitlab CI documentation
docs.gitlab.com/
📌 DevOps with Valentine
/ devops-with
--
⭐️ 📩 Contact & Imprint ⭐️
👉 Follow me on Twitter:
/ vdespa
👉 Follow me on Medium:
/ vdespa
👉 Follow me on Facebook:
/ valentin-despa-1130381...
👉 Let's connect on LinkedIn:
/ vdespa
vdespa.com/imprint
__

Пікірлер: 102

  • @brincolnar5191
    @brincolnar5191 Жыл бұрын

    Thank you, not only does it work but you also nicely explained it!

  • @vdespa

    @vdespa

    Жыл бұрын

    Glad to hear that. You are welcome. Thank you for leaving a comment.

  • @tensetv
    @tensetv Жыл бұрын

    Thank you so much man! The privilege error fix helped me a lot!

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @tonyBeloni
    @tonyBeloni Жыл бұрын

    After wasting several hours trying to get the my pipeline working - was failing on the "dind" service - this video saved the day. Thanks a lot Valentin - love your presentation skills and technical insights 👍

  • @vdespa

    @vdespa

    Жыл бұрын

    Thanks, Tony. I'm glad this saved you a few hours ;)

  • @ThomasKanzig
    @ThomasKanzig Жыл бұрын

    Really good tutorial. You helped me a lot. Thanks.

  • @vdespa

    @vdespa

    Жыл бұрын

    Glad it helped! You are welcome. Thank you for leaving a comment.

  • @shavis4811
    @shavis48113 ай бұрын

    Very clear and concise tutorial.

  • @vdespa

    @vdespa

    3 ай бұрын

    Glad you think so! Thank you for leaving a comment.

  • @manfredmann1905
    @manfredmann19052 жыл бұрын

    Thanks, It was very helpful.

  • @vdespa

    @vdespa

    2 жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @wendel1210
    @wendel1210 Жыл бұрын

    that video is absolutely gold

  • @vdespa

    @vdespa

    Жыл бұрын

    Glad you liked it. You are welcome. Thank you for leaving a comment.

  • @ARRABELLO
    @ARRABELLO Жыл бұрын

    The best vídeo of all youtube to learn CI CD !!!

  • @vdespa

    @vdespa

    Жыл бұрын

    Glad that it helped. Thank you for leaving a comment.

  • @Sparklegoat11
    @Sparklegoat1111 ай бұрын

    Great tutorial, subscribed

  • @vdespa

    @vdespa

    11 ай бұрын

    Awesome, thank you!

  • @frankreich5018
    @frankreich5018 Жыл бұрын

    Awesome video. Keep going.

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @vignesh3405
    @vignesh3405 Жыл бұрын

    Thank you valentin. I was stick in this place from past one week it made me happy. Also i liking your teaching style and showing us what happing in background with error this may helpful for every one when they follow this tutorial if they get any error like this. Error and pactice is make way for us to learn more. Please create this type of content that may help us to learn new things. I suggest to you to create docker full tutorial for beginners to get into devops practice. Thank you so much for sharing your knowledge to us.☺️

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @rohitw548
    @rohitw5482 жыл бұрын

    Your Tips and Tricks have helped me so much. as a result of which I have now Powerful automation sitting in Jenkins . You are Fabulous. God Bless You

  • @vdespa

    @vdespa

    2 жыл бұрын

    Great to hear! I am glad that it was helpful.

  • @OsascogamingBrSP
    @OsascogamingBrSP Жыл бұрын

    Thank you Valentin!

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @HaPham-mk6od
    @HaPham-mk6od2 жыл бұрын

    Thanks, It was very helpful. Still waiting for your new videos.

  • @vdespa

    @vdespa

    2 жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @GabrielPyron
    @GabrielPyron Жыл бұрын

    help me alot, I bought some of your curses and on the gitlab course I was looking for this information, thanks

  • @vdespa

    @vdespa

    Жыл бұрын

    Great to hear! You are welcome. Thank you for leaving a comment.

  • @GabrielPyron

    @GabrielPyron

    Жыл бұрын

    @@vdespa one doubt, im doing it into a internal lab using the gitlab community for study. My pipelines is getting stuck initializing the git repository. We have to manually configure the git client on runner?

  • @vdespa

    @vdespa

    Жыл бұрын

    @@GabrielPyron Yes, the runner needs to have access to Git.Is the runner on Windows or Linux?

  • @GabrielPyron

    @GabrielPyron

    Жыл бұрын

    @@vdespa thanks for reply, I found the solution. The main problem is the installed version of gitlab thats conflicts with gitlab runner. So I updated the gitlab and it back to work as expected. Thanks

  • @sohampatel5805
    @sohampatel5805 Жыл бұрын

    You made my day bro......Thanks Buddy......

  • @vdespa

    @vdespa

    Жыл бұрын

    My pleasure! You are welcome. Thank you for leaving a comment.

  • @hanrin062
    @hanrin06210 ай бұрын

    Thankyou dude

  • @vdespa

    @vdespa

    9 ай бұрын

    No problem. You are welcome. Thank you for leaving a comment.

  • @dambujopaulo4204
    @dambujopaulo42042 жыл бұрын

    Wow first to comment 🤪... Great start I think will be great tutorial

  • @vdespa

    @vdespa

    2 жыл бұрын

    Thank you so much 😁

  • @NIVI800
    @NIVI800 Жыл бұрын

    Thanks!

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @kumaresanperumal2581
    @kumaresanperumal2581 Жыл бұрын

    Thanks bro

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @hoseinhojatansari8952
    @hoseinhojatansari89522 ай бұрын

    thanks a lot..

  • @vdespa

    @vdespa

    2 ай бұрын

    Welcome 😊

  • @guidocaru
    @guidocaru Жыл бұрын

    thanks!!

  • @vdespa

    @vdespa

    Жыл бұрын

    You are welcome. Thank you for leaving a comment.

  • @user-vw6wx2np4z
    @user-vw6wx2np4z4 ай бұрын

    Thanks, it is working for me. Can you also do the same thing for deployment on EKS after building the image?

  • @marklong6572
    @marklong65722 жыл бұрын

    Thank you so much for this. What would be a production ready setup?

  • @vdespa

    @vdespa

    2 жыл бұрын

    Mark, this largely depends on your organization needs. For personal projects, this setup is enough to get you started. However, for an organization, let the IT operations team handle this.

  • @tkmania1
    @tkmania1 Жыл бұрын

    Thanks Valentin for the great tutorial. I'm a bit confused when it comes to Docker in Docker image. I see when you registered the runner, you mentioned alpine image and when you ran the test job you mentioned alpine image as well, but in build job you mentioned "docker" image and in services section you used "docker:dind". I don't understand which image is used at which stage and is the docker container ran on the gitlab runner server (EC2 instance) or inside another container. Can you please explain which image is used at which stage and which image value overrides the others and where the container is ran.

  • @vdespa

    @vdespa

    Жыл бұрын

    Check the job logs, they will tell you which image is being used. docker:dind is a service that allows you to interact with the Docker daemon. Try removing it to see what happens.

  • @usmansaeed9377
    @usmansaeed93775 ай бұрын

    Valentine, thank you for the excellent tutorial; it has been incredibly informative. However, I'm encountering an issue while attempting to execute Docker in detached mode using the command 'docker run -d ' and utilizing Docker-in-Docker as a service with the Docker executor. Although the job appears to complete successfully within the pipeline logs, I'm unable to find the running container when using the 'docker ps' command on the GitLab-runner host. Is there a method to run containers in detached mode using the Docker executor with Docker-in-Docker?

  • @yesubabupataballa4867
    @yesubabupataballa48672 жыл бұрын

    Thanks for doing this. Would you mind doing DB migrations with Sql server , flyway, and GitLab CICD pipeline?

  • @vdespa

    @vdespa

    2 жыл бұрын

    Thanks for your suggestion. I will look into it. Is there something in particular that does not work for you?

  • @joepinto5066
    @joepinto5066 Жыл бұрын

    hi, and about security groups, we should give permission to all IPs ?

  • @momopatel9054
    @momopatel90542 жыл бұрын

    Hey Valentin, These Tutorials have helped me a lot in terms of build docker image and testing it(I've also set up my runner on my ec2 instance with a docker executor), I was now wondering how would I go about deploying my react web app(which is in an docker container) from my GitLab pipeline to an ec2 instance? would I have to ssh into said instance, pull container from GitLab registry and then run the docker build? If so I am not sure how to do this. Thanks.

  • @vdespa

    @vdespa

    2 жыл бұрын

    You can deploy a static web app directly in S3 (without Docker) or using ElasticBeanstalk with Docker. I don't think you want to use EC2 directly for this.

  • @momopatel9054

    @momopatel9054

    2 жыл бұрын

    ​@@vdespa I see what if my web app had a backend? Would it still be advisable to do it on EB(Since s3 can only host static web pages)? I am just trying to understand what would be the pros or cons of using EC2 vs. the other software's. And also asked this questions since I was asked to directly deploy to EC2 instance Thanks.

  • @vdespa

    @vdespa

    2 жыл бұрын

    @@momopatel9054 EB is a managed service. If one of your container dies, EB will try to restart it. On EC2, you are the one who needs to manage everything, including software updates & co.

  • @messysoulinjapan8755
    @messysoulinjapan8755 Жыл бұрын

    Hey Valentin Thankyou very much for the simplified explanation...I was having one doubt ....What if I want to Pull Docker image from Azure Container Registry......Do I need to Specify Image path of container registry during Runner registration step and in the .gitlab.ci.yaml file

  • @vdespa

    @vdespa

    Жыл бұрын

    You need to specify that in your YAML file.

  • @jun-uen0
    @jun-uen0 Жыл бұрын

    Thank you, Valentin! This tutorial is extremely interesting and helpful for me, and my job. I have a question, how come we installed “git”? I could not find any command using git.

  • @vdespa

    @vdespa

    Жыл бұрын

    How about cloning the project files using Git?

  • @jun-uen0

    @jun-uen0

    Жыл бұрын

    @@vdespa I get it. That’s right. I was just wondering if I can remove installing git if my team doesn’t need to clone during the CI. Thank you for answering my question!

  • @momopatel9054
    @momopatel9054 Жыл бұрын

    Hey valentine was wondering if you could do a video on how to deploy a MERN Stack App on AWS ec2 Using GItLab CI

  • @erhon007
    @erhon007 Жыл бұрын

    Please show the command in terminal docker images?

  • @nitikasinghal522
    @nitikasinghal522 Жыл бұрын

    Hi Valentin, Video is very informative and helpful. But I am facing one issue. I have followed all the steps mentioned in the video and it worked for me, my pipeline started working, but once job is successful and I tried to ran another job, the runner went offline.. Can you help me how can make online again?

  • @vdespa

    @vdespa

    Жыл бұрын

    Did you try checking the status of the runner?

  • @nitikasinghal522

    @nitikasinghal522

    Жыл бұрын

    yes, I have checked the runner status and it was running. @Valentin Now I am facing some weird issue, the CI/CD job is stuck in-between after running fine for 6-7 mins, and when I go to my EC2 instance that also stops responding. Any idea why this is happening? I have checked the space on the instance, and it has enough space to run the job.

  • @allanvaldez3209

    @allanvaldez3209

    Жыл бұрын

    I got this error I'm trying from a lxd container when i change to privileged = true ERROR: Job failed (system failure): Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply caps: operation not permitted: unknown (docker.go:403:1s)

  • @Billiam112
    @Billiam112 Жыл бұрын

    Can your show something that one should do for production? You said in the beginning that one should not do this for production..

  • @vdespa

    @vdespa

    Жыл бұрын

    Good point, Bill. What "production" means largely differs from project to project. There is no single definition. However I understand the point you are making and I am considering more advanced tutorials that would be a better fit for most production situations where reliability and scalability are a must. Thanks!

  • @Billiam112

    @Billiam112

    Жыл бұрын

    @@vdespa that would be great. There are so many tutorials on stuff up to actual deployment, but then actually deploy something, perhaps a project for a small business (one might want to do some small freelancing for), how to deploy (not with heroku 😁), would be really good. 👍

  • @user-zt5rb2rd9j
    @user-zt5rb2rd9j8 ай бұрын

    created a gitlab cicd pipeline to build the docker image and push it to container registry in gitlab and when I run the pipeline it shows following error. Copy failed file not found in build context or excluded by .dockerignore:stat pkg/demo.war:file does not exist. Can u help in getting through this error

  • @febrian1546
    @febrian15462 жыл бұрын

    Can i do this with ubuntu and another cloud provider to configure gitlab runner ?

  • @vdespa

    @vdespa

    2 жыл бұрын

    Yes. The process is the same, you just need the package for Ubuntu/Debian.

  • @aniketjaa5416
    @aniketjaa54162 жыл бұрын

    Hello Sir, I have connected my GitLab with remote rdc with specific runners (tag )installed at remote windows workstations. not able to trigger scripts from .yml file on remote workstations.(windows 10) (.yml file is successfully building pipeline by passing maven command for Windows) Thanks

  • @vdespa

    @vdespa

    2 жыл бұрын

    I am sorry this is not working. What is the error you are getting?

  • @likbezpapuasov4888
    @likbezpapuasov4888 Жыл бұрын

    Can somebody here please clarify what course of actions i have to perform to realize this what i read on someone's gitlab project i want to download, compile and run. It has just 3 short lines of text: "Getting started. The first time, run these commands to download afivo, particle_core etc.: git submodule init git submodule update". Then compile the code". First obvious question immediately comes: is just downloading and using ZIPped archive with the code available there not enough as usually ?

  • @vdespa

    @vdespa

    Жыл бұрын

    Just downloading the files usually is enough. However, since you need to run additional Git commands, you need to clone this repo with git.

  • @likbezpapuasov4888

    @likbezpapuasov4888

    Жыл бұрын

    @@vdespa Thanks Valentin. But why this was done? Cloning means that i will make one more gitlab place (my own clone of original one) with absolutely not clear reason why i have to do that as i do not need to keep there anything. The original place has sources Zipped, and i just can download them without bothering with useless cloning. Additionally the commands "git submodule init, git submodule update" etc are cryptic because obviously are incomplete (they show no address from which address to which address will be performed init or update. What i do not see here?

  • @geyangxiao9666
    @geyangxiao9666 Жыл бұрын

    How do i solve this error when i input"sudo rpm -i gitlab-runner_amd64.rpm", it shows that error: gitlab-runner_amd64.rpm: not an rpm package (or package manifest):

  • @vdespa

    @vdespa

    Жыл бұрын

    Are you sure that file is not empty?

  • @highway9691
    @highway969111 ай бұрын

    Please increase terminal font size while recording

  • @vdespa

    @vdespa

    11 ай бұрын

    Thanks for raising this point. Are you watching on a mobile device?

  • @highway9691

    @highway9691

    11 ай бұрын

    @@vdespa yes i am watching on mobile, thanks for noticing my comment 🙂. Increasing terminal font will help to see it clearly

  • @vdespa

    @vdespa

    11 ай бұрын

    @@highway9691 Got it. Thanks for letting me know.

  • @kritsatornsaengweang715
    @kritsatornsaengweang715Ай бұрын

    g8 video. Like + Sub here !

  • @vdespa

    @vdespa

    Ай бұрын

    Thank you!

  • @dongpham7789
    @dongpham77892 жыл бұрын

    I ran the command "sudo rpm -i gitlab-runner_amd64.rpm" then got the following error messages: error: Failed dependencies: gitlab-ci-multi-runner conflicts with gitlab-runner-15.0.0-1.x86_64 gitlab-ci-multi-runner conflicts with (installed) gitlab-runner-15.0.0-1.x86_64 Please help, I did not how to fix it. Thanks in advanced

  • @vdespa

    @vdespa

    2 жыл бұрын

    It seems that it is already installed.

  • @dongpham7789

    @dongpham7789

    2 жыл бұрын

    @@vdespa Thanks Valentin, do I need to create one EC2 instance for each project?

  • @vdespa

    @vdespa

    2 жыл бұрын

    @@dongpham7789 No. This tutorial shows you how to create a group runner. In a group you can have multiple projects.

  • @pipe201196
    @pipe2011967 ай бұрын

    Where is push u docker image? Dockerhub?

  • @vdespa

    @vdespa

    4 ай бұрын

    Sorry for the late reply. You can push to AWS, Dockerhub or GitLab.

  • @QuangPham-bc7lc
    @QuangPham-bc7lc8 ай бұрын

    i set up an vm in GCP and do everything like you but when run pipline, it show "This job has not started yet This job is in pending state and is waiting to be picked by a runner"

  • @vdespa

    @vdespa

    8 ай бұрын

    Did you configure a runner?

  • @QuangPham-bc7lc

    @QuangPham-bc7lc

    8 ай бұрын

    @@vdespa yes, i have registered my vm, set correct tags, give gitlab-runner user sudoer permission, install docker, check status of gitlab-runner and docker service but i still not work after i type "sudo reboot" in that vm - vm on GCP

  • @vdespa

    @vdespa

    8 ай бұрын

    @@QuangPham-bc7lc did you configure it to run jobs without tags?

  • @QuangPham-bc7lc

    @QuangPham-bc7lc

    8 ай бұрын

    yes @@vdespa