Automate Python script execution on GCP

This tutorial shows how to automate Python script execution on GCP with Cloud Functions, Pub/Sub and Cloud Scheduler.
Link to the GitHub repo with code from this tutorial: github.com/rafaello9472/c4ds/...
00:00 Introduction
00:21 Architecture overview
01:09 GUI - Pub/Sub
01:28 GUI - Cloud Functions
02:39 Python code walkthrough
05:18 GUI - Cloud Scheduler
08:27 gcloud CLI - Pub/Sub
09:33 gcloud CLI - Cloud Functions
11:31 gcloud CLI - Cloud Scheduler

Пікірлер: 32

  • @marianklose1197
    @marianklose11978 ай бұрын

    Really interesting, thank you!

  • @user-ok3to6kg3j
    @user-ok3to6kg3j3 ай бұрын

    Please answer to my question, I need to do the same thing as you did in this video. My python script works just fine under Google cloud shell. However, I am still having trouble making it work as cloud function. The purpose is to schedule the execution of the function. It consist of extracting a data from a web site and save in google sheet. I was able to make run it under google cloud shell. Any clue from you ?

  • @AlexBurov-wk8my
    @AlexBurov-wk8my9 ай бұрын

    Awesome. Thank you

  • @jsuhwork1970
    @jsuhwork1970 Жыл бұрын

    great tutorial!

  • @cloud4datascience772

    @cloud4datascience772

    Жыл бұрын

    Thank you!

  • @marcelomaia4274
    @marcelomaia427410 ай бұрын

    Excelent!

  • @fafenley00
    @fafenley009 ай бұрын

    Great tutorial. Wondering - why save the data for versioning as csv and not parquet? Wouldn't that allow you to create easy partitions, save on storage, and expedite processing/ querying?

  • @cloud4datascience772

    @cloud4datascience772

    9 ай бұрын

    The csv format was just for the demonstration purposes. You are absolutely right that there are different file formats which could be better for this task!

  • @umamaheshmeka1032
    @umamaheshmeka10323 ай бұрын

    Great job! I followed your instructions, and everything started working smoothly for me. Your tutorial is fantastic - keep up the excellent work! You have the potential to reach 1 million subscribers. Keep pushing forward !!!

  • @cloud4datascience772

    @cloud4datascience772

    3 ай бұрын

    That is great to hear! Thank you for the kind words :)

  • @user-ze3ll5rf9h
    @user-ze3ll5rf9h8 ай бұрын

    thank you for the tutorial. Is there an easy way to replace the .py file with a .ipynb file? what are the changes that need to be done?

  • @GTFS-lg7tw
    @GTFS-lg7tw Жыл бұрын

    We’re does the script get its internet from to load URLs/api when it’s in the cloud? Is it generated or from a mast in the providers grounds?

  • @cloud4datascience772

    @cloud4datascience772

    Жыл бұрын

    Internet is provided by the Google Cloud Platform, no need to do anything additional once the script is successfully deployed to Cloud Functions.

  • @freddiemarrero6512
    @freddiemarrero651210 ай бұрын

    Very helpful tutorial. Can you do the GUI code for Environment version 2 for the function?

  • @cloud4datascience772

    @cloud4datascience772

    10 ай бұрын

    Hey, I am not quite sure what are you asking about, could you clarify your question?

  • @freddiemarrero6512

    @freddiemarrero6512

    10 ай бұрын

    @@cloud4datascience772 thanks for your response. You're using the first generation environment for this example. I was able to replicate it fine. But I'm trying to deploy a new version on the 2nd generation environment to have more control of the resources for my function and the code did not worked. So I figured out that for 2nd generation environment there should be some change to the code. Once again thanks for this tutorial!

  • @cloud4datascience772

    @cloud4datascience772

    10 ай бұрын

    @@freddiemarrero6512 I see, from my experience there should be minor changes to the code for the 2nd gen. I am not planning to cover this in my tutorials in the nearest future so I can only recommend the official documentation on that topic: cloud.google.com/sdk/gcloud/reference/functions/deploy or cloud.google.com/functions/docs/create-deploy-gcloud Good luck!

  • @victorricardo8482
    @victorricardo84824 ай бұрын

    Hi! First of all, great video! Really simple and intuitive. It worked for me when I called only one function inside the hello_pubsub, but when I tried to call several others, from others .py, looked like the function run perfectly but with no results. Is there a way to make the cloud functions wait before every function finishes before moving to the next one? Thanks

  • @victorricardo8482

    @victorricardo8482

    4 ай бұрын

    In other words, I need the second function to get the result from the first, and so on. Thats why I need it to wait

  • @user-lz7uh4lh7c
    @user-lz7uh4lh7c8 ай бұрын

    Thanks for the detailed explanation. I try to follow the first method but am facing some issues. Basically, I could load the data into Cloud Storage but when I try to load it to BigQuery, I received 404 error saying reqeust could not be served. Any ideas why it happens. FYI in case it helps. I have modified the python script a bit using local environment os which I could load data into cloud storage and then upload into BigQuery successfully through my laptop. However, when I try to automate it in Google Cloud Function, I faced the 404 issues. Any comment would be appreciated.

  • @cloud4datascience772

    @cloud4datascience772

    8 ай бұрын

    Thanks for your comment, as a first thing you can check if the location of the services are all within the same region (e.g. all services in the EU).

  • @flosrv3194
    @flosrv3194Ай бұрын

    what is the zip file ? i didnt understant what is it about..

  • @user-xu3yi6vd1j
    @user-xu3yi6vd1j3 ай бұрын

    Can we do this using google compute like GPU? how can we do that?

  • @cloud4datascience772

    @cloud4datascience772

    3 ай бұрын

    There is no easy direct way to do it with Cloud Functions. For GPU you would need to use Google Compute Engine and select GPU machine type, but process of automating some script execution would be much different, and it's not covered in my tutorial.

  • @Rajdeep6452
    @Rajdeep64523 ай бұрын

    Lol it’s not working. Can’t deploy. What can be the problem? I did exactly what you did, except can’t create the same bucket so named the bucket as c4ds1. Also changed the code for that.

  • @nguyenvantruong8141
    @nguyenvantruong81418 ай бұрын

    Is it free?

  • @cloud4datascience772

    @cloud4datascience772

    8 ай бұрын

    There is a free tier on GCP, a lot of services that I am using in this video are available for free up to certain level of usage: cloud.google.com/free Generally running the workload with Cloud Functions should be cost effective solution.

  • @nguyenvantruong8141

    @nguyenvantruong8141

    8 ай бұрын

    @@cloud4datascience772 but I have to pay 300 dollars in advance and I'm just a student so I don't have it

  • @rogerrendon8976
    @rogerrendon89769 ай бұрын

    Hi This tutorial is great but I encountered one problem while runing the function: google.api_core.exceptions.NotFound: 404 Not found: Table arya-teste:arya_dataset.arya-leads was not found in location southamerica-east1" any clue how to solve this?

  • @cloud4datascience772

    @cloud4datascience772

    9 ай бұрын

    Hey, are you sure that your dataset exists at the time of running the function?

  • @rogerrendon8976

    @rogerrendon8976

    9 ай бұрын

    Yes I think it was another temporarly error. @@cloud4datascience772