Scheduling Google Cloud Functions

Posted in: Big Data, Cloud, Google Cloud Platform, Technical Track

Currently, there is no straightforward way to schedule Google Cloud Functions.

It is still possible to achieve this by different means, such as (but not limited to):

  • deploying Compute Engine instance and setting crontab entry
  • configuring HTTP/S uptime checks via Stackdriver Monitoring
  • deploying App Engine cron job

Question mays arise: why would one need to do this at all? There may be different use cases, for example, pulling data from external sources periodically, or triggering internal pipelines based on the schedule, etc.

Here’s a summary of the options mentioned above:

Compute Engine instance

Deploying separate Compute Engine instance just for this does not sound right. Also, to achieve High Availability, multiple instances have to be deployed, leading to each instance scheduling tasks independently of the others.

Stackdriver Monitoring uptime check

Stackdriver Monitoring uptime checks are performed from all available locations, by default. There is a way to limit checks by one or more location, but some of them, US for example, have multiple regions from where the checks are performed. Again, the function will be iinvoked more often than required.

App Engine cron job

App Engine cron job, on the other side, is one of the recommended ways to implement reliable cron functionality.

It comes with its own limitations, such as App Engine cron job can only invoke App Engine endpoints, so an application needs to be deployed as well. By default, it will be deployed in High Availability setup, having 2 instances running.

Implementing Cloud Function scheduling via App Engine cron job

When following instructions assume there is a Google Cloud Platform project named test-gcf-scheduling and service account named deployment with Project Owner permissions created, and service account JSON key has been downloaded under deployment-key.json.

Also, sample code repository has been cloned into current working directory:

git clone https://github.com/ilgarm/test-gcf-scheduling.git
cd test-gcf-scheduling

Authorize service account first and enable required services:

gcloud config set project test-gcf-scheduling
gcloud auth activate-service-account --key-file=test-gcf-scheduling-key.json

gcloud services enable appengine
gcloud services enable cloudfunctions

Cloud Function

Next, we could create staging bucket to upload functions to, otherwise, one will be created by the service.

gsutil mb -c regional -l us-central1 gs://gcf-test-scheduling-staging
cat > gcf-test-scheduling-staging-lifecycle.json << EOL
{
  "rule":
  [
    {
      "action": {"type": "Delete"},
      "condition": {"age": 7}
    }
  ]
}
EOL
gsutil lifecycle set gcf-test-scheduling-staging-lifecycle.json gs://gcf-test-scheduling-staging

To deploy the function:

gcloud beta functions deploy scheduled --source=gcf/scheduled-function --stage-bucket gcf-test-scheduling-staging --memory=128MB --region=us-central1 --trigger-http

A few notes on the sample function. As the function is exposed via HTTP/S it might be important to password protect it, to avoid unexpected costs incurred by unauthorized invocation, especially, if function triggers some heavy processing.

App Engine application

Next, App Engine application needs to be deployed. The current version is deployed to the standard python27 environment with manual scaling of 1 instance, to avoid unnecessary costs. Single B1 instance feets into Free Tier also.

The code is straightforward, the only important thing to mention here is the following section:

if self.request.headers.get('X-Appengine-Cron', False) != "true":
  raise ValueError('Not invoked from the task queue job')

The header is injected into the request if originated from the App Engine cron job. For other requests, the header is stripped out if present. This should be enough to ensure that the request is issued by our cron job.

To deploy to App Engine:

gcloud app deploy gae/scheduler-app/app.yaml
gcloud app deploy gae/scheduler-app/cron.yaml

Now, navigate to the Cloud Functions logs, there should be log entries for the function invocation appearing every minute.

This example could also be extended to trigger multiple functions on different schedules, implement crontab expression evaluation logic and so on.

email

Interested in working with IIgar? Schedule a tech call.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *