Automated Database Backups for Cloud Spanner Using Terraform

terraform

    This post shows how to use cloudspannerecosystem/scheduled-backups to configure scheduled backups for your cloud spanner database. The setup of the required resources (Cloud Scheduler, pub/sub and cloud function) will be done by using terraform in this tutorial.  …

Read More >

Orchestrating dbt Pipelines with Google Cloud: Part 2

dbt pipelines

In part 1, we defined and deployed two data services to Cloud Run. Each service provides endpoints that perform specific tasks, such as loading a file to BigQuery or running dbt models. In this post, we’ll define and deploy some…

Read More >

Orchestrating dbt Pipelines With Google Cloud: Part 1

orchestra

In my previous post I showed you how to use dbt to expedite data preparation tasks on Google BigQuery. This time, I’ll show you how to integrate those dbt pipelines into workflows that load, validate and transform data.    …

Read More >

SQL Server AlwaysOn Availability Groups on Google Kubernetes Engine

Infrastructure Design

This tutorial shows you how to configure SQL Server AlwaysOn availability groups (AG) using DH2i DxEnterprise and Google Kubernetes Engine (GKE). This offering helps you achieve high availability for your SQL Server databases running on containers in Kubernetes. GCP Resources…

Read More >

5 Ways to Maximize Your Google Cloud Next 2021 Experience

google next 2021 banner

Google Cloud Next. My favorite annual event. Making new connections, reconnecting with those I haven’t seen for a year, and learning about the continuous innovation at Google generates so much energy inspiration. And despite the virtual format again this year,…

Read More >

Caching Alternatives in Google Dataflow: Avoiding Quota Limits and Improving Performance

The problem When building data pipelines, it’s very common to require an external API call to enrich, validate or obfuscate data using external services. This might happen with streaming or batch pipeline. The situation is the same: call external services…

Read More >

Using El Carro Operator on AWS

Introduction Google recently released a Kubernetes operator for Oracle, El Carro. The project includes good examples of how it works on GCP and on your local computer (using minikube). Given it is a portable implementation, we wanted to give it…

Read More >

Data preparation with dbt and BigQuery

Raw incoming data needs to go through a series of data preparation steps before it can be used for analysis. These steps include tasks such as type casting, renaming columns, cleaning values and identifying duplicates. Writing code to perform these…

Read More >

Near Real-Time Data Processing for BigQuery: Part Two

This post is part two of describing (near) real-time data processing for BigQuery. In this post, I will use Dataform to implement transforms as well as ASSERTS on the data and unit testing of BigQuery code and SQL statements. Part…

Read More >

Near Real-Time Data Processing for BigQuery: Part One

This post describes (near) real-time data processing for BigQuery with unique and other check constraints, and unit testing. This is part one of two, and describes the real-time ingestion of the data. Part two will describe how to implement ASSERTS…

Read More >
Page 1 of 512345