Automate data collection with GCP Cloud Functions
Introduction In the previous post, we have demonstrate steps to build an Airflow ELT data pipeline for crime…
Dark
Light
Build a data pipeline with Airflow and GCP
Using London crime data to create an automatic end-to-end pipeline.
Transforming data in BigQuery
We will go through three different transformations on a dataset in BigQuery. Filter for unique data Using REGEX_CONTAINS…
Running BigQuery DAG in Airflow locally with Docker
Before we deploy new DAG to production, it’s best practice to test it out locally to spot any…
Using pandas_gbq to import dataframe to BigQuery
Given data in a text file which is the format of a python dictionary, we can first turn…
How do I prepared and passed the Google Cloud Professional Data Engineer Certification Exam
As a data analyst who wants to up-skill myself and hopefully become a data engineer, I figure that…
London crime rate study with BigQuery and Data Studio
Finding the safest neighborhood in London with DataStudio 1. Getting our data and process it Following step 1…
London crime rate study with GeoViz
Try to find the safest neighborhood in London. 1. Get crime raw data from London Datastore No better…
Setup Cloud Spanner database and exporting it
Cloud spanner is a transactional database that designed for cloud with horizontal and vertical scaling capability. We will…
Streaming data to BigQuery with Pub/Sub
In order to create a streaming data pipeline in Google Cloud Platform, we are going to make use…