RedditPP Mascot

RedditPP

r/dataengineering • … • u/PaddyAlton

BigQuery External Queries + DBT = instant pipeline

I had a nice reaction to posting my articles on data modelling last week, so I've tidied up and posted one that's been sitting in my drafts for a while.

This time it's about [an approach I've been using to bring data from GCP Cloud SQL into BigQuery]( https://medium.com/apolitical-engineering/how-we-use-dbt-and-bigquery-external-connections-to-easily-and-reliably-warehouse-cloud-sql-data-b0f68e873aad).

I admit it's a trifle hacky, but if your needs are simple (small-ish tables, no need for live data or an exact history of changes) then it might help you get up-and-running quickly! At the end of the article I discuss some of the tradeoffs and potential improvements.

Anyway, hope it's of interest.

— 0