BigQuery Data Eng Basics Series Data Engineering

Automate a daily ETL of Currency Rates into BigQuery

Step by Step Guide – how to automate adding currency rates to BigQuery via a basic ETL flow. 1. Signup to the free plan from OpenExchangeRates Generate an ‘App ID’. This will let you get the latest rates 1000 times per month for free. 2. Create a Dataset in BigQuery to hold the data called ‘currency_rates’ 3. Create a BigQuery Table called “openexchangerates_responses” to Hold all the API responses We will need a “data” column…

Continue reading

For Agencies

Best way to share DataStudio reports with your clients

When you’re an Agency making DataStudio reports for clients you generally have the same problems: The Problems: Organising a group of Reports together for a Client (the standard DataStudio list isn’t ideal) Making sure the client is using the latest version of each Report (especially if each Report has multiple revisions) Knowing who has looked at a Report, and when (in Realtime so you can follow-up for feedback) Controlling access to a group of Reports…

Continue reading

BigQuery

Monitoring you BigQuery costs per Query and per User

aka – Cutting costs without Cutting value while upskilling your team The Problem We love BigQuery. It’s easy to setup and maintain, meaning anyone with basic SQL skills (or can copy paste from StackOverflow) can get value super quick.  I love seeing the effect it has on businesses who become liberated once they migrate from another platform where they are constrained by query speed (MySQL/Postgres), query language (Mongo) or maintenance and tooling (Redshift). Access to…

Continue reading

Dev-Log

Scratching our own Itch

Why did we make this? With any project, there’s so many other competing priorities it’s important to be clear with the “why” – “why” are we spending time on this, “why” does it help other people, and “why” does it help us? We make a lot of Reports for Clients with a variety of tools. Google DataStudio, Power BI, Tableau, Looker, Metabase, Retool – even data centric custom apps in Vue or React. The problems…

Continue reading