Mortimer XYZ Blog

Vibe coding a game in 2026 as a data engineer

Thu, 11 Jun 2026 00:00:00 GMT

The Ceiling of my vibes: Building a Game with LLMs when you are not a game developer

If you read my other blog posts, or know me well enough you know I am not a game developer. In my day job I’m building data pipelines. Recently, I decided to build a kids’ game (well my daughter decided I would).

“You write computer programs don’t you Daddy?“.

The experience certainly wasn’t pure magic so I thought I would share how far I pushed “vibe coding” and some things that plain didn’t work (at least for me).

The result is live at https://mortie23.github.io/blockvillie/.

Maybe I wasn’t using the right model, maybe I wasn’t paying for enough tokens or using the best method to vibe. This isn’t a tutorial on how to prompt a game. It’s just me telling me story of what happened when I tried to vibe my way through a genre of programming that I’ve never touched before, in early 2026.

My initial architecture

I know enough about web development and hosting web sites and applications so to ensure I at least had some idea of what the LMM generated I decided to have the AI build a web based game. If I didn’t know the tech stack at all I thought the complexity would have killed my momentum.

I also wanted to be able to easily and cheaply host the final product and share it with friends. So I decided to get the AI to build the game as a static web site with all logic client side in JavaScript.

Zero Backend: No databases, no authentication servers, no API Gateways. If I introduced a server, I’d have running costs, maintenance, and other overhead that would have been no value.
No Real State Management: Everything runs entirely in memory within the browser session. If you refresh, it resets. For a kids’ game that only my kids would play, this is perfect.

The AI vibe developer tooling

I used Google’s Antigravity to build the game. At the time, it included these models.

The quotas on usage of each of them were different. I was in no rush, so sometimes when I ran out of quota, I just paused the development for a few days until it refreshed.

What I found (and also took advice from others) is that you can save money/token quota but being selective on what model you choose for different tasks.

For example, for the initial scaffolding and set-up of the core game, including:

web libraries/frameworks
developer tooling
game mechanics
data structures

I used the heavy, expensive models.

Once that was all built and I had a development site working, I switched to the cheaper, faster models for new features or bug fixes.

If a major feature or refactor was required, I switched back to the more expensive models.

Getting started

A reality check on where a prompt like the following will get us in a single shot:

Generate a grid based game. Use Vite and REACT.

The player can move a character on the grid.

The theme is Lego, for UI elements such as characters and obstacles use lego based images.

Controls are with keyboard arrows and movement is to adjacent grid tiles from where the character is. The character can pickup gems and they collect in a counter.

There is a shop where the character can buy outfits with the gems.

The character can also pickup outfits and chose to wear or put them in a closet.

The grid can contain walkable tiles and non-walkable tiles. Tiles can be grass, paving, walls, sand.

Antigravity (and most vibe coding IDEs) will take from a prompt like this to an “Implementation plan”.

It gives the “vibe developer” the opportunity to add some more context. Let’s provide this extra context.

User Review Required, IMPORTANT:

Use CSS for the grid tiles (grass, paving, sand, walls) and for gems. Only generate images for characters and outfits.

Open Questions, NOTE:

Maps should be statically defined. The user should be able to select from many maps. Start with two maps, a park and a house. Maps should be a json array of values representing tile types.
Initial specific outfits should be school uniform and pyjamas
Initial grids should start with 40x40

So what did this get us?

Amazing we got something bug free. But, gems are aqua coloured balls, and outfits are red squares. The characters are bounded by grey square backgrounds. And if you could play the game, the character movement is not smooth at all.

The shop UI is ok, but we are buying completely new characters, not outfits.

The closet is not populated with outfits, but with characters.

Where the vibe stalled, the major issues

This single shot prompt to working game is actually amazing. Even in the early days of LLMs this wasn’t possible at all. A total bug free game with one prompt.

But my daughters weren’t happy at all.

“Daddy, I thought it would look like Minecraft”

These are the types of comments from my daughter after the first iteration. It needed refinement to be something she would even play once.

Map generation

The first major wall I hit was map generation. I wanted maps that were sensible layouts, in fact my daughter wanted the maps to represented real spaces she explores in real life. Firstly, left to its own devices, the maps it generated were incredibly boring. I attempted to upload screenshot from Google maps, or house plans and ask it to generate a map based on the image, these were even worse.

To get past this, I stopped asking the AI to build the maps. Instead, I used it to build an Age of Empires-style Level Creator directly inside the game space. I shifted from trying to automate the content to automating the creative tools. Once the level creator interface was functional, I could sit down and design the maps myself, bringing human intention to the grid.

One tool for converting images to the JSON array of tile identifiers.

One app for manually editing the tiles, and adding objects and gems (diamonds).

Image generation

This required a more involved process of generating images with Gemini’s image generator, however the raw images generate were inherently messy.

Every asset went through Photoshop. I had to manually:

mask out backgrounds
clean up edges
fix alignment artifacts

Then I could export clean, transparent .png files that would map perfectly to a coordinate system.

Game play

An LLM does not understand what makes a game engaging for a child. It doesn’t know what triggers a sense of reward, what is too difficult, or what causes frustration. I brought in the ultimate user experience consultant: a real child.

Every mechanic, win condition, and level progression rule came from my daughters. I watched how they interacted with the screen. When they got bored, we changed the rules. When a level completion didn’t feel rewarding, we adjusted the logic.

So we ended up with a cinema map with popcorn you can collect.

And a school level with interactions such as a chess game where the pieces move.

The 2026 Reality Check for me

Being an entrepreneur or building software in 2026 seems like it has never been more accessible. The barrier to technical execution has dropped through the floor.

But the reality is that launching something meaningful is just as difficult as it has ever been. Why? Because while code generation is cheap, human attention and wallets are scarcer, more competitive, and more stressed than ever. You cannot prompt your way into someone’s attention span.

Conclusion: The Pure Joy of Building

I have zero delusions about Blockvillie. I’m not launching a startup, I’m not trying to make money, and I don’t even expect my daughters to play it. Real games require immense depth, complex optimization, and narrative layers that go far beyond a few nights of vibe coding up a frontend experiment.

Maybe this is the point. The value I got out of vibe coding wasn’t achieving the myth of the automated product to start a business. It’s the joy of learning. Diving into a genre of programming I had zero experience with. Growing from raw prompts into an interactive experience you can touch, play with, and share.

The code might be generated, but the creative fulfillment is entirely human.

If databases were rappers (thoughts from a Generative AI)

Tue, 17 Mar 2026 12:00:00 GMT

If databases were rappers (thoughts from a Generative AI)

Can you relate to these?

Following from my previous posts where we explored what generative AI thinks cloud platforms and programming languages would be like if they were people, I decided to do something similar but a bit different.

As a Hip-hop fan, I know each rapper has a certain aura about them and a history of success and failures. They have periods of becoming “pop” or periods of being underground. I asked a Generative AI to bridge the gap between my world as a hip-hop fan and my professional world in data. Here is what it thinks of 90s databases as rappers.

1. Oracle is The Notorious B.I.G.

The Persona: The King of New York (and the Enterprise). Massive, expensive, and radiating “Big Poppa” energy.

The Tech Parallel: In the 90s, Oracle was the undisputed heavyweight. It was the most powerful and flamboyant database on the market. However, like Biggie, it was synonymous with “Mo Money Mo Problems”—if you wanted that top-tier performance, you had to pay the “Biggie” tax. Running on Oracle was a status symbol for any 90s corporation.

2. IBM DB2 is Nas

The Persona: The “Illmatic” lyricist. Deeply technical, incredibly dense, and respected by the most hardcore purists.

The Tech Parallel: While Oracle was the “pop” star of the enterprise world, DB2 was for the technical elites. Built on the research of Codd and IBM labs, it prioritised the “purest” form of the craft. Much like Nas’s complex internal rhyme schemes, DB2’s indexing and optimization engines were a masterclass in complexity. It didn’t need the flash; the performance bars spoke for themselves.

Nas in the 2020s is exactly like IBM DB2 in the modern enterprise: a foundation laid in the 90s that everyone assumed would be replaced by ‘newer, shinier’ things, yet here it is, still running the most critical workloads on the planet.

3. SQL Server is Ice Cube

The “Group” Era (N.W.A / Sybase): Just as Ice Cube started in N.W.A, SQL Server started its life as a port of Sybase SQL Server. Microsoft was the “lyrical contributor” that eventually went solo to build its own empire.

Ice Cube recognized that his original fanbase from the early 90s had grown up and were now parents themselves. He wanted to make movies that his own children and his fans’ children could enjoy. Like Ice Cube, SQL Server became the “everyman’s” database, and family name.

4. Microsoft Access is MC Hammer

The Persona: The massive crossover hit that everyone knew, even if the “hardcore” heads didn’t respect it as much.

The Tech Parallel: “U Can’t Touch This” (quite literally, because the .mdb file is usually locked by another user). Access brought data to the masses, putting it on every desktop in the mid-90s. It wasn’t a “hardcore” enterprise engine, but it was flashy, ubiquitous, and sold millions.

Since you made it this far, you deserve a bonus AI generated song by MS Hammer. Enjoy!

For all those database people who also appreciate hip-hop music (who knows how many of us there are?), do these resonate with your instincts? Or is this just a hallucination?

Shout outs

An encore performance of this blog post might be due with my friend Jacob requesting artists like Redis Man and Postgres Malone be covered.

Implementing a component based design system in PowerBI

Wed, 11 Feb 2026 12:00:00 GMT

Implementing a Component-Based Design System in Power BI

Standard Power BI headers, buttons, and callouts often feel disconnected from an organization’s broader web presence. If you are building reports for a large enterprise, especially if it is going to be public facing, following a design system is essential for a cohesive user experience.

In this post, I’ll show you how to move away from manually drawing rectangles and text boxes in Power BI and instead move toward a component-based architecture using a database, HTML/CSS and the ‘HTML Content (Lite)’ visual.

This post is based on the repository at: https://github.com/mortie23/powerbi-web-components

Why use a Database for Design?

Hard-coding hex codes and font sizes into every Power BI report is a maintenance nightmare. By storing our UI components in a database table, we gain:

Centralized Control: Update a brand color in one SQL table, and all reports refresh to match.
High Fidelity: Recreate complex CSS layouts that the native Power BI UI simply can’t do.
Consistency: Ensure every report uses the exact same design System header and padding.

The Architecture

Postgres: Acts as the “Source of Truth” for our HTML snippets, in an enterprise this might be your data warehouse.
Power BI Semantic Model: Fetches the HTML strings via a standard SQL connection.
DAX: Dynamically selects the right component based on the report page or context, replacing templated variables where needed.
HTML Content Visual: Renders the code directly on the canvas.

Setting up the Postgres Store

First, we create a schema to house our design assets.

-- Create a schema for your design system components
create schema if not exists design_system
;

-- Create the reference table
create table
  design_system.reference_web_component (
    component_id serial primary key
  , component_name varchar(100) not null
  , component_html text not null
  , last_updated timestamp default current_timestamp
  )
;

After we run this we see the table created in Postgres.

Now in PowerBI we need to get this data (along with our NFL data to create some visuals for the demonstration).

Example components

We are going to see if we can create Power BI reports that fit into the Gold Design System (AU) (formerly the DTA system) design system:

https://gold.designsystemau.org/, and the associated https://github.com/designsystemau/gold-design-system

We are going to change it a little to make it fit the common navigation of Power BI dashboards. We have a few options for how we add styles.

Each component is self contained with all styles
Host styles on a separate web server and import

For this example, for simplicity of architecture we will keep the styles self contained.

<html>

<style>
  body {
    margin: 0;
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
    background: #f5f5f5;
  }

  .au-header {
    background: linear-gradient(to right, #313131, #585858);
    color: #fff;
    padding: 1.5rem 1rem;
    border-bottom: 1px solid rgba(255, 255, 255, 0.1);
  }

  .au-header__heading {
    max-width: 1200px;
    margin: 0 auto;
    font-size: 2rem;
    font-weight: 700;
    line-height: 1.25;
  }

  .header__badge {
    display: inline-block;
    font-size: 0.875rem;
    border: 1px solid rgba(255, 255, 255, 0.5);
    border-radius: 2rem;
    padding: 2px 10px;
    margin-left: 8px;
    vertical-align: middle;
    text-transform: lowercase;
    letter-spacing: 0.05em;
  }

  @media (max-width: 768px) {
    .au-header__heading {
      font-size: 1.5rem;
    }
  }
</style>

<header class="au-header" role="banner">
  <h1 class="au-header__heading">
    {{TITLE}} <span class="header__badge">beta</span>
  </h1>
</header>

</html>

Note the inclusion of {{TITLE}} within the H1 tag. This will be utilized later in conjunction with DAX to make the component reusable.

Let’s quickly view this component in a browser to see that it is rendering how we would like. For this I simple use the Live Server extension in VScode.

Great, this is just what we want. Now, to insert this into our Postgres table:

insert into
  design_system.reference_web_component (component_name, component_html)
values
  (
    'Gold Header'
  , '<html>
<style>
    body {
        margin: 0;
...
</html>
'
)
;

See we have given the component a name in the table, we will use this later in the DAX.

Use of the HTML components in Power BI

We need to use the HTML Content (Lite) visualization from the Viz store.

Now let’s render out some of our components.

The HTML Content (Lite) visual renders the components nicely. But now it is time to make use of the templating we built into the components. We create a new measure that replaces the {{TITLE}} with our own custom title for each page.

TitleBar = 
   
   VAR BaseHTML = LOOKUPVALUE('design_system reference_web_component'[component_html], 'design_system reference_web_component'[component_name], "Gold Header")
   VAR Title = "NFL Quarterbacks (2014)"
   
   RETURN
   SUBSTITUTE(
         BaseHTML, 
         "{{TITLE}}", Title
       )

Now let’s set the visuals to align with the colour accents in the visuals.

Now, let’s embed this in our public facing website.

Power BI PBIP Git version control

Sat, 31 Jan 2026 12:00:00 GMT

How to commit a Power BI report to Git

The files from this blog post can be accessed here: https://github.com/mortie23/pbip-nfl/pull/1/changes

What is the problem

In the past, a PBIX file was all we had for saving a PowerBI report. Under the hood, it is a series of text files archived into a single file, but tracking these was cumbersome (see Power BI PBIX Git version control).

It has been a while since I was seriously working on solutions with Power BI and it seems a lot has changed.

Power BI June 2023 Feature Summary

In June 2023 Microsoft announced the PBIP format:

Power BI Desktop Developer Mode

Microsoft Power BI Updates Blog November 2025

In November 2025, this type of format became the default for the PowerBI Service.

PBIR will become the default Power BI Report Format – Get Ready for the Transition

No longer do developers who want to track changes to their Power BI reports need to have complex Git hooks and scripts to expand and archive back the files to and from PBIX. We now have PBIR and PBIP formats.

How have I found using it?

It is a super welcome change. Exactly what we have been needing in the Power BI development community.

Example

First let’s get some data. I have a GCP Compute Engine VM in which I installed a Postgres.

This is just because it is easier for me to source data this way than any other way (including Excel files)

I’m using a Windows laptop, so to make the server address simple I’ll add an entry to the hosts file. Using vim from PowerShell in a terminal opened with Administrator rights.

vim C:\Windows\System32\drivers\etc\hosts

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# ...

# Postgres on GCP
<ip-address> thefog

Now, in PowerBI let’s connect to this Postgres database an import our tables.

I’m going to create a really simple report just to demonstrate the PBIP. A single bar chart with total touchdowns by player.

Save as PBIP

git add .
git status

Look at all the files that were created. Tremendous. Lots of things to explore and edit.

On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   .gitignore
        new file:   pbip-nfl.Report/.platform
        new file:   pbip-nfl.Report/StaticResources/SharedResources/BaseThemes/CY25SU12.json
        new file:   pbip-nfl.Report/definition.pbir
        new file:   pbip-nfl.Report/definition/pages/92dc9803368342374bcd/page.json
        new file:   pbip-nfl.Report/definition/pages/92dc9803368342374bcd/visuals/287fab4078bc01b5a7ad/visual.json
        new file:   pbip-nfl.Report/definition/pages/pages.json
        new file:   pbip-nfl.Report/definition/report.json
        new file:   pbip-nfl.Report/definition/version.json
        new file:   pbip-nfl.SemanticModel/.pbi/editorSettings.json
        new file:   pbip-nfl.SemanticModel/.platform
        new file:   pbip-nfl.SemanticModel/definition.pbism
        new file:   pbip-nfl.SemanticModel/definition/cultures/en-AU.tmdl
        new file:   pbip-nfl.SemanticModel/definition/database.tmdl
        new file:   pbip-nfl.SemanticModel/definition/model.tmdl
        new file:   pbip-nfl.SemanticModel/definition/relationships.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/DateTableTemplate_773ad8ec-7f97-41c8-b011-1889725e1c74.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_369ec58b-5270-409d-b6c5-f495eab467bb.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_52994651-a681-4c39-b4a3-e5451d63e1ba.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_80eb41b8-b599-42df-8492-fa5f11fe19ff.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_9e34a27d-0d33-4a72-87b6-8463bf2c84ca.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_ac434e9a-c556-4da1-90c7-bfcccc08448a.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_baf0fdc5-35be-4651-95e7-04e98cc5941a.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_c0e78857-80b9-451f-ab97-b4c962f02c0c.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/LocalDateTable_e8be3960-bc06-4306-bc8d-e471476af1cd.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl game.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl game_stats.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl game_type.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl game_venue.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl player.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl team_lookup.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl venue.tmdl
        new file:   pbip-nfl.SemanticModel/definition/tables/mortimer_nfl weather.tmdl
        new file:   pbip-nfl.pbip

Now let’s make a change to the basic report we have created.

We can clearly see the diff is only to the visual.json file and is an addition of an objects properties adding a color from a theme.

Some tips

VScode

For working on the files you will want to use VSCode and install the TMDL extension. When I am developing a PowerBI report I have both PowerBI Desktop and VSCode open so I can monitor how changes made in the desktop change the PBIP files.

Now you get some tremendous syntax highlighting in the TMDL for each of your tables in your data model. In this example we can see the weather table’s TMDL, including the Postgres source it is coming from.

For bulk changes, or changes that are easier with a Find/Replace or in a code editor, we can edit the TMDL as opposed to something that might take a lot of clicks in PowerBI Desktop.

Loan Logic, an application to help with mortgage scenario modelling

Fri, 14 Nov 2025 00:00:00 GMT

Loan Logic: A Mortgage scenario modelling application

Background

I have always been interested in financial modelling and how different financial decisions can impact long term outcomes. Given I am a web developer, a while back I decided to make myself a mortgage managing app, and at the time I wrote the backend in PHP.

With the introduction of AI coding assistants, I thought it would be a great opportunity to:

refactor an old PHP application I had written into a Python/Cloud based application.
test the process of using Google’s Antigravity and VSCode Copilot to do a large refactor

Architecture

1. Frontend (Client & Serving)

Technology: Vanilla JavaScript, TailwindCSS, DaisyUI.
Serving: Nginx container acts as a web server, delivering static assets (HTML, JS, CSS) to the browser.
Auth Handling:
- after user logs in client receives a JWT (Access Token) that is stored in localStorage.
- the ApiClient intercepts all requests and injects the Authorization: Bearer header.

2. Backend (Application API)

Technology: Python FastAPI, running on Uvicorn.
Authentication (Internal):
- Uses python-jose for JWT encoding/decoding.
- Uses passlib[argon2] for secure password hashing.

3. Database Layer

Technology: Remote MySQL Database (managed externally, e.g., AWS RDS, Azure MySQL, or a VPS).
ORM: SQLAlchemy handles object-relational mapping, translating Python objects to SQL queries.
Migrations: Alembic manages database schema versioning (creating tables, altering columns) directly against the MySQL database.

A (fictitious) journey of home ownership and financial uncertainty

Now, to demonstrate a use case for the application with screen shots, I thought it would be good to step through how it might be used with an example.

Meet Sarah Thompson, a fictitious (and AI generated) Australian professional who is about to buy her first property in the Canberra suburb of Franklin.

Her financial situation was stable for a long period of time, and she was on track to pay off her mortgage in the standard 30 years. But as with everyone, financial circumstances changed over time. A few years into her home ownership and she gets a decent tax refund from the ATO. Then a few years later, her uncle Peter passes away and leaves her a small inheritance.

She also had some unexpected expenses over the course of her mortgage.

This became a roller coaster of financial ups and downs for Sarah and she needed a way to model different scenarios to see how these events impacted her mortgage.

Loan Logic

Sarah looked around for something that could give her an idea of how these large financial events would impact her mortgage. She couldn’t find anything that she liked (well she is a fictitious person, but you get the idea).

Since she actually worked in the tech industry and was comfortable spinning up applications with Docker and setting up backend databases, her friend suggested she try Loan Logic.

After setting up her MySQL database, and migrating the schema using Alembic she confirmed the tables were deployed.

She creates her .env file for the DB connection and then spins up the application with docker-compose up --build and opens up the service https://localhost:3000.

# .env
# Database Configuration
# Replace with your actual remote MySQL database connection details
DATABASE_URL=mysql+pymysql://your_username:your_password@your_mysql_host:3306/mortgage_app

# Authentication Configuration
SECRET_KEY=your-super-secret-jwt-key
ACCESS_TOKEN_EXPIRE_MINUTES=30

# docker-compose.yaml
version: '3.8'

services:
  # Backend API
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    container_name: loanlogic_backend
    env_file:
      - .env
    ports:
      - "8000:8000"
    volumes:
      - ./backend:/app
    networks:
      - loanlogic_network
    command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

  # Frontend
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    container_name: loanlogic_frontend
    ports:
      - "3000:80"
    depends_on:
      - backend
    networks:
      - loanlogic_network
    volumes:
      - ./frontend/src:/usr/share/nginx/html

networks:
  loanlogic_network:
    driver: bridge

After she registers, she confirms that new user in the users tables.

First she inputs the baseline mortgage parameters as was the case when she bought the property.

After doing all this she setups up a baseline scenario.

She clicks on the baseline scenario to see the mortgage payments over time.

She can see the mortgage payments over time decreasing the loan balance. She also now can track the cumulative interest paid. She is amazed that over the course of the 30 year mortgage she will pay more in interest than the original loan amount.

She then sets up a new scenario with the extra expenses and windfalls and clicks on the scenario tile to compare with the baseline scenario.

The way Loan Logic works is by tracking all events in a scenario as changes to an off-set account balance. In the chart below, the light blue line shows the offset account balance over time for the baseline scenario (flat on $0), and the dark blue line shows the offset account balance for the events scenario.

Sarah can now easily see that even with the extra expenses, the windfalls will allow her to pay off her mortgage in 28 years which will save her more than $82,000 in interest.

Load to Teradata with a Python wrapper for TPT

Wed, 08 Oct 2025 12:00:00 GMT

PyTPT: A Python Wrapper for High-Performance Teradata Data Loading

This blog post is paired with the associated Github repository https://github.com/mortie23/python-teradata

In my previous blog post about Connecting to Teradata from Python, I demonstrated how to connect to Teradata from Python using the teradatasql driver and load CSV data using the built-in teradata_read_csv function. While that approach works well for many use cases, it has limitations when dealing with larger datasets or when you need the full power of Teradata’s native loading tools.

In this post I have implemented a Python package PyTPT that is basically a wrapper for Teradata’s TPT (Teradata Parallel Transporter) that can be used for bulk loading of CSV files. This approach leverages Teradata’s optimized loading engine while maintaining the simplicity and flexibility of Python.

What is TPT?

From the Official docs

Teradata Parallel Transporter (PT) is an object-oriented client application that provides scalable, high-speed, and parallel data:

Extraction
Loading
Updating

A TPT job requires at least a Job Script and Job Variables.

PyTPT bridges this gap by automatically generating these files from templates and orchestrating the entire loading process through Python.

Local Python Environment Setup

For this implementation, I’m using:

Python 3.13
Poetry for dependency management
Windows and Linux compatibility

Let’s set up the environment:

# Create your python virtual environment
# Linux
python -m venv ~/venv/pytpt
source ~/venv/pytpt/bin/activate

# Windows
python -m venv C:\venv\pytpt
C:\venv\pytpt\Scripts\Activate.ps1

# Install poetry and then packages
pip install poetry
poetry install

Prerequisites

Before we begin, you’ll need:

A Teradata machine (I’m using TD Express VM)
Teradata Studio (optional, but helpful for checking results)
Teradata Tools and Utilities (TTU) - This is crucial as it includes the tbuild binary that TPT requires

The TTU installation provides the tbuild command that PyTPT calls under the hood to execute TPT operations.

Configuration Structure

PyTPT uses a hierarchical configuration system powered by Hydra, making it easy to manage different environments and datasets.

Project Configuration

The main project configuration in conf/config.yaml. The database default will then become the name of the next configuration file and must be the same name as the database you are planning to load to.

# Main configuration file for pytpt examples
# This file sets up the default configuration structure

defaults:
  - database: PRD_ADS_PYTHON_NFL_DB
  - _self_

# Data directory configuration
data:
  dir: "../../data"

This configuration tells PyTPT where to find your CSV files and how they map to Teradata tables.

Database Configuration

First, configure your Teradata connection in conf/database/PRD_ADS_PYTHON_NFL_DB.yaml:

# @package _global_
# Teradata NFL database configuration

database:
  ddl_host: "tdvm"
  target_host: "tdvm"
  working_database: "PRD_ADS_PYTHON_NFL_DB"
  name: "PRD_ADS_PYTHON_NFL_DB"
  ddl_logmech: "TD2"
  target_logmech: "TD2"

# Table configurations for NFL data
tables:
  game: "GAME"
  game_type: "GAME_TYPE"
  game_stats: "GAME_STATS"
  game_venue: "GAME_VENUE"
  player: "PLAYER"
  venue: "VENUE" 
  team_lookup: "TEAM_LOOKUP"
  weather: "WEATHER"

How PyTPT Works

PyTPT follows a simple workflow:

Discovery: Scans the configured data directory for CSV files
Mapping: Matches CSV files to configured table names
Template Generation: Creates TPT job variables and scripts from Jinja2 templates
Execution: Runs the TPT operations in sequence:
- Drop existing table (if it exists)
- Create new table structure
- Load CSV data using TPT’s optimized bulk loading

Let’s look at the core loading function:

def load_table(
    table_name: str,
    csv_file_path: str,
) -> bool:
    """Load a single CSV file to a table using TPT."""
    from .templates import create_tpt_files

    logger.info(f"Starting table load process: {csv_file_path} -> {table_name}")

    # Create TPT files
    jvar_file, drop_file, create_file, load_file = create_tpt_files(
        table_name, csv_file_path
    )

    # Step 1: Drop table
    logger.info(f"Step 1/3: Dropping table {table_name}")
    drop_success = run_tbuild(jvar_file, drop_file, "drop")

    if not drop_success:
        logger.error(f"Failed to drop table {table_name}")
        return False

    # Step 2: Create table
    logger.info(f"Step 2/3: Creating table {table_name}")
    create_success = run_tbuild(jvar_file, create_file, "create")

    if not create_success:
        logger.error(f"Failed to create table {table_name}")
        return False

    # Step 3: Load data
    logger.info(f"Step 3/3: Loading data into {table_name}")
    load_success = run_tbuild(jvar_file, load_file, "load")

    if load_success:
        logger.success(f"Successfully completed table load: {table_name}")
    else:
        logger.error(f"Failed to load data to {table_name}")

    return load_success

Template-Driven TPT Generation

PyTPT uses Jinja2 templates to generate TPT files dynamically. Here’s an example of the create table template:

-- Create table template TPT
DEFINE JOB create_table_job
DESCRIPTION 'Create Table TPT JOB' (
  STEP drop_table (
        APPLY ('DROP TABLE {{working_database}}."{{table_name}}";')
        TO OPERATOR ($DDL);
    );
);

Here is the job variables template:

# Template Job variables file
DDLTdpId='{{ddl_host}}'
, DDLUserName='{{ddl_username}}'
, DDLUserPassword='{{ddl_password}}'
, DDLLogonMech='{{ddl_logon_mech}}'
, TargetTdpId='{{target_host}}'
, TargetUserName='{{target_username}}'
, TargetUserPassword='{{target_password}}'
, TargetLogonMech='{{target_logon_mech}}'
, TargetWorkingDatabase = '{{working_database}}'
, LoadTargetTable='{{working_database}}."{{table_name}}"'
, LoadCharacterSet = 'UTF8'
, FileReaderFileName='{{csv_file_path}}'
, FileReaderFormat='Delimited'
, FileReaderOpenmode='read'
, FileReaderTextDelimiter=','
, FileReaderSkipRows=1
, FileReaderCharSet = 'UTF8'
, FileReaderPrivateLogName='.\logs\tpt_{{table_name}}_filereader.log'
, FileReaderQuotedData = 'Optional'
, FileReaderMultipleRecordsInField = 'Y'
, FileReaderEscapeQuoteDelimiter = '"'
, FileReaderTruncateColumns = 'Y'
, FileReaderIndicatorMode = 'Y'
, LoadPrivateLogName='.\logs\tpt_{{table_name}}_load.log'
, DDLPrivateLogName='.\logs\tpt_{{table_name}}_ddl.log'

Execution and Monitoring

To run PyTPT just execute this command:

python app.py

The application provides logging throughout the process to files in the logs directory. Example of loading one table.

2025-09-29 17:11:13.883 | INFO     | __main__:main:73 - Processing 1 CSV files with table mappings
2025-09-29 17:11:13.884 | INFO     | __main__:main:82 - Processing: venue.csv -> VENUE
2025-09-29 17:11:13.884 | INFO     | pytpt.tpt_runner:load_table:144 - Starting table load process: C:\git\github\mortie23\python-teradata\eng\pytpt\..\..\data\venue.csv -> VENUE
2025-09-29 17:11:13.919 | INFO     | pytpt.tpt_runner:load_table:152 - Step 1/3: Dropping table VENUE
2025-09-29 17:11:13.920 | INFO     | pytpt.tpt_runner:run_tbuild:48 - Starting drop operation: tbuild -f render_tmp\VENUE_drop.tpt -v render_tmp\VENUE.jvar
2025-09-29 17:11:20.383 | SUCCESS  | pytpt.tpt_runner:run_tbuild:110 - drop operation completed successfully
2025-09-29 17:11:20.383 | INFO     | pytpt.tpt_runner:run_tbuild:111 - Complete TPT output saved to: logs\tpt_VENUE_drop_combined.log
2025-09-29 17:11:20.383 | INFO     | pytpt.tpt_runner:load_table:160 - Step 2/3: Creating table VENUE
2025-09-29 17:11:20.383 | INFO     | pytpt.tpt_runner:run_tbuild:48 - Starting create operation: tbuild -f render_tmp\VENUE_create.tpt -v render_tmp\VENUE.jvar
2025-09-29 17:11:26.824 | SUCCESS  | pytpt.tpt_runner:run_tbuild:110 - create operation completed successfully
2025-09-29 17:11:26.824 | INFO     | pytpt.tpt_runner:run_tbuild:111 - Complete TPT output saved to: logs\tpt_VENUE_create_combined.log
2025-09-29 17:11:26.824 | INFO     | pytpt.tpt_runner:load_table:168 - Step 3/3: Loading data into VENUE
2025-09-29 17:11:26.825 | INFO     | pytpt.tpt_runner:run_tbuild:48 - Starting load operation: tbuild -f render_tmp\VENUE_load.tpt -v render_tmp\VENUE.jvar
2025-09-29 17:11:43.988 | SUCCESS  | pytpt.tpt_runner:run_tbuild:110 - load operation completed successfully
2025-09-29 17:11:43.988 | INFO     | pytpt.tpt_runner:run_tbuild:111 - Complete TPT output saved to: logs\tpt_VENUE_load_combined.log
2025-09-29 17:11:43.988 | INFO     | pytpt.tpt_runner:run_tbuild:122 - Rows sent to RDBMS: 32
2025-09-29 17:11:43.989 | INFO     | pytpt.tpt_runner:run_tbuild:124 - Rows applied: 32
2025-09-29 17:11:43.989 | SUCCESS  | pytpt.tpt_runner:load_table:172 - Successfully completed table load: VENUE
2025-09-29 17:11:43.989 | SUCCESS  | __main__:main:86 - Successfully processed: venue.csv
2025-09-29 17:11:43.990 | INFO     | __main__:main:93 - Load process completed: 1/1 files loaded successfully
2025-09-29 17:11:43.990 | SUCCESS  | __main__:main:103 - All files loaded successfully!

PyTPT also captures detailed TPT output in log files for troubleshooting:

tpt_VENUE_create_combined.log
tpt_VENUE_load_combined.log
tpt_VENUE_drop_combined.log

Results

After I ran it on my machine the tables were created and records loaded.

Error Handling and Robustness

The system validates that rows sent equal rows applied, providing confidence in data integrity.

Comparison with Previous Approach

Feature	teradatasql + teradata_read_csv	PyTPT
Performance	Good for small-medium datasets	Optimized for large datasets
Complexity	Simple, direct SQL	Template-driven, more setup
Scalability	Limited by Python/ODBC	Native TPT parallelism
Error Handling	Basic SQL error messages	Detailed TPT logs and metrics
Configuration	Hardcoded in scripts	Flexible YAML configuration
Dependencies	teradatasql driver	TTU installation required

Debugging

Some of the defaults I have chosen were a result of issues loading certain types of CSV files that took a long time to find the correct variables.

For example, loading CSV data where text fields have newlines such as:

integer_field,text_field_with_newline
1,test
2,test
3,"this long line.
that has newlines"
4,test

Requires these job variables:

, FileReaderQuotedData = 'Optional'
, FileReaderMultipleRecordsInField = 'Y'
, FileReaderEscapeQuoteDelimiter = '"'
, FileReaderTruncateColumns = 'Y'

If cloud platforms were people (thoughts from a Generative AI)

Wed, 10 Sep 2025 12:00:00 GMT

Can you relate to these?

As of writing this blog post, I have been really enjoying the new releases in Google’s Gemini AI for both images (Nano Banana) and videos (Veo 3). I was inspired to continue from my previous blog post If programming languages were people (thoughts from a Generative AI) and try something new. The following is the conversation I had with ChatGPT and the resulting generated images and videos.

The prompt

Christopher:

I wrote this blog post https://mortimer.xyz/if-programming-languages-were-people-ai-thoughts/ in 2024, it was fun and (some) people liked it. This time I wanted to use Google Veo 3 to do something similar. “What AI thinks cloud platforms would be if they were personified”?

I first will use ChatGPT (you) to generate the prompts for the videos and then pass that to Google Veo3.

Can you write a short script for each of these to generate the 8 second videos

The responses

ChatGPT:

Thanks for sharing your blog post—from …

This is a great fit for Veo. Short, snappy, visually symbolic, and fun. For each 8-second clip, Veo prompts work best when they’re compact, descriptive, and directive, almost like “scene notes”.

Here’s a draft script for each of your three “cloud humans”:

The following video prompts were generated by ChatGPT, and videos were generated with Google Veo 3.

AWS - The Swiss Army Person

A human shaped like a Swiss Army multitool in a bright workshop. Wearing a rugged jacket with endless orange and black gadgets and attachments. Tools unfold rapidly from arms and shoulders, sparks flying as gears click. Camera zooms in on many hands juggling tools, then pulls back to reveal a cluttered, overwhelming workspace. Ambient audio of clanking metal and machinery hum.

My take from personal experience

AWS, although extremely capable, can seem complex. To get from vanilla to a working solution with correct IAM isn’t easy. I was once forced to sign up for developer support on an AWS account to get passed an error just to be told I did nothing wrong, my IAM was correct and it was an “internal” issue that has been resolved (of course I asked for a refund for the developer support and they said yes, I can’t fault the support team at all).

The names of the services, the shear number of them with many providing overlapping solutions.

I definitely feel the:

pulls back to reveal a cluttered, overwhelming workspace

Azure - The Polished Corporate Networker

A corporate professional in a sleek blue suit, standing in a futuristic glass office. Blue holographic network diagrams float around them as they connect glowing threads between icons. They pause to check a floating Outlook email window, then smoothly return to connecting nodes. Camera slowly circles around. Soft background lounge music, occasional email notification chime. Color theme: Azure blue and silver glow.

My take from personal experience

Azure is usually used when at work. It can sometimes feel forced upon you because, well, your enterprise IT team already supports Microsoft Windows and M365, and probably already has a great relationship with Microsoft. Azure is chosen as the cloud and it focuses on being appealing to large enterprises.

I chuckled when the AI titled it:

Corporate Networker

Google Cloud - The Data-Savvy Friend

A cheerful person in colorful casual clothes, sitting at a cafe table. Bright holographic charts and icons in Google’s red, yellow, green, and blue colors hover around them. They swipe through graphs with ease, wink at the camera, and slide a glowing chart to someone nearby. Camera pans close to their friendly face, then tilts up to show vibrant floating data visuals. Cafe ambience with light upbeat tones and a soft chime.

My take from personal experience

GCP feels the most approachable and can be more enjoyable (with the fair share of IAM pain). The terminology around “projects” seems to make sense to how you normally separate work. You can get a solution up in GCP quickly from vanilla. The fact the AI focuses on data-savvy may be because GCP has BigQuery which is a great serverless database if you have big data solutions to tackle.

My favourite line form the AI generated prompt:

wink at the camera, and slide a glowing chart to someone nearby

Quackpipe, running DuckDB local and on AWS Glue

Fri, 15 Aug 2025 12:00:00 GMT

Quackpipe

Stop waiting for AWS Glue jobs while in development by using a local-first workflow with DuckDB and deploy with Terraform. Your new addiction to fast, flexible data pipelines. The quickest high from local to cloud.

Based on the following Github repository https://github.com/mortie23/quackpipe

Developing on a platform that wasn’t made for it

Have you ever had the pain of being in a situation where, as a developer you are asked to do your development inside a tool that really isn’t meant for development, like the AWS Glue web console.

Each change to your code leads to:

slow runs/job startups that take half a minute
limited debuggability, where you can’t add breakpoints
multiple clicks to open log files
downloading and opening resulting files from S3 is slow
packaging friction and dependency trial-and-error
permissions / IAM / S3 path errors only show up late

Not the optimal DX (Developer Experience).

Quack Pipe gives you a local-first workflow

Develop and iterate Python (and DuckDB SQL) locally against local data (initially CSV)
Re-use the exact same code when packaging for AWS Glue Python Shell
Deploy with deterministic Terraform infrastructure (S3 artifacts, IAM, Glue Job)
Run fast (seconds) for small/medium datasets and can scale later if needed

Architecture

Components:

Example local runner: nfl_team_season_total.py (calls DuckDB over local or S3 CSVs)
Example data lake (S3):

folder	description
`scripts/`	For source script and wheel artifacts
`data/nfl/`	The source data
`results/`	Output results
`tmp/`	Required in Glue job properties

Artifact Publishing: Terraform-managed S3 objects for script and wheel
Glue Python Shell Job: Executes the same script with --extra-py-files pointing to built wheel
IAM Roles/Policies: Least-privilege access to bucket objects & logs

Data flow	Data files	Execution	Query	Output
local	CSV on disk	Python	DuckDB in-memory	stdout / local file
cloud	CSV in S3	Glue Python Shell	DuckDB in-memory	S3

Why DuckDB Here?

The main reason here was an alternative to Glue Spark jobs and to make the local runner option easy to install and setup. However, DuckDB is also extremely suitability for this type of analytical workflow on moderate sized datasets. There are zero external services required and you get a consistent behavior locally and in Glue.

Local Development Workflow

Create & activate virtual environment
Install deps via Poetry
Run the script (iterative changes are seconds)

I built this on my Windows 11 machine (but it also runs on Linux machine), so I am using PowerShell. The instructions will be similar if Linux|MacOS system.

Let’s create a virtual python environment.

mkdir C:\venv
python -m venv C:\venv\quackpipe
C:\venv\quackpipe\Scripts\Activate.ps1

Create a dot env file. (.env) in the root of the project. Copy the contents of the .example file and paste it in the .env file

Your virtual environment should be activated. We will use pip to install poetry and then poetry to manage all other dependencies.

# From this cloned directory as the working directory
pip install poetry
poetry install

Run a local quackpipe

# From this cloned directory as the working directory
python nfl_team_season_total.py

Inspect outputs / adjust transformations as you need.

Switching to Remote (Glue) Execution

Firstly produce a versioned wheel under dist/

poetry build

Building quackpipe (0.1.0)
Building sdist
  - Building sdist
  - Built quackpipe-0.1.0.tar.gz
Building wheel
  - Building wheel
  - Built quackpipe-0.1.0-py3-none-any.whl
(quackpipe)

Now, we can use Terraform to:

Deploy all the infra and IAM required
Sync the Python script and wheel to S3
Sync the data to S3
Create the Glue job

Terraform variable and state hygiene

Before you deploy the infra using IaC (Terraform) it is important to know what belongs in variables.tf vs terraform.tfvars.

variables.tf declares schema (name, type, default, validation).
terraform.tfvars (untracked by Git) supplies environment / secret values.
Commit terraform.tfvars.example only.

NOTE: the state files should be stored in remote cloud storage (such as S3) but for this simple demo example they are just kept local

Create a terraform.tfvars and populate it with the contents of the example (with setting the correct secret values for you use case)

# First check the plan
terraform plan
# Apply the plan
terraform apply -auto-approve

Issue: I couldn’t see the Glue job in the web console after the successful deploy, but I realised that I have the wrong region selected. 🤪

Data Input Modes

The simple solution that solves the two modes (local and remote).

Execution Mode	Description
Local	Uses on-disk CSVs under `data/`
Remote	Uploads same CSVs to S3 and script switches to `s3://` paths

In the main Glue script we use this handy function to get the job parameters. The key parameter is execution_mode which is set to ‘local’ in the .env for local running and left absent in the Glue job.

def get_job_params(
    required_params: list,
) -> dict:
    """Get job parameters for a AWS Glue job

    Args:
        required_params (list): list of the parameters to get

    Returns:
        dict: a dictionary of the parameters values
    """
    try:
        # If in Glue get from job parameters
        from awsglue.utils import getResolvedOptions

        args = getResolvedOptions(sys.argv, required_params)
    except ModuleNotFoundError:
        # Local fallback using environment variables (populated by .env)
        args = {k: os.getenv(k) for k in required_params}
    return args


# Load .env file into environment
load_dotenv()
execution_mode = os.getenv("execution_mode")

Since the leave it absent in the Glue job, it drops into the else and looks for the files in S3.

if execution_mode == "local":
    try:
        file_path = Path("data") / data_collection / file_name
        df = pd.read_csv(file_path)
        df = sanitize_column_names(df)
        con.register(table_name, df)
        registered_status = (True, None)
    ...
    except Exception as e:
        registered_status = (False, f'{str(e)}: {file_name}')
else:
    try:
        key = f"data/{data_collection}/{file_name}"
        content = read_s3_file(
            bucket=bucket,
            key=key,
        )

        df = pd.read_csv(StringIO(content))
        df = sanitize_column_names(df)
        # Add a file row number for tracking later
        df.insert(0, 'Row Number', range(1, len(df) + 1))
        con.register(table_name, df)
        registered_status = (True, None)
    ...
    except Exception as e:
        registered_status = (False, f'{str(e)}: {file_name}')

Run the Glue job via the web console

You can see that before I got a successful run, I had some issues. I initially had an error saying the Glue version was not valid.

InvalidInputException: Given GlueVersion [3.0] is not valid.
InvalidInputException: Given GlueVersion [4.0] is not valid.

I removed glue_version from the Terraform definition and set python_version = "3.9". It defaulted to deploy to Glue version 5.

The first time I got the configuration correct and a successful run started, the job spent many minutes (over 11 until I stopped it) compiling DuckDB from source. I realised it was that DuckDB >=1.2.x had dropped Python 3.7 wheels. Actual Glue runtime resolved to Python 3.7 (despite attempts for 3.9) and since there is no wheel it resulted in a source build.

To fix this I pinned the DuckDB dependency to:

duckdb==1.1.3

This version still publishes Python 3.7 manylinux wheels. I rebuilt the quackpipe wheel and redeployed and the startup time was much faster.

After getting a successful run, the results exported to S3 folder.

Terraform Infrastructure Highlights

Bucket name includes region for uniqueness
Explicit folder objects created for clarity (not strictly required by S3)
fileset used to push all data/nfl/*.csv into data/nfl/ prefix
Glue job avoids hard-coded Glue version (region variability) while pinning Python 3.9
IAM policies refer to actual bucket resource IDs—prevents stale string drift

Example from terraform plan when a new wheel is built.

Handling Python Dependencies

Strategy: Build once locally → upload wheel → use --extra-py-files
DuckDB pinned to a version with prebuilt manylinux wheels (faster cold start)
Avoids runtime pip install penalties

Summary

DuckDB in-memory operations sub-second for small test set running locally
Running in cloud the cold start time is dominated by Glue container and dependency load

This is really just a very slim example of an idea. It could be extend to more jobs or Parquet outputs and adding features such as:

Add CI step to build/upload artifacts before Terraform apply (ensures script exists).
Migrate to Python 3.9 runtime once consistently supported (or adopt Glue 5.0 when stable) and restore newer DuckDB.

Query Salesforce with Python

Sat, 05 Jul 2025 00:00:00 GMT

Query SalesForce objects with Python

Ever found yourself exporting data from Salesforce using the Web UI over and over again. Ever wished you could just automate the process with scripts and then setup a job schedule and never click through the Salesforce Web UI again? Maybe you will find this useful?

The source code for this blog post is here: https://github.com/mortie23/sf-de/tree/master/eng/sf/sf-extract. If you are simply interested in the example usage and already have a Salesforce organistaion and user with a security token, then skip to Example Usage

Setting Up a Salesforce Developer Edition for API Access

The first step is getting an API user set up. We can use a Salesforce Developer Edition account to create a user specifically for API use and then get an API key (security token).

Go to https://developer.salesforce.com/signup.
Fill in your details and create your account.
Check your email and verify your Salesforce Developer account.

Step 2: Log Into Your Developer Org

Visit https://login.salesforce.com and sign in with your new Developer account.
Click the ⚙️ gear icon → Setup to access the admin panel.

Step 3: Create a New Standard User for API Access

To keep your main account separate from scripts, it’s best to create a dedicated API user:

In Setup → Quick Find, search Users → click Users.
Click New User.

Fill in the required fields:
- First Name / Last Name: API User
- Email: your accessible email
- Username: unique across all Salesforce orgs
- Alias: e.g., apiuser
- User License: Salesforce (standard)
- Profile: Standard User
Save the user.

You would have gotten an email to the new users email inbox.

Step 4: API Access

For a Standard User, API access is usually enabled by default.

Step 5: Get the Security Token

Salesforce requires a security token for API login from outside their UI:

Log in as the new Standard User.
Click your avatar → Settings.

In Quick Find, search Reset My Security Token.

Click Reset Security Token → Salesforce emails it to you.

You will get an email with the security token in the email.

Example usage

Setup your Python environment.

# Create a Python virtual environment
python3 -m venv ~/venv/sfde
# Activate the environment
source ~/venv/sfde/bin/activate
# Install poetry into the virtual environment
pip install poetry
# Use poetry to install the dependencies 
poetry lock && poetry install

Add you environment variables (if you need step by step guide on setting up a developer SalesForce org and creating the credentials read above) in a new .env file:

SALESFORCE_USERNAME=your_salesforce_username
SALESFORCE_PASSWORD=your_salesforce_password
SALESFORCE_SECURITY_TOKEN=your_salesforce_security_token

Run the Python script to execute the queries

python3 app.py --query all

2025-05-05 14:47:40,042 - INFO - Successfully connected to Salesforce
2025-05-05 14:47:40,051 - INFO - Executing query: 
select
  count(Id)
from
  account
2025-05-05 14:47:40,553 - INFO - Query returned 1 records
OrderedDict({'attributes': OrderedDict({'type': 'AggregateResult'}), 'expr0': 14})
2025-05-05 14:47:40,559 - INFO - Executing query: 
select
  count(Id)
from
  contact
2025-05-05 14:47:40,908 - INFO - Query returned 1 records
OrderedDict({'attributes': OrderedDict({'type': 'AggregateResult'}), 'expr0': 21})
2025-05-05 14:47:40,915 - INFO - Executing query: 
select
  Id
, name
from
  contact
2025-05-05 14:47:41,194 - INFO - Query returned 21 records
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>hQAE'}), 'Id': '<>hQAE', 'Name': 'Rose Gonzalez'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>iQAE'}), 'Id': '<>iQAE', 'Name': 'Sean Forbes'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>jQAE'}), 'Id': '<>jQAE', 'Name': 'Jack Rogers'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>kQAE'}), 'Id': '<>kQAE', 'Name': 'Pat Stumuller'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>lQAE'}), 'Id': '<>lQAE', 'Name': 'Andy Young'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>mQAE'}), 'Id': '<>mQAE', 'Name': 'Tim Barr'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>nQAE'}), 'Id': '<>nQAE', 'Name': 'John Bond'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>oQAE'}), 'Id': '<>oQAE', 'Name': 'Stella Pavlova'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>pQAE'}), 'Id': '<>pQAE', 'Name': 'Lauren Boyle'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>qQAE'}), 'Id': '<>qQAE', 'Name': 'Babara Levy'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>rQAE'}), 'Id': '<>rQAE', 'Name': 'Josh Davis'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>sQAE'}), 'Id': '<>sQAE', 'Name': 'Jane Grey'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>tQAE'}), 'Id': '<>tQAE', 'Name': 'Arthur Song'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>uQAE'}), 'Id': '<>uQAE', 'Name': 'Ashley James'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>vQAE'}), 'Id': '<>vQAE', 'Name': 'Tom Ripley'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>wQAE'}), 'Id': '<>wQAE', 'Name': "Liz D'Cruz"})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>xQAE'}), 'Id': '<>xQAE', 'Name': 'Edna Frank'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>yQAE'}), 'Id': '<>yQAE', 'Name': 'Avi Green'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/<>zQAE'}), 'Id': '<>zQAE', 'Name': 'Siddartha Nedaerk'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/?'}), 'Id': '?', 'Name': 'Jake Llorrac'})
OrderedDict({'attributes': OrderedDict({'type': 'Contact', 'url': '/services/data/vM.m/sobjects/Contact/&'}), 'Id': '&', 'Name': 'Shane Warne'})

Publishing a Python Streamlit app to Posit Connect

Wed, 09 Apr 2025 12:00:00 GMT

Python applications published to Posit Connect

This is a simple Streamlit application structured as a modular project and managed using Poetry. The application is designed to be deployed to a Posit Connect server.

Reference: https://github.com/mortie23/ml/tree/master/app/streamlit-posit-app

Project Structure

📁streamlit-posit-app
├── 📁streamlit_posit_app
│   └── 📁pages
│       └── __init__.py
│   │   └── home.py
├── 📁tests
│   └── __init__.py
├── app.py
├── .gitignore
├── pyproject.toml
└── README.md

Configure Python applications on Posit Connect

I built my own Posit Connect server VM for testing this out. If you want to do the same, I suggest reading my previous blog post Install Posit Connect on Ubuntu

I followed the recommended installation of Python with the latest version available at the time of writing this. Docs can be found at https://docs.posit.co/resources/install-python.html

# Set preferred Python version
export PYTHON_VERSION="3.12.4"
# Download and install
curl -O https://cdn.rstudio.com/python/ubuntu-2404/pkgs/python-${PYTHON_VERSION}_1_amd64.deb
sudo apt-get update
sudo apt-get install ./python-${PYTHON_VERSION}_1_amd64.deb
# check install
/opt/python/"${PYTHON_VERSION}"/bin/python --version

Then you need to edit the Connect config file to enable Python apps and set the location of the kernel. Docs can be found at https://docs.posit.co/connect/admin/python/

; /etc/rstudio-connect/rstudio-connect.gcfg
[Python]
Enabled = true
Executable = /opt/python/3.12.4/bin/python3

Development of the application

I used a local development environment of WSL2 on my Dell XPS with Windows 11. I could have done the development through RStudio on the VM server, but I prefer VSCode myself.

On the development machine, create a virtual environment and install poetry.

python3 -m venv ~/venv/posit
source ~/venv/posit/bin/activate
pip install poetry

All packages will be managed by poetry. The application directory on the referenced Github repo was created with this following command.

poetry new ./streamlit-posit-app

Then each new package is added with the following command.

poetry add rsconnect-python

But, if we restoring this from the referenced repo, then we won’t manually be adding each package, we just use poetry to install all dependencies.

poetry lock && poetry install --no-root

Running the App

To run the Streamlit application locally, use the following command:

poetry run streamlit run app.py
# or just
streamlit run app.py

Deployment

To deploy the app to a Posit Connect server use the following command.

# Adding server and api key for use later via name
rsconnect add --name vm --server http://192.168.110.133:3939 --api-key <api-key>
# Deploy
rsconnect deploy streamlit --name vm --entrypoint app.py .

    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server...    [OK]
Validating app mode...  [OK]
Making bundle ...       [OK]
Deploying bundle ...    [OK]
Saving deployed information...  [OK]
Building Streamlit application...
Bundle created with Python version 3.12.4 is compatible with environment Local with Python version 3.12.4 from /opt/python/3.12.4/bin/python3.12
Bundle requested Python version 3.12.4; using /opt/python/3.12.4/bin/python3.12 from Local which has version 3.12.4
2025/04/09 11:20:54.466483344 [rsc-session] Content GUID: a90a5b19-d989-4cf1-99fd-7fa748fadae4
2025/04/09 11:20:54.467263345 [rsc-session] Content ID: 6
2025/04/09 11:20:54.467448246 [rsc-session] Bundle ID: 10
2025/04/09 11:20:54.467591546 [rsc-session] Job Key: CXzbEAP0fl5PEUf6
Determining session server location ...
Connecting to session server http://127.0.0.1:35199 ...
Connected to session server http://127.0.0.1:35199
2025/04/09 11:20:55.567199857 Running on host: ubuntu-2404-server
2025/04/09 11:20:55.613260740 Linux distribution: Ubuntu 24.04.2 LTS (noble)
2025/04/09 11:20:55.633468077 Running as user: uid=996(rstudio-connect) gid=987(rstudio-connect) groups=987(rstudio-connect)
2025/04/09 11:20:55.633742877 Connect version: 2024.09.0
2025/04/09 11:20:55.634468878 LANG: en_US.UTF-8
2025/04/09 11:20:55.634618579 Working directory: /opt/rstudio-connect/mnt/app
2025/04/09 11:20:55.635216580 Building environment using Python 3.12.4 (main, Feb 25 2025, 22:32:47) [GCC 13.3.0] at /opt/python/3.12.4/bin/python3.12
2025/04/09 11:20:55.647333402 Using cached environment: Bq5M9EoyWyCJi_KEsAiGSg
2025/04/09 11:20:58.194650391 Packages in the environment: altair==5.5.0, anyio==4.9.0, asttokens==3.0.0, attrs==25.3.0, blinker==1.9.0, build==1.2.2.post1, CacheControl==0.14.2, cachetools==5.5.2, certifi==2025.1.31, cffi==1.17.1, charset-normalizer==3.4.1, cleo==2.1.0, click==8.1.8, comm==0.2.2, crashtest==0.4.1, cryptography==44.0.2, debugpy==1.8.13, decorator==5.2.1, distlib==0.3.9, dulwich==0.22.8, executing==2.2.0, fastjsonschema==2.21.1, filelock==3.18.0, findpython==0.6.3, gitdb==4.0.12, GitPython==3.1.44, h11==0.14.0, httpcore==1.0.7, httpx==0.28.1, idna==3.10, installer==0.7.0, ipykernel==6.29.5, ipython==9.1.0, ipython_pygments_lexers==1.1.1, jaraco.classes==3.4.0, jaraco.context==6.0.1, jaraco.functools==4.1.0, jedi==0.19.2, jeepney==0.9.0, Jinja2==3.1.6, jsonschema==4.23.0, jsonschema-specifications==2024.10.1, jupyter_client==8.6.3, jupyter_core==5.7.2, keyring==25.6.0, MarkupSafe==3.0.2, matplotlib-inline==0.1.7, more-itertools==10.6.0, msgpack==1.1.0, narwhals==1.34.0, nest-asyncio==1.6.0, num
py==2.2.4, packaging==24.2, pandas==2.2.3, parso==0.8.4, pbs-installer==2025.3.17, pexpect==4.9.0, pillow==11.1.0, pkginfo==1.12.1.2, platformdirs==4.3.7, poetry==2.1.2, poetry-core==2.1.2, prompt_toolkit==3.0.50, protobuf==5.29.4, psutil==7.0.0, ptyprocess==0.7.0, pure_eval==0.2.3, pyarrow==19.0.1, pycparser==2.22, pydeck==0.9.1, Pygments==2.19.1, PyJWT==2.10.1, pyproject_hooks==1.2.0, python-dateutil==2.9.0.post0, pytz==2025.2, pyzmq==26.4.0, RapidFuzz==3.13.0, referencing==0.36.2, requests==2.32.3, requests-toolbelt==1.0.0, rpds-py==0.24.0, SecretStorage==3.3.3, semver==3.0.4, setuptools==78.1.0, shellingham==1.5.4, six==1.17.0, smmap==5.0.2, sniffio==1.3.1, stack-data==0.6.3, streamlit==1.44.1, tenacity==9.1.2, toml==0.10.2, tomlkit==0.13.2, tornado==6.4.2, traitlets==5.14.3, trove-classifiers==2025.3.19.19, typing_extensions==4.13.1, tzdata==2025.2, urllib3==2.3.0, virtualenv==20.30.0, watchdog==6.0.0, wcwidth==0.2.13, wheel==0.46.1, zstandard==0.23.0,
2025/04/09 11:20:58.195191592 Creating lockfile: python/requirements.txt.lock
Stopped session pings to http://127.0.0.1:35199
Completed Python build using Local against Python version: '3.12.4'
Launching Streamlit application...
Deployment completed successfully.
         Dashboard content URL: http://192.168.110.133:3939/connect/#/apps/a90a5b19-d989-4cf1-99fd-7fa748fadae4/access
         Direct content URL: http://192.168.110.133:3939/content/a90a5b19-d989-4cf1-99fd-7fa748fadae4/
Verifying deployed content...   [OK]

Making Machine Learning predictions from BigQuery Dataform

Thu, 03 Apr 2025 12:00:00 GMT

NFL Touchdown Prediction with Dataform and BigQuery ML

This blog post demonstrates how to build an end-to-end ML pipeline using Google Cloud Platform (GCP) services to predict NFL game touchdowns.

There is a lot more that is setup and explained in previous blog posts:

Architecture

The solution uses:

Dataform for data transformation and pipeline orchestration
BigQuery for data storage and SQL operations
Cloud Run for hosting the ML model endpoint https://github.com/mortie23/ml/blob/master/infra/tf/gcp/environment/cloudrun.tf
Python for ML model training (stored separately)

Project Structure

ml-df/
├── 📁definitions/         # Dataform SQL definitions
│   ├── 📁t0_mdl/          # Model declarations
│   ├── 📁t1_raw/          # Raw data loading
│   └── 📁t2_cur/          # Curated/transformed/predicted data
├── 📁docs/                # Documentation and screenshots
│   ├── 📁img/             # Screenshots
└── workflow_settings.yaml # Dataform configuration

Setup and Usage

Set up a GCP project with BigQuery, Dataform and many other GCP infra components enabled
Fork this repository and setup a workspace in Dataform
Configure workflow_settings.yaml with your project details
Run the Dataform pipeline

Pipeline Flow

Raw NFL game statistics are loaded into BigQuery
Data is transformed and prepared for prediction
Each game’s features are sent to the ML model endpoint (Cloud Run service is required and discussed in other previously linked blog posts)
Predictions are stored back in BigQuery

The main part of interest would be the SQLX for the prediction:

config {
  type: "incremental",
  description: "NFL game touchdown predictions",
  uniqueKey: ["game_team_id"],
  columns: {
    game_team_id: 'Unique identifier for game and team combination',
    predicted_touchdowns: 'Predicted number of touchdowns',
    created_timestamp: 'Created timestamp'
  },
  schema: dataform.projectConfig.vars.layer_cur
}

with batched_games as (
    select
      ifnull(
        array_agg(
          json_object(
            'game_team_id', game_team_id,
            'total_first_downs', total_first_downs,
            'total_yards', total_yards,
            'interceptions', interceptions,
            'punts', punts
          )
          ignore nulls
        ), [json_object()]
      ) as batch
    from
      ${ref("game_stats")}
      ${when(incremental(), `where game_team_id not in (select distinct game_team_id from ${self()})` )}
)
, predictions as (
    select
      ${ref("nfltouchdown_cloudrun")}(parse_json(to_json_string(batch))) as predictions_json
    from
      batched_games
)
, unnested_predictions as (
    select
        json_value(prediction, '$.game_team_id') as game_team_id
      , json_value(prediction, '$.prediction') as predicted_touchdowns
    from
      predictions
    , unnest(json_query_array(predictions_json, '$')) as prediction
)
select
    cast(up.game_team_id as INT64) as game_team_id
  , cast(up.predicted_touchdowns as FLOAT64) as predicted_touchdowns
  , current_timestamp() as created_timestamp
from
  unnested_predictions up

This method will incrementally take new game stats (for arguments sake, let’s say it was run weekly), and then only make one request for all the games that week to the Cloud Run predict service.

It does this by using array_agg and json_object to create an array of JSON objects and POSTing all that data to the BigQuery Remote Function connection. It then uses unnest to split back out the responses.

Example BigQuery objects

This shows what the BigQuery objects look like when viewed in the BigQuery web UI.

Example execution

A successful execution of the complete pipeline in Dataform looks like this:

Example results

When the pipeline is run, the predicted touchdowns are then available in the game_touchdown_predict table.

The Tale of Two Routes: A Love Story Between BigQuery and Vertex AI for Machine Learning predictions

Mon, 31 Mar 2025 12:00:00 GMT

This blog post is the first, and probably last where I thought I would see what the Github Copilot VSCode extension could do with writing a blog post giving one file as context. The blog post is written with respect to the following Python script https://github.com/mortie23/ml/blob/master/model/nfl/touchdown/nfltouchdownpredict/predict.py. I gave it to Claude 3.5 and asked it to write a blog post based on explaining this file and asked it to tell it as a story with some humor. I had to create the images, add some extra context here and there and fix some mistakes (like you have to do with the code it generates). Here is the result:

Have you ever tried to make two different client applications happy with the same ML prediction service? It’s like trying to serve breakfast to both an early bird and a night owl - they want the same food but at different times and in different ways! This blog post will outline how I used FastAPI and a Vertex AI trained model, to make both Cloud Run and Vertex AI online endpoints be able to host the same model and allow BigQuery remote functions to call the service as well.

The Characters in Our Story

First, let’s meet the cast:

FastAPI: The lightning-fast matchmaker
Pydantic: The strict but fair bouncer checking IDs at the door
Vertex AI: The enterprise consultant who loves formal meetings at /predict
BigQuery: The data engineer who prefers verbose catch-ups at the root path /

The Plot Thickens: Two Routes, One Model

This is our bouncer’s checklist (Pydantic) making sure everyone’s properly dressed for the party. It’s like a strict dress code, but instead of “no sneakers,” it’s “no string where an integer should be”!

For demonstrating our use case, we will be creating a prediction service that predicts the number of touchdowns a team scored based on the:

Number of first downs
Total yards gained
Number of interceptions thrown
Number of punts

The Python class looks like the following:

class ModelInputItem(BaseModel):
    game_team_id: int = Field(..., example=411)
    total_first_downs: int = Field(..., example=26)
    total_yards: int = Field(..., example=419)
    interceptions: int = Field(..., example=0)
    punts: int = Field(..., example=4)

This bouncer also polite enough to give a really detailed of description of why you are not allowed in if you don’t meet the checklist.

This example shows what response is given when we arrive at the endpoint with a POST request that contains only a bad field name. We are told we missed the required fields.

The Tale of Two Requests

Vertex AI: The Formal One

Vertex AI shows up being very particular about hosting at /predict wearing a three-piece suit, wrapping everything in an “instances” array. Very formal, very proper.

{
  "instances": [
    {
      "game_team_id": 411,
      "total_first_downs": 26,
      "total_yards": 419,
      "interceptions": 0,
      "punts": 4
    }
  ]
}

We can host the Vertex AI trained model using the Deploy and test directly from the model registry.

BigQuery: The Chatty One

BigQuery comes directly to root route / (and this cannot be changed). It also sends it’s life story (and this cannot be changed) - who it is, where it’s from, and oh by the way, the actual data is nested three levels deep because… why not? 🤷‍♂️

{
  "requestId": "unique-request-id",
  "caller": "//bigquery.googleapis.com/projects/...",
  "sessionUser": "user@example.com",
  "calls": [
    [
      [
        {
          "game_team_id": 411,
          "total_first_downs": 26,
          "total_yards": 419,
          "interceptions": 0,
          "punts": 4
        }
      ]
    ]
  ]
}

The Magic of FastAPI Routes

Once I started attempting to get BigQuery remote functions to call the /predict route that Vertex AI produced, I realised this wouldn’t work. So we needed another route. It is like having two different entrances to the same restaurant - the formal dining room for Vertex AI and the casual café for BigQuery. Same kitchen (our ML model), different serving styles!

@app.post("/")
async def predict_bigquery(input_data: BigQueryRemoteRequest):
    # BigQuery's casual coffee chat
    # ...existing code...

@app.post(os.environ["AIP_PREDICT_ROUTE"])
async def predict_vertex(input_data: VertexPredictionRequest):
    # Vertex AI's formal business meeting
    # ...existing code...

The Setup: Environment Magic and Config Sorcery

Before our restaurant even opens its doors, we need to set up the kitchen! Let’s peek behind the curtain:

The .env Chef’s Secret Recipe

It’s like having a secret recipe card - keeping our environment spices separate from the main cookbook! Vertex AI is very opinionated about how it uses environment variables, what they need to be called and in fact, sometimes different names for the same variable in different parts of the life cycle because… why not? 🤷‍♂️

env=dev
project_id=prj-xyz-dev-nfl-0
# For training
AIP_MODEL_DIR=gs://bkt-xyz-dev-nfl-vertex-0/aiplatform-custom-training-yyyy-mm-dd-hh:mm:ss.ms/model
# For prediction
AIP_STORAGE_URI=gs://bkt-xyz-dev-nfl-vertex-0/aiplatform-custom-training-yyyy-mm-dd-hh:mm:ss.ms/model
AIP_HEALTH_ROUTE=/ping
AIP_PREDICT_ROUTE=/predict
etc=

Vertex AI will be setting these for us. We need to be clever about setting them when deploying the service to Cloud Run for our BigQuery remote function.

Hydra: The Configuration Sous Chef

model_bucket: bkt-xyz-<env>-nfl-vertex-0
model_path: nfl-touchdown/model.joblib
etc:

Hydra is our sous chef who knows how to adapt the recipe for different kitchens (environments). Watch the magic:

initialize(version_base=None, config_path=".")
cfg = compose(config_name="predict")

# Season the config with environment flavoring
for k, v in cfg.items():
    cfg[k] = v.replace("<env>", env)

The Model: From Training Kitchen to Serving Station

Remember that Vertex AI training job that created our model (well, this is for another blog post)? It’s like a master chef preparing components in the main kitchen. Now we need to get that perfectly trained model to our serving station:

with open("model.joblib", "wb") as model_f:
    client.download_blob_to_file(
        f"{os.environ['AIP_STORAGE_URI']}/model.joblib", model_f
    )

_model = joblib.load("model.joblib")

This is like getting our special sauce from the central kitchen (Cloud Storage) where the training job left it. The AIP_STORAGE_URI environment variable is our delivery instructions, courtesy of Vertex AI!

Think of it as a relay race:

Training Job: “I’ve created this perfect model! Storing it in GCS!”
Cloud Run or Vertex Endpoint Service: “Got the location from AIP_STORAGE_URI!”
Model Loading: “Model loaded and ready to make predictions!”

# Get our environment settings
env_path = Path(__file__).parent.parent / ".env"
load_dotenv(env_path)
env = os.getenv("env")

# Set up our prediction server
app = FastAPI()
client = storage.Client(os.environ["project_id"])

# Download our pre-trained model
with open("model.joblib", "wb") as model_f:
    client.download_blob_to_file(
        f"{os.environ['AIP_STORAGE_URI']}/model.joblib",
        model_f
    )

The Happy Ending

Thanks to FastAPI and Pydantic, we’ve created a ML model serving solution that’s like a restaurant that makes both fine diners and casual brunchers feel at home. The model happily predicts NFL touchdowns while BigQuery and Vertex AI are think they’re getting special treatment!

In the end, it’s not about the route you take, it’s about the predictions you make.

A good ML service is like a well-run restaurant - the customers don’t need to know about the complex kitchen operations, they just enjoy the consistent, delicious predictions.

GCP Serverless prediction service using Cloud Run from a Vertex AI model

Sat, 15 Feb 2025 12:00:00 GMT

Deploying ML Models on Cloud Run: From Placeholder to Production

This guide walks through deploying a machine learning model on Google Cloud Run, using a two-phase deployment strategy: placeholder service followed by model deployment.

Architecture Overview

The architecture connects these components:

Vertex AI for training and model registry
Model artifacts stored in Cloud Storage
Cloud Run service running our prediction container
BigQuery for making predictions via remote functions (not discussed in this blog post)

Prerequisites

I have built this on my Windows development machine with WSL2, VScode and Docker Desktop. The are some of the other requirements on your local development machine.

Google Cloud Project with required APIs enabled
Terraform >= 1.0
Python >= 3.9
gcloud CLI configured

Phase 1: Placeholder Service

First, we deploy a minimal Nginx container that serves a simple status endpoint. This helps us establish our infrastructure and permissions before deploying the actual model.

Placeholder Container Structure

Our placeholder service uses Nginx to serve a static JSON response:

📁devops/serve/placeholder/
├── Dockerfile
├── nginx.conf
└── status.json

server {
    listen 8080;
    server_name localhost;

    location / {
        root /usr/share/nginx/html;
        try_files /status.json =404;
        default_type application/json;
    }
}

Initial Deployment

The placeholder is deployed via Terraform:

resource "google_cloud_run_service" "cr_nfl_predict" {
  name     = "nfl-touchdown"
  location = var.region

  template {
    spec {
      containers {
        image = "${var.region}-docker.pkg.dev/prj-xyz-shr-rep-0/rpo-bld-dkr-0/placeholder:latest"
        // ...configuration...
      }
    }
  }
}

ModelPorter: A Custom MLOps CLI

ModelPorter is our custom command-line tool that orchestrates the ML model lifecycle across Google Cloud Platform services. It simplifies the complex workflow of training, deploying, and serving machine learning models.

Why We Built It

Traditional ML deployment often involves juggling multiple cloud services, configuration files, and deployment steps. For our NFL touchdown prediction model, we needed to:

Build and push Docker containers for both training and prediction
Manage training jobs on Vertex AI
Deploy prediction endpoints to Cloud Run
Handle environment-specific configurations
Maintain consistent model versioning

Rather than managing these steps manually or with complex shell scripts, ModelPorter provides a unified interface.

Key Features

# Build containers
modelporter --env=dev nfl.touchdown build --phase=predict

# Train models
modelporter --env=dev nfl.touchdown train --service=vertex

# Deploy prediction service
modelporter --env=dev nfl.touchdown serve --service=cloudrun

The CLI handles:

Environment-specific configuration management
Container image building and pushing
Vertex AI training job orchestration
Cloud Run service updates
Model artifact management

Under the Hood

ModelPorter uses several key technologies:

Click: For CLI interface and command structure
Hydra: For configuration management
Cloud Build API: For container image building
Vertex AI API: For model training orchestration
Cloud Run API: For deployment management

Each command in ModelPorter maps to a specific workflow:

Build Command: Packages model code into containers

# Key environment variables set during build
buildargs = {
    "env": env,
    "project_id": cfg.project_id
}

Train Command: Manages Vertex AI training jobs

# Vertex AI training configuration
job = aiplatform.CustomContainerTrainingJob(
    display_name=f"{display_name}-train-job",
    container_uri=container_uri,
    model_serving_container_image_uri=model_serving_container_image_uri
)

Serve Command: Updates Cloud Run services

# Cloud Run service configuration
service = {
    "template": {
        "containers": [{
            "image": model_serving_container_image_uri,
            "env": [
                {"name": "AIP_STORAGE_URI", "value": model.uri},
                {"name": "AIP_PREDICT_ROUTE", "value": "/predict"}
            ]
        }]
    }
}

This abstraction allows our team to focus on model development rather than deployment mechanics, while maintaining consistency across environments.

Phase 2: Model Training and Deployment

Training on Vertex AI

Using our custom modelporter CLI tool:

modelporter --env=dev nfl.touchdown train --service=vertex

This command:

Initiates a training job on Vertex AI
Saves the trained model to GCS bucket
Registers the model in Vertex AI registry

No existing model found with display name 'nfl-touchdown'. Creating a new model.
⠴ Training model nfl.touchdown is training on vertex....Training Output directory:
gs://bkt-xyz-dev-nfl-vertex-0/aiplatform-custom-training-2025-02-28-16:29:02.387
⠼ Training model nfl.touchdown is training on vertex....View Training:
https://console.cloud.google.com/ai/platform/locations/australia-southeast1/training/<training-job-id>?project=<project-number>
⠸ Training model nfl.touchdown is training on vertex....CustomContainerTrainingJob projects/<project-number>/locations/australia-southeast1/trainingPipelines/<training-job-id> current state:
PipelineState.PIPELINE_STATE_RUNNING
View backing custom job:
https://console.cloud.google.com/ai/platform/locations/australia-southeast1/training/<id>?project=<project-number>
⠴ Training model nfl.touchdown is training on vertex....CustomContainerTrainingJob projects/<project-number>/locations/australia-southeast1/trainingPipelines/<training-job-id> current state:
PipelineState.PIPELINE_STATE_RUNNING
⠙ Training model nfl.touchdown is training on vertex....CustomContainerTrainingJob run completed. Resource name: projects/<project-number>/locations/australia-southeast1/trainingPipelines/<training-job-id>
⠴ Training model nfl.touchdown is training on vertex....Model available at projects/<project-number>/locations/australia-southeast1/models/<model-id>

The Training pipeline results in a registered model in Vertex AI.

As well as the model artifact in storage.

Deploying the Model Service

Once training is complete, we deploy our FastAPI prediction service:

modelporter --env=dev nfl.touchdown serve

Behind the scenes, modelporter:

Locates the latest trained model in Vertex AI
Updates the Cloud Run service with our prediction container
Configures environment variables to point to the model in GCS

Waiting for operation to complete...
Cloud Run service updated. URL: https://nfl-touchdown-<>.a.run.app

Once this has finished, if we browse to the Cloud Run service in the web console, and look at the Revisions, we can see a new revision taking 100% of the traffic.

# Key environment variables set during deployment
env = [
    {"name": "AIP_STORAGE_URI", "value": model.uri},
    {"name": "AIP_PREDICT_ROUTE", "value": "/predict"},
    {"name": "PROJECT_ID", "value": project_id}
]

Testing the Deployment

You can verify the deployment first getting an identity token to authenticate to the Cloud Run API:

# Get an identity token to pass as a bearer
gcloud auth print-identity-token

I like to use Thunder Client (The VSCode extension to test constructing API calls).

Thunder Client can also help you generate the programmatic calls in many languages.

# Make a prediction
curl -X POST \
  'https://nfl-touchdown-<project-id>.<region>.run.app/predict' \
  --header 'Accept: */*' \
  --header 'User-Agent: Thunder Client (https://www.thunderclient.com)' \
  --header 'Authorization: Bearer <identity-token>' \
  --header 'Content-Type: application/json' \
  --data-raw '{"instances":[[23.0, 150.0, 1.0, 20.0]]}'

Inspecting the Google Cloud Logs Explorer we can see the service was started and called successfully, confirming the response we got in Thunder Client was indeed returned from the Cloud Run service.

Infrastructure as Code

All base Cloud infrastructure components are managed via Terraform:

Cloud Run service configuration
IAM permissions
BigQuery connections
Storage buckets

This ensures consistent deployments across environments and makes it easy to replicate the setup.

terraform init -backend-config="vars/backend-dev.hcl"

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Using previously-installed hashicorp/google v4.51.0
- Using previously-installed hashicorp/random v3.7.1

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The complete solution demonstrates a production-ready ML serving infrastructure that’s both scalable and maintainable.

Checking the public IP address of a GCP Cloud Function

Mon, 04 Nov 2024 12:00:00 GMT

Public IP address of a GCP Cloud Function

When you deploy a Cloud Function without configuring a VPC, it operates within Google’s internal network. This means it leverages Google’s infrastructure to handle incoming and outgoing traffic.

When your function needs to make outbound requests to external services (e.g., HTTP requests), it will use Google’s internal network to reach those services.
The specific IP addresses used for outbound traffic will change over time.

This blog post details how to check this dynamic IP address (and log it to a database).

The source code can be found in the following repository https://github.com/mortie23/ml/tree/master/eng/misc/fn-misc-ipchecker.

Directory structure

We will start with the standard directory structure for this GCP Cloud Functions following on from the previous post of developing a GCP Cloud Function.

📁 fn-misc-ipchecker/
├── 📁 src/
│   │   📄 __init__.py
│   │   📄 misc.yaml
│   │   📄 main.py
│   │   📄 requirements.txt
├── 📁 tests/
│   │   📄 test_fn-misc-ipchecker.py
│   📄 deploy.sh
│   📄 poetry.lock
│   📄 pyproject.toml
│   📄 pytest.ini
📄 .env

How it works

The configuration file defines the BigQuery database (project), schema (dataset) and table where to log the IP addresses and the base URL of the service that returns the IP address that made the HTTP request.

project_id: 'prj-xyz-<env>-misc-ip-1'
dataset_name: 'demo'
table_name: 'ipchecker'
base_url: 'https://api64.ipify.org/?format=json'

The Python function simple makes the HTTP request and consumes and extracts the IP address that is returned.

def ip_checker() -> str:
    """Call a service that returns the calling network IP address

    Returns:
        str: The IP Address
    """
    data = requests.get(cfg.base_url)
    ip_address = json.loads(data.content.decode("utf-8"))["ip"]
    return ip_address

Running the source code in interactive

As a test, running the source code main.py script from the developer machine is a good way to have a comparison IP address logged to the BigQuery table.

Development and Deployment

Given this function is developed and deployed in the same way as a previous blog post, to avoid duplication, I will just reference it here: GCP Cloud Function, Development process and tooling.

However, this time I will outline the process to deploy the infrastructure using the GCP Terraform provider as opposed to the Gcloud SDK CLI.

Infrastructure as code

To ensure that all required infrastructure is deployed, we will use a Infrastructure as Code (IaC) solution, namely Terraform. The below example is from the main.tf file and shows how the Google Terraform provider is required and how the Google provider is authenticated with the Application Default Credentials (ADC).

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.51.0"
    }
  }
}

provider "google" {
  credentials = file("~/.config/gcloud/application_default_credentials.json")
  region      = var.region
}

resource "google_project" "misc_ip_1" {
  name            = "prj-xyz-${var.env}-misc-ip-1"
  project_id      = "prj-xyz-${var.env}-misc-ip-1"
  billing_account = var.billing_account
}

Requirements

I am using a WSL2 installation of the Ubuntu 20.04 distro. I have previously installed the following requirements on the machine.

Terraform https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
Gcloud SDK https://cloud.google.com/sdk/docs/install#deb

The GCP Terraform provider uses an Application Default Credential (ADC) to authenticate to GCP as we mentioned before. Use the following to create yours.

# Create an ADC JSON file
gcloud auth application-default login

Initialising the backend

The next step is to re-initialise the backend.

re-initialises and configures a Terraform working directory
identifies the providers and modules used in your configuration files and downloads them
loads up variables used in the Terraform configuration for managing different environments

# Change to the Terraform working directory
cd infra/tf/
# Initialise
terraform init -reconfigure -var-file="env-dev.tfvars"

Plan and Apply

First we run plan which generates a detailed execution plan. The plan let’s us know the changes that Terraform will make to the cloud infrastructure before we apply them, including:

creating (add) new resources,
modifying (change) existing ones, or
deleting (destroy) unnecessary resources

terraform plan -var-file="env-dev.tfvars"

Example output:

google_project.misc_ip_1: Refreshing state... [id=projects/prj-xyz-dev-misc-ip-1]
...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # google_project.misc_ip_1
-/+ resource "google_project" "misc_ip_1" {
      ~ id                  = "projects/prj-xyz-dev-misc-ip-1" -> (known after apply)
      - labels              = {} -> null
      ~ name                = "prj-xyz-dev-misc-ip-1"
      ~ number              =
      ~ project_id          = "prj-xyz-dev-misc-ip-1"
    }
...

Plan: 17 to add, 0 to change, 17 to destroy.

The above plan let’s us know that 17 resources will be added, and what they are. Assuming we are happy with the plan, we will continue with the apply step, which creates all the infra defined in the configurations.

terraform apply -auto-approve -var-file="env-dev.tfvars"

Once a successful apply has finished, you can log into the Cloud console and check that the resources, such as the new project and/or new BigQuery dataset have been created.

Test calling the function

Once you have deployed the required infrastructure and the Cloud Function it is now time to run it. To call the Cloud Function we can use a basic curl command. Ensure you are logged in as a user that has invoke permissions.

curl https://australia-southeast1-prj-xyz-dev-misc-ip-1.cloudfunctions.net/fn-misc-ipchecker-0 \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)"

If you check the target table, it should include an extra row with the public IP address that was used to make the external HTTP request. The first record is from the successful run of the source code from the development machine, which will log the public IP address of the development machines network.

Where does the IP address geo locate to

This result could be obvious, but I found it interesting none the less.

Testing

To run the unit tests during development we can run this:

pytest ./tests/test_fn-misc-ipchecker.py::test_ip_checker

====================== test session starts ===========================
platform linux -- Python 3.10.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /mnt/c/git/github/mortie23/ml/eng/misc/fn-misc-ipchecker
configfile: pytest.ini
plugins: env-1.1.5, hydra-core-1.3.2
collected 1 item

tests/test_fn-misc-ipchecker.py .                               [100%]

====================== 1 passed in 2.72s =============================

If programming languages were people (thoughts from a Generative AI)

Fri, 25 Oct 2024 00:00:00 GMT

Can you relate to these?

Inspired from these (https://leftoversalad.com/c/015_programmingpeople/), let’s see what a Generative AI thinks of what personifications of programming languages might be.

The following text and images were all generated with AI by Meta

General purpose

Python

Name: Clement (Clem for short)
Age: Late 20s
Personality: Easy-going, versatile, and adaptable
Appearance: Casual, laid-back attire (hoodie, jeans)
Interests: Data science, machine learning, web development, automation
Skills: Quick learner, problem-solver, team player
Quirks: Loves indentation, hates unnecessary complexity
Favorite hangouts: Data science conferences, hackathons, research labs

Clem is the friendly, approachable tech enthusiast. Always willing to lend a helping hand or share knowledge.

Python: versatile, easy-to-learn, and widely applicable

JavaScript

Name: Jax (short for JavaScripton)
Age: Early 20s
Personality: Dynamic, energetic, and creative
Appearance: Trendy, edgy attire (streetwear, bold accessories)
Interests: Web development, frontend engineering, gaming, VR
Skills: Quick thinker, innovative problem-solver, master of async
Quirks: Loves callbacks, hates IE browser compatibility issues
Favorite hangouts: Tech meetups, hackathons, gaming communities

Jax is the vibrant, entrepreneurial coder. Always pushing boundaries and exploring new possibilities.

JavaScript: dynamic, innovative, and web-centric

C#

Name: Jane
Age: Polished professional (30s)
Personality: Modern, efficient, and business-savvy
Appearance: Tailored suit, sleek tech gadgets
Interests: Enterprise software, web development, mobile apps
Skills: Expert in .NET framework, master of LINQ
Quirks: Loves garbage collection, hates manual memory management
Favorite hangouts: Corporate offices, tech meetups

Jane is the sophisticated, enterprise-focused developer. Always prioritizing productivity.

C#: enterprise-focused, modern, efficient, .NET framework

C++

Name: Axel (Cpp)
Age: Ambitious young adult (20s)
Personality: Powerful, complex, and assertive
Appearance: Edgy, avant-garde attire (black leather, piercings)
Interests: Game development, high-performance computing, systems programming
Skills: Template metaprogramming master, performance optimization expert
Quirks: Loves operator overloading, hates unnecessary constraints
Favorite hangouts: Gaming communities, tech conferences

Axel is the bold, cutting-edge developer. Always pushing boundaries.

C++: high-performance, complex, game development

Java

Name: Jake (The Architect)
Age: Established/Professional (40s)
Personality: Structured, organized, and meticulous
Appearance: Business attire (suit, tie), clean-cut hair
Interests: Enterprise software, Android app development, scalability
Skills: Master of object-oriented design, expert in multithreading
Quirks: Loves stability, hates reinventing the wheel
Favorite hangouts: Oracle conferences, Java user groups

Jake is the seasoned, methodical enterprise developer. Always building robust systems.

Java: structured, scalable, enterprise-focused

Maths and stats and data science

R

Name: Darren (Daz for short)
Age: Late-20s
Personality: Analytical, meticulous, and detail-oriented
Appearance: Professional, polished (business attire)
Interests: Statistics, data visualization, research, academia
Skills: Expert data wrangler, visualization master, research ninja
Quirks: Loves tidy data, hates messy code
Favorite hangouts: Academic conferences, research institutions, statistical meetups

Darren is the sharp, insightful statistician. Always focused on precision and accuracy.

R: precise, analytical, and research-focused

SAS

Name: Glendon (Glen for short)
Age: Late 40s
Personality: Conservative, reliable, and detail-oriented
Appearance: Classic business attire (suit, pearls)
Interests: Data analysis, reporting, pharmaceuticals, finance
Skills: Expert data manipulation, precise reporting
Quirks: Loves structured data, hates unformatted code
Favorite hangouts: Corporate offices, conference rooms

Glen is the seasoned, no-nonsense professional. Always prioritizing accuracy and precision.

SAS: reliable, precise, and industry-focused (pharmaceuticals, finance)

Matlab

Name: Matilda (Maddie)
Age: Late 20s (with a grad student vibe)
Personality: Intellectual, creative, and slightly introverted
Appearance: Academic attire (fleece, jeans)
Interests: Numerical computing, signal processing, research
Skills: Expert numerical analyst, algorithm designer
Quirks: Loves vectorized operations, hates loop optimizations
Favorite hangouts: Research labs, academic seminars

Matilda is the brilliant, analytical thinker. Always exploring new mathematical concepts.

Matlab: intellectual, creative, and research-oriented (numerical computing, signal processing)

Haskell

Name: Helena (The Theorist)
Age: Intellectual/Philosophical (30s)
Personality: Logical, abstract, and innovative
Appearance: Academic attire (turtleneck, glasses), eclectic accessories
Interests: Functional programming, type theory, category theory
Skills: Master of lazy evaluation, expert in monads
Quirks: Loves elegance, hates imperative programming
Favorite hangouts: Academic conferences, functional programming meetups

Helena is the brilliant, theoretical functional programming enthusiast. Always exploring new concepts.

Haskell: logical, abstract, functionally elegant

Databases

SQL

Name: Sammy (SQLicious)
Age: Mid-30s
Personality: Logical, methodical, and direct
Appearance: Simple, functional attire (button-down, khakis)
Interests: Database management, querying, data modeling
Skills: Master query writer, database optimizer
Quirks: Loves normalization, hates data redundancy
Favorite hangouts: Database conferences, online forums

Sammy is the straightforward, analytical expert. Always seeking efficient data solutions.

SQL: logical, methodical, and database-centric

Mostly Web

HTML

Name: Hollis (Ollie)
Age: Structured/Organized (25)
Personality: Logical, systematic, and foundational
Appearance: Clean-cut, minimalist attire (button-down, slacks)
Interests: Web development, structure, accessibility
Skills: Master of semantic markup, expert in document flow
Quirks: Loves order, hates ambiguity
Favorite hangouts: Web development forums, W3C meetups

Hollis is the meticulous, detail-oriented web architect. Always building solid foundations.

HTML: structural, logical, foundational

CSS

Name: Cassie (Stylista)
Age: Fashionable/Creative (28)
Personality: Artistic, expressive, and versatile
Appearance: Trendy, stylish attire (designer clothing, bold accessories)
Interests: Web design, visual effects, user experience
Skills: Expert in layout, master of color theory
Quirks: Loves aesthetics, hates bland designs
Favorite hangouts: Design conferences, web design communities

Cassie is the creative, visually-oriented web designer. Always styling and refining.

CSS: artistic, expressive, visual

PHP

Name: Pete (PHPster)
Age: Seasoned/Pragmatic (35)
Personality: Dynamic, flexible, and server-side savvy
Appearance: Comfortable, practical attire (t-shirt, jeans)
Interests: Web development, server-side scripting, database integration
Skills: Master of dynamic content, expert in database interactions
Quirks: Loves flexibility, hates unnecessary complexity
Favorite hangouts: Web development forums, LAMP meetups

Pete is the experienced, pragmatic web developer. Always dynamically solving problems.

This personification reflects their characteristics, strengths, and typical use cases:

PHP: dynamic, flexible, server-side focused

Low Level

C

Name: Charlie (Cee)
Age: Grizzled veteran (50s)
Personality: Pragmatic, efficient, and no-nonsense
Appearance: Weathered denim, leather jacket
Interests: Operating systems, embedded systems, performance optimization
Skills: Master of memory management, expert coder
Quirks: Loves pointers, hates unnecessary abstraction
Favorite hangouts: Basement labs, hackathons

Charlie is the seasoned, battle-hardened programmer. Always focusing on raw performance.

C: performance-oriented, low-level, embedded systems

Rust

Name: Ruby (Rusty)
Age: Environmentalist/Outdoor enthusiast (late 20s)
Personality: Safety-conscious, resourceful, and meticulous
Appearance: Eco-friendly attire (recycled materials, outdoor gear)
Interests: Systems programming, security, sustainable coding
Skills: Memory safety expert, performance optimizer
Quirks: Loves borrow checker, hates data races
Favorite hangouts: Open-source communities, hacking meetups

Ruby is the diligent, earthy systems programmer. Always prioritizing safety and efficiency.

Rust: safety-focused, systems programming, performance optimization

Go

Name: Gideon (Goody)
Age: Efficient productivity enthusiast (mid-30s)
Personality: Simple, pragmatic, and concurrent
Appearance: Minimalist attire (clean lines, monochromatic)
Interests: Network programming, cloud computing, scalability
Skills: Goroutine master, performance optimizer
Quirks: Loves simplicity, hates unnecessary complexity
Favorite hangouts: Cloud conferences, DevOps meetups

Gideon is the streamlined, productivity-focused developer. Always optimizing for efficiency.

Go: efficient, concurrent, scalable, cloud computing

Zig

Name: Zara (Zee)
Age: Rebellious/Nonconformist (early 20s)
Personality: Fearless, experimental, and liberated
Appearance: Counterculture attire (bold, eclectic)
Interests: Systems programming, embedded systems, compiler design
Skills: Master of compile-time evaluation, performance hacker
Quirks: Loves generics, hates runtime overhead
Favorite hangouts: Alternative tech communities, hackathons

Zara is the bold, unconventional systems programmer. Always pushing boundaries.

Zig: experimental, liberated, compile-time evaluation, performance hacking

Assembly

Name: Axel (The Tinkerer)
Age: Grizzled/Veteran (50s)
Personality: Low-level, practical, and resourceful
Appearance: Wearable tech, utilitarian attire (overalls, goggles)
Interests: Embedded systems, reverse engineering, optimization
Skills: Expert in machine code, master of memory management
Quirks: Loves efficiency, hates unnecessary abstraction
Favorite hangouts: Hacker gatherings, embedded systems meetups

Axel is the battle-hardened, hands-on embedded systems expert. Always optimizing performance.

Assembly: low-level, efficient, embedded systems-oriented

Scripting

Bash

Name: Barney (Bashy)
Age: Seasoned veteran (40s)
Personality: Practical, straightforward, and reliable
Appearance: Comfortable, worn-in attire (flannel, jeans)
Interests: System administration, scripting, Unix philosophy
Skills: Master of pipes, expert in command-line fu
Quirks: Loves brevity, hates unnecessary complexity
Favorite hangouts: Unix forums, command-line meetups

Barney is the dependable, no-frills sysadmin. Always getting the job done.

Bash: reliable, practical, Unix-centric

PowerShell

Name: Dale (PowerDale)
Age: Ambitious professional (30s)
Personality: Modern, efficient, and integrated
Appearance: Business-casual attire (button-down, khakis)
Interests: Windows administration, DevOps, automation
Skills: Expert in cmdlets, master of pipeline management
Quirks: Loves consistency, hates outdated methods
Favorite hangouts: Microsoft conferences, DevOps meetups

Dale is the polished, results-driven Windows admin. Always streamlining processes.

PowerShell: modern, efficient, Windows-focused

Perl

Name: Penny (Perlie)
Age: Quirky/Artistic (20s)
Personality: Versatile, expressive, and slightly eccentric
Appearance: Eclectic attire (vintage clothing, colorful hair)
Interests: Text processing, regex, open-source activism
Skills: Master of pattern matching, expert in scripting
Quirks: Loves complexity, hates boredom
Favorite hangouts: Open-source communities, hacker gatherings

Penny is the creative, unconventional problem-solver. Always finding innovative solutions.

Perl: versatile, expressive, text-processing expert

Debugging Dataform with Node.js in VScode

Mon, 16 Sep 2024 12:00:00 GMT

Debugging Dataform with Node.js in VScode

Debugging Dataform JavaScript

If you happen to be a data engineer who is not familiar with the Node.js ecosystem, and end up having to debug complex JavaScript in a BigQuery Dataform code base might be feeling lost.

I will try to explain how I debug JavaScript and maybe you will find this helpful.

Run and Debug in VSCode

Let’s start by clicking on the Run and Debug tab in VScode and opening a JavaScirpt Debug Terminal.

When you click on this you will get a second terminal. This will be below any other terminals you may already have open.

Now, we can add debugger; statements within our JavaScript code for breakpoints.

In Dataform, we then need to run a compile within the Debug Terminal. We also need to pass the timeout option to a high time (otherwise it will error out before we drop into debug mode).

dataform compile --timeout 10m

Now we can inspect the state of variables, step through lines of code and use the Debug Console to test lines of JavaScript code.

The above screenshot of VSCode debugger shows the debugger actions at the top, highlights the current line of code, and the state of all variables in different context (global, local etc). Down the bottom we have a console where we can type in JavaScript commands to test them.

Try cloning this example repo and testing it out for yourself. I have left the debugger; statement in this SQLX file.

config {
  type: "table",
  name: "debug_example",
  schema: dataform.projectConfig.vars.ref,
}

js {
    // Place this where you need to debug
    debugger;
    const simple_sql = `select 'hello world' as test_column`
}

${ simple_sql }

GCP Cloud Functions for Web Scraping AFL data

Fri, 02 Aug 2024 12:00:00 GMT

Scraping web data using GCP Cloud Functions

The main reason for this blog post is a practical example of writing a GCP Cloud Function with in the box features of:

Tests
Configurations
Environment management for deployment
The best sport in the world (AFL)

The source code can be found in the following repository https://github.com/mortie23/ml/tree/master/eng/afl/fn-afl-teams.

Directory structure

We will start with a standard directory structure for each of our Cloud Functions like below.

📁 fn-afl-teams/
├── 📁 src/
│   │   📄 __init__.py
│   │   📄 afl.yaml
│   │   📄 main.py
│   │   📄 requirements.txt
├── 📁 tests/
│   │   📄 test_fn-afl-teams.py
│   📄 deploy.sh
│   📄 poetry.lock
│   📄 pyproject.toml
│   📄 pytest.ini
📄 .env

Development process and tooling

In this example I am running on a Windows machine with WSL (Ubunutu). The first step it so install all packages on a local Python virtual environment (venv).

# create the venv. assuming created within a venv directory on your home directory
python3 -m venv ~/venv/fn_afl_teams
source ~/venv/fn_afl_teams/bin/activate
# install poetry to manage the remaining packages
pip install poetry

Now we use poetry to install everything else from the pyproject.toml file.

# install all packages to the venv
poetry lock && poetry install --no-root
# register the ipykernel for the user
python3 -m ipykernel install --user --name fn_afl_news

Given this Cloud Function writes to BigQuery, we need to authenticate the gcloud SDK on our WSL installation.

gcloud auth login
gcloud atuh application-default login

With the ./src/main.py Python script open in VScode, and ensure the active kernel is the venv.

Run the script in interactive.

When in interactive mode in VScode, another good way to explore any resulting dataframes is using the Data Wrangler extension:

If you have to add any new packages while you are developing, you add them using poetry:

poetry add pandas

Deploy

If you have had to add or update any packages in the pyproject.toml then export to the requirements file before you deploy. The Cloud Build step when deploying a Python based Cloud Functions depends on a requirements.txt file in the source directory.

poetry export --without-hashes --format=requirements.txt > ./src/requirements.txt

To deploy the funciton run the deploy script passing the target environment.

./deploy.sh --env dev

Preparing function...done.
✓ Deploying function...
  ✓ [Build] Logs are available at [https://console.cloud.google.com/cloud-build/builds;region=australia-southeast1/xyz]
  ✓ [Service]
  ✓ [ArtifactRegistry]
  ✓ [Healthcheck]
  ✓ [Triggercheck]
Done.

Calling

To test calling the Cloud Function we can use a basic curl command. Ensure you are logged in as a user that has invoke permissions.

curl https://australia-southeast1-prj-xyz-dev-fruit-0.cloudfunctions.net/fn-afl-teams-0 -H "Authorization: Bearer $(gcloud auth print-identity-token)"

If you check the target table, it should now include data from the scraped API.

Testing

To run the unit tests during development we can run this:

pytest ./tests/test_fn-afl-teams.py::test_webscrape_teams

============================ test session starts =============================
platform linux -- Python 3.10.7, pytest-8.3.2, pluggy-1.5.0
rootdir: /mnt/c/git/github/mortie23/ml/eng/afl/fn-afl-teams
configfile: pytest.ini
plugins: env-1.1.3, hydra-core-1.3.2
collected 1 item

tests/test_fn-afl-teams.py .                                            [100%]

============================= 1 passed in 1.24s ==============================

IAM and other infrastructure

The following is some of the prerequisites required for BigQuery, Storage Buckets and IAM permissions for developers (members of the gcp-developers group) and service account (serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com).

Creation of BigQuery dataset for Horizon Scanning

bq mk --location=australia-southeast1 --project_id=prj-xyz-dev-fruit-0 afl

Setup of the Cloud Build things

Create a custom bucket for the source code

gcloud storage buckets create --location=australia-southeast1 --project=prj-xyz-dev-fruit-0 gs://bkt-xyz-dev-afl-fn-0

Create service accounts for Cloud Functions and Cloud Build

# Create a service account
gcloud iam service-accounts create sa-xyz-dev-fruit-cf --display-name "Cloud Function Service Account"
# Allow gcp-developers to act as the Cloud Function service account for deployment
gcloud iam service-accounts add-iam-policy-binding sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com --member='group:gcp-developers@tremendousdomain.xyz' --role='roles/iam.serviceAccountUser'
# Allow developers to invoke functions
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member='group:gcp-developers@tremendousdomain.xyz' --role='roles/run.invoker'
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="group:gcp-developers@tremendousdomain.xyz" --role="roles/cloudfunctions.invoker"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="group:gcp-developers@tremendousdomain.xyz" --role="roles/cloudfunctions.developer"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="group:gcp-developers@tremendousdomain.xyz" --role="roles/cloudfunctions.viewer"
# Give service account permissions to all the things needed (might be too many permissions)
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/cloudfunctions.invoker"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/cloudfunctions.developer"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/cloudfunctions.viewer"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/logging.logWriter"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/artifactregistry.writer"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/storage.objectAdmin"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/cloudbuild.builds.builder"
# Cloud Functions use Cloud Run
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/run.admin"
# Needed for the BigQuery part after getting bigquery
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/bigquery.dataEditor"
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member="serviceAccount:sa-xyz-dev-fruit-cf@prj-xyz-dev-fruit-0.iam.gserviceaccount.com" --role="roles/bigquery.jobUser"

Flask application with custom package deployed to GCP Artifact Registry and Cloud Run

Thu, 18 Jul 2024 12:00:00 GMT

Flask application with custom package deployed to GCP Artifact Registry and Cloud Run

What we are looking to do in this example is demonstrate how you might develop a Python based web application that uses modules from a custom Python package that is contained within the same mono repo as the source code for the application.

The source code (https://github.com/mortie23/ml/) is same as from my previous post about Python mono repos for Data Science with Poetry.

Some of the benefits are:

Module reuse across exploratory data analysis (EDA), and many applications
Changes can be tested across the complete code base
Changes and associated deployment can be orchestrated for both the custom modules and applications with one feature branch

Our custom Python package Fruitbowl

First we will start with the super useful custom package we have called Fruit bowl, which has a Fruit module and function that returns a random name of a fruit. Just for demonstration purposes.

Publish to a GCP Artifact Registry Python package repository

Credit goes to Rodrigo Agundez who wrote this blog post:

GCP Python packages via GCPS Artifact Registry

# Update poetry
poetry self update
# Add Google artifact registry keyring to poetry
poetry self add keyrings.google-artifactregistry-auth
# Check the python repository on GCP
gcloud artifacts print-settings python --project=prj-xyz-dev-fruit-0 --repository=rpo-xyz-dev-fruit-pyn-0 --location=australia-southeast1
# Configure a poetry repository to point to the python artifact registry
poetry config repositories.gcp https://australia-southeast1-python.pkg.dev/prj-xyz-dev-fruit-0/rpo-xyz-dev-fruit-pyn-0/
# publish the package to the repository
poetry publish --build --repository gcp

The publish step should look like this if successful.

Using python3.12 (3.12.3)
Building fruitbowl (0.1.0)
  - Building sdist
  - Built fruitbowl-0.1.0.tar.gz
  - Building wheel
  - Built fruitbowl-0.1.0-py3-none-any.whl

Publishing fruitbowl (0.1.0) to gcp
 - Uploading fruitbowl-0.1.0-py3-none-any.whl 100%
 - Uploading fruitbowl-0.1.0.tar.gz 100%

The Hello Fruit Flask application

Now for the simple example of a Flask application that uses this Fruit bowl function from the custom Python module.

Setup

Firstly let’s just test if we can run it locally in development mode. In this mode we could test changing code from either the module or the application and see the changes.

# Change to the root of the application
cd app/flask-hellofruit/
# Install application dependencies using poetry
poetry install

Running the application

poetry run flask run

 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
127.0.0.1 - - [17/Jul/2024 21:03:36] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [17/Jul/2024 21:03:36] "GET /favicon.ico HTTP/1.1" 404 -

Hosting on GCP with Cloud Run

Hosting this on GCP has many options, but we are going to use Cloud Run to host this in a serverless way.

Build a Docker container image

We need the Docker build process to have access to the Artifact registry repository. For this we create a service account and associated key file that can be temporarily copied during the Docker build process.

# Create a service account
gcloud iam service-accounts create gsvc-xyz-dev-fruit-rpo --display-name "Artifact Registry Service Account"
# Give service account permissions to artifact registry in project
gcloud projects add-iam-policy-binding prj-xyz-dev-fruit-0 --member=serviceAccount:gsvc-xyz-dev-fruit-rpo@prj-xyz-dev-fruit-0.iam.gserviceaccount.com --role=roles/artifactregistry.reader
# Create application keys JSON file for service account
gcloud iam service-accounts keys create gsvc-xyz-dev-fruit-rpo.json --iam-account gsvc-xyz-dev-fruit-rpo@prj-xyz-dev-fruit-0.iam.gserviceaccount.com

Put the JSON keys file in this location on your developer client machine ~\.config\gcloud\gsvc-xyz-dev-fruit-rpo.json. In this case it is the home path on the WSL2 Windows client laptop. It will be used later in the hellofruit app Docker build step.

Now we can build the container image.

docker build . --build-arg SA_KEY_BASE64=`base64 -w0 ~/.config/gcloud/gsvc-xyz-dev-fruit-rpo.json` -t australia-southeast1-docker.pkg.dev/prj-xyz-dev-fruit-0/rpo-xyz-dev-fruit-dkr-0/app/hellofruit:0.1

Run the container locally.

docker run -p 5000:5000 australia-southeast1-docker.pkg.dev/prj-xyz-dev-fruit-0/rpo-xyz-dev-fruit-dkr-0/app/hellofruit:0.1

[2024-07-17 11:34:23 +0000] [1] [INFO] Starting gunicorn 22.0.0
[2024-07-17 11:34:23 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2024-07-17 11:34:23 +0000] [1] [INFO] Using worker: gthread
[2024-07-17 11:34:23 +0000] [7] [INFO] Booting worker with pid: 7
[2024-07-17 11:34:23 +0000] [8] [INFO] Booting worker with pid: 8

Once we have tested that it works locally we can push it to the Docker registry.

# First auth to the docker artifact registry
gcloud auth configure-docker australia-southeast1-docker.pkg.dev
# Push the local image to the artifact registry
docker push australia-southeast1-docker.pkg.dev/prj-xyz-dev-fruit-0/rpo-xyz-dev-fruit-dkr-0/app/hellofruit:0.1

Cloud Run the service

Using the GCP web console, it is simple to select the container from the Artifact Registry and set the Container port to 5000. The remaining settings are left as the default settings.

Once the Cloud Run service has completed, you should see the service running and be given a URL to access the service.

Next steps

Set up CICD for the deployment of both the package and the application.

Converting table schemas from any database to BigQuery for Dataflow

Wed, 01 May 2024 00:00:00 GMT

Why do this?

When using Dataflow Google-provided templates for any RDBMS to BigQuery (such as PostgreSQL to BigQuery) the target table in BigQuery has to already exist. It logically must have the correct column names and data types.

We want a way to automate the process of reading a source table schema and converting it to a BigQuery schema and then deploying it.

Example source system

For demonstration purposes we will use a Postgres database. It is hosted on a Cloud VM and the firewall rules have been set so that my local machine can access it as well as the Dataflow worker node.

I am using the HeidiSQL IDE to connect to one of the databases on the instance.

The table schema can be obtained from the information_schema.columns view.

BeamMeUp

Managing many table loads that use Dataflow Google-provided templates across many environments is not straight forward. One such issue is the one outlined in this blog post regarding the need to encrypt and encode credentials.

In an attempt to not repeat myself DRY while also trying to not over engineer things YAGNI, I developed a Python package with some reuse helper functions for working with Dataflow. I called it BeamMeUp.

One of the functions within BeamMeUp that this process relies on is the get_db_type_encoder function.

def get_db_type_encoder(
    db_type: str,
) -> pd.DataFrame:
    """Returns an encoder dataframe for source system to target system (BigQuery in this case)

    Args:
        db_type (str): The type of database (postgres or mssql)

    Returns:
        pd.DataFrame: a dataframe of the data type mappings
    """
    columns = ["db_type", "source_type", "target_type"]
    data = [
        ("mssql", "char", "string"),
        ...
        ("mssql", "date", "date"),
        ("mssql", "varchar", "string"),
        ("postgres", "character", "string"),
        ...
        ("postgres", "date", "date"),
    ]
    db_type_encoder = pd.DataFrame(data, columns=columns)
    return db_type_encoder[db_type_encoder["db_type"] == db_type]

DDL Helper

I developed a helper script that can be used for both PostgreSQL and Microsoft SQL Server source systems.

bqddl-postgres-from-source.py

python3 bqddl-postgres-from-source.py --env dev

[11:25:21] Creating table DDL from source 🚀  mortimer_nfl.weather       dbtools.py:409
               writing DDL to repo                                       dbtools.py:435

This will generate a script in appropriate folder within the repo.

📁 ddl/
│   ├── 📁 bigquery/
│   │   ├── 📁 fruit-0/
│   │   │   │   📄 nfl.weather.sql

create table nfl.weather (
`weather_game_id` int64
  ,`game_id` int64
  ,`temperature` int64
  ,`weather_condition` string
  ,`wind_speed` int64
  ,`humidity` int64
  ,`wind_direction` string
  ,`created_date` timestamp
  ,`create_user` string
)
;

We can now deploy this to the appropriate project in GCP using the --run_flag.

python3 bqddl-postgres-from-source.py --env dev --run_flag

[11:25:21] Creating table DDL from source 🚀  mortimer_nfl.weather       dbtools.py:409
[11:25:22]     writing DDL to repo                                       dbtools.py:435
               Deploying TABLE DDL to BigQuery                           dbtools.py:451

This will have dropped the table if it existed and created the new one ready to be loaded by Dataflow.

A way to build CRUD apps with R Shiny and SQLite

Wed, 01 May 2024 00:00:00 GMT

Background

Often as software engineers working within organisations we are coerced into using a software stack that we may not have chosen if we were tasked with designing it from the start. This is part of the fun and part of what keeps our jobs interesting. Also we may be able to come to the conclusion that it makes sense to utilise a component in a system design that is less than optimal for it’s specific use case, but makes sense overall.

Take R Shiny for example. An R centric framework for full stack web applications. From my limited exposure, common use cases for R Shiny are read only dashboards, but often we require mutations to data such as create, update and delete operations.

Shiny helps you turn your analyses into interactive web applications without requiring HTML, CSS, or JavaScript knowledge.

There is some really great work that has been done in this space including but not limited to:

Persistent data storage in Shiny apps
Barbara Borges Ribeiro’s presentation on Interacting with Databases from R and Shiny

Although there is a fair bit of content out there already on this, none of it exactly met my use case, so I decided to give a go at a basic scaffold of a R package for a CRUD application using R Shiny and SQLite.

Welcome ShineLite.

A basic lite CRUD application framework for RShiny application development with a SQlite back end.

The extra things I wanted to build into it were:

Environment life cycle to connect to different DBs based on development/production etc
Package dependencies with renv

Setup

OS binaries

I started developing this on a vanilla Ubuntu VM and tried to capture all the things needed to get it up and running.

# Run this from your terminal (not console)
# Install all these OS binaries or you will get caught during the renv restore
sudo apt install build-essential
sudo apt install libcurl4-openssl-dev
sudo apt install libfontconfig1-dev
sudo apt install libharfbuzz-dev libfribidi-dev
sudo apt install libfreetype6-dev libpng-dev libtiff5-dev libjpeg-dev
sudo apt install libxml2-dev

The SQLite DB

The package has a configuration function within R/config.R. It will return the path for where your SQLite DB file is hosted. In this case the path is /u01/data/shinelite/<env>/todo_app.db. This would be changed per use case. There is probably a better way to do this, but it works for now.

#' Set environment
#'
#' @return config
#' @export
#'
#' @examples set_environment()
set_environment <- function() {
  # Get the environment
  env <- Sys.getenv("env")

  # Define configurations for each environment
  dev_config <- list(
    db_path = "/u01/data/shinelite/dev/todo_app.db",
    other_config_variable = "value"
  )

  prd_config <- list(
    db_path = "/u01/data/shinelite/prd/todo_app.db",
    other_config_variable = "value"
  )

  # Set configuration based on environment
  if (env == "dev") {
    return(dev_config)
  } else if (env == "prd") {
    return(prd_config)
  } else {
    stop("Unknown environment specified")
  }
}

The next setup step is to initialise the SQLite DB in the development environment.

# Run this from your terminal (not console)
export env=dev
db_path="/u01/data/shinelite/dev/todo_app.db"
sqlite3 $db_path <<EOF
CREATE TABLE IF NOT EXISTS todos (
    id INTEGER PRIMARY KEY,
    todo TEXT,
    done BOOLEAN DEFAULT FALSE
);
EOF

The R environment

Create a .Renviron file in the root of the cloned repository and populate it with this:

env=dev

Start with the initial install of renv.

install.packages('renv')

Now install everything else with a renv restore

renv::restore()

We also need the devtools package to make all functions available.

devtools::load_all(".")

Run the RShiny application in dev mode

runShineLite()

Remember to add the two most important things to your new todo list:

Checking the DB

Just to show how to confirm that the DB was written to using the CLI.

Adding documentation (optional)

If you want to render out a site for your package you can use package down.

pkgdown::build_site()

KMS encrypted credentials with Dataflow on GCP

Fri, 19 Apr 2024 00:00:00 GMT

Encrypting credentials

Google KMS

Cloud Key Management Service (Cloud KMS) lets you create and manage encryption keys for use in compatible Google Cloud services and in your own applications. So first things first, we have to create a Google managed KMS keyring and associated key.

Google-provided Dataflow templates

Some background on why we are doing this. Google provides open source Dataflow templates that you can use instead of writing pipeline code.

If we were to use the PostgeSQL to BigQuery one for example, we find that the only way to provide the credentials in a secure way is to encrypt them with a Google managed KMS key.

https://cloud.google.com/dataflow/docs/guides/templates/provided/postgresql-to-bigquery#template-parameters

Parameter	Description
username	Optional: The username to use for the JDBC connection. You can pass in this value encrypted by a Cloud KMS key as a Base64-encoded string.
password	Optional: The password to use for the JDBC connection. You can pass in this value encrypted by a Cloud KMS key as a Base64-encoded string.

We have created a template shell script that will execute the GCloud CLI command for running a Dataflow job.

https://github.com/mortie23/beam-poetry-mono/blob/master/eng/dataflow-run/nfl-postgres-tandl/dtf-gcloud-template.sh

In this shell script, we have the credentials as templated parameters:

gcloud dataflow flex-template run dtf-postgres-nfl-tandl-<<job_name>> \
    ...
    --parameters connectionURL="<<connectionURL>>" \
    --parameters username="<<username>>" \
    --parameters password="<<password>>" \

But wow do we get the encrypted versions of these into the template.

KMS helper function

We have written a series of helper functions to make this easier for us. Firstly a custom Beam Me Up package has been written with a KMS module.

https://github.com/mortie23/beam-poetry-mono/blob/master/eng/lib/beammeup/beammeup/kms.py

The function within the module that is used to create the Base64 encoded encrypted credentials is the encode_dataflow_parameter function. Let’s look at what it does step by step.

# Uses another Beam Me Up function to encrypt any string value
encrypted_parameter = encrypt_symmetric(
    project_id=project_id,
    location_id=location_id,
    key_ring_id=key_ring_id,
    key_id=key_id,
    plaintext=parameter,
)
# Base64 encodes the returned ciphertext (which is in bytes) and then decodes the bytes to a string
base64_encrypted_parameter = base64.b64encode(
    encrypted_parameter.ciphertext
).decode("utf-8")

Let’s look at each of the parts of this to figure out what is happening. Dataflow jobs require a KMS encoded

Encrypted parameter

The result from the call to encrypt_symmetric is an object with multiple parts. We just need the ciphertext.

name: "projects/prj-xyz-prd-fruit/locations/australia-southeast1/keyRings/keyring-fruit/cryptoKeys/key-fruit/cryptoKeyVersions/1"
ciphertext: "\n$\000\304\227|\301\324\343z\227\227\206I\214^\310\326\301\234O`\243|A\n\342\275v\332|\314|(>\252;\243\022J\000\254\004\343\003\375[\3760\333\027>(\311|k\"\302\01426\334\207/\361\\\226-\001MP\375R0\017\270v\310\323\351\331\2140\217v2P9\242\333Y\262\307\225G\200\345a\373;)\257\260\263\344]1\323\322W\241\270J\t"
ciphertext_crc32c {
  value: 2931620773
}
verified_plaintext_crc32c: true
protection_level: SOFTWARE

Cipher text

The ciphertext itself is in bytes:

b'\n$\x00\xc4\x97|\xc1\xd4\xe3z\x97\x97\x86I\x8c^\xc8\xd6\xc1\x9cO`\xa3|A\n\xe2\xbdv\xda|\xcc|(>\xaa;\xa3\x12J\x00\xac\x04\xe3\x03\xfd[\xfe0\xdb\x17>(\xc9|k"\xc2\x0c26\xdc\x87/\xf1\\\x96-\x01MP\xfdR0\x0f\xb8v\xc8\xd3\xe9\xd9\x8c0\x8fv2P9\xa2\xdbY\xb2\xc7\x95G\x80\xe5a\xfb;)\xaf\xb0\xb3\xe4]1\xd3\xd2W\xa1\xb8J\t'

Base64 encoded

The result of Base64 encoding these bytes are bytes again.

b'CiQAxJd8wdTjepeXhkmMXsjWwZxPYKN8QQrivXbafMx8KD6qO6MSSgCsBOMD/Vv+MNsXPijJfGsiwgwyNtyHL/Fcli0BTVD9UjAPuHbI0+nZjDCPdjJQOaLbWbLHlUeA5WH7OymvsLPkXTHT0lehuEoJ'

Decoded

Decoding these bytes leaves us with a string we can pass into the GCloud CLI Dataflow template parameter.

'CiQAxJd8wdTjepeXhkmMXsjWwZxPYKN8QQrivXbafMx8KD6qO6MSSgCsBOMD/Vv+MNsXPijJfGsiwgwyNtyHL/Fcli0BTVD9UjAPuHbI0+nZjDCPdjJQOaLbWbLHlUeA5WH7OymvsLPkXTHT0lehuEoJ'

Filling in the parameters

A Python script is used that can source the Beam Me Up package and use our helper functions to encrypt a given credential from a .dotenv file and then replace the values in the shell script.

https://github.com/mortie23/beam-poetry-mono/blob/master/eng/dataflow-run/nfl-postgres-tandl/dataflow-generate-run.py#L174

encoded_username = encode_dataflow_parameter(
    project_id=cfg.project_id,
    location_id=cfg.location,
    key_ring_id=cfg.key_ring_id,
    key_id=cfg.key_id,
    parameter=username,
)

Python Beam and DataFlow to write data to BigQuery with a custom Docker image

Fri, 22 Dec 2023 00:00:00 GMT

Dataflow build and run

Basic example of how to develop an Apache Beam pipeline written in Python that writes data to BigQuery. To then build it as a flex template and then run it as a Dataflow job.

This example avoids complexities of reading from source systems or storage and simply creates synthetic data.

Working from these examples:

Setup

Reference code is in the following Github repo https://github.com/mortie23/beam-jdbc-testing/tree/master/docker.

The GCP prerequisites are in the following sections.

Project

Firstly the basic GCP project naming convention we will have will include an environment.

prj-<org>-<env>-<business-unit>

In this example, the Organisation (<org>) is xyz, and the Business unit (<business-unit>) is fruit

The project requires:

GCP resource
	GCP Artifact registry Docker repository
	DataFlow API
	Cloud storage
	BigQuery

IAM

Service account (gsvc-xyz-<env>-fruit@<project_id>.iam.gserviceaccount.com) with roles
- Artifact Registry Repository Administrator
- Storage Object Admin
- Dataflow Worker
- Workflows Invoker
- BigQuery custom role (permissions)
  - bigquery.tables.create
  - bigquery.tables.get
  - bigquery.tables.update
  - bigquery.tables.updateData

Cloud storage

Cloud storage bucket (bkt-xyz-<env>-fruit) with sub folders

📁 dataflow/
│   ├── 📁 flex-template/
│   ├── 📁 staging/
│   ├── 📁 logs/
│   ├── 📁 temp/

BigQuery

Make sure the target table within the appropriate dataset (fruit) exists.

create table fruit.hellofruit (
  name string
  , test_number int64
)

Build

The build step involves building a Docker container image, and associated Flex template file.

The Docker file:

starts FROM the base image
sets the required environment (ENV) variables
COPY the source files to the Docker image
RUN operating system updates and installs of packages as well as the Python packages
defines the ENTRYPOINT

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="/template/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="/template/hellofruit.py"

COPY . /template

# We could get rid of installing libffi-dev and git, or we could leave them.
RUN apt-get update \
    && apt-get install -y libffi-dev git \
    && rm -rf /var/lib/apt/lists/* \
    # Upgrade pip and install the requirements.
    && pip install --no-cache-dir --upgrade pip \
    && pip install --no-cache-dir -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE \
    # Download the requirements to speed up launching the Dataflow job.
    && pip download --no-cache-dir --dest /tmp/dataflow-requirements-cache -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE

# Since we already downloaded all the dependencies, there's no need to rebuild everything.
ENV PIP_NO_DEPS=True

ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"]

Options

Using the build script

The build script, will run all steps for you and do parameter replacement from the config file:

The config file sets out the parameters for the DataFlow job.

project_id: 'prj-xyz-<env>-fruit'
artifact_registry: 'australia-southeast1-docker.pkg.dev'
artifact_repo_docker: 'rpo-xyz-<env>-fruit-dkr'
storage_bucket: 'bkt-xyz-<env>-fruit'
service_account: 'gsvc-xyz-<env>-fruit'
location: 'australia-southeast1'

./dataflow-build.sh --env dev

Manual step by step

This is manually performing the steps from the bash script previously mentioned.

Build the Docker image on your local client development machine, tagging the image with the full reference to the GCP Artifact registry repository.

docker build . -t australia-southeast1-docker.pkg.dev/<project_id>/<artifact_repository_docker>/dataflow/hellofruit:0.1

The build step will take quite a bit of time, it has to pull the base image, copy the scripts and then install the dependencies etc.

Before you try to push the container image to the Artifact registry repository you need to authenticate your gcloud CLI with the artifact registry.

gcloud auth configure-docker australia-southeast1-docker.pkg.dev

Push the image to the registry.

docker push australia-southeast1-docker.pkg.dev/<project_id>/<artifact_repository_docker>/dataflow/hellofruit:0.1

Now we can build a flex template.

gcloud dataflow flex-template build "gs://<storage_bucket>/dataflow/flex-template/hellofruit.json" \
     --image "australia-southeast1-docker.pkg.dev/<project-id>/<artifact_repository_docker>/dataflow/hellofruit:0.1" \
     --sdk-language "PYTHON" \
     --metadata-file "metadata.json"

This should run quite quickly and result in a JSON file in the bucket location you requested.

Successfully saved container spec in flex template file.
Template File GCS Location: gs://<storage_bucket>/dataflow/flex-template/hellofruit.json
Container Spec:

{
    "defaultEnvironment": {},
    "image": "australia-southeast1-docker.pkg.dev/<project-id>/<artifact_repository_docker>/dataflow/hellofruit:0.1",
    "metadata": {
        "description": "Hello fruit Python flex template.",
        "name": "Hello fruit",
        "parameters": [
            {
                "helpText": "Name of the BigQuery output table name.",
                "isOptional": true,
                "label": "BigQuery output table name.",
                "name": "output_table",
                "regexes": [
                    "([^:]+:)?[^.]+[.].+"
                ]
            }
        ]
    },
    "sdkInfo": {
        "language": "PYTHON"
    }
}

Using the gcloud CLI

This method will do the build and push as well as create the flex template JSON file. However in certain GCP setups it may not run. Some of the magic it does requires certain permissions that may not be granted.

gcloud dataflow flex-template build "gs://<storage_bucket>/dataflow/flex/hellofruit.json" \
     --image-gcr-path "australia-southeast1-docker.pkg.dev/<project_id>/<artifact_repository_docker>/dataflow/hellofruit:latest" \
     --staging-location "gs://<storage_bucket>/dataflow/staging" \
     --temp-location "gs://<storage_bucket>/dataflow/temp" \
     --gcs-log-dir "gs://<storage_bucket>/dataflow/logs" \
     --sdk-language "PYTHON" \
     --flex-template-base-image "PYTHON3" \
     --metadata-file "metadata.json" \
     --py-path "." \
     --env "FLEX_TEMPLATE_PYTHON_PY_FILE=hellofruit.py" \
     --env "FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE=requirements.txt" \
     --log-http \
     --verbosity debug

Example error log

These are example logs that you might see if this fails for the reason such as you do not have storage bucket create access. Building it this way uses GCP Cloud Run which will attempt to create a temporary bucket with the naming convention <project_id>_cloudbuild

DEBUG: Running [gcloud.dataflow.flex-template.build]

uri: https://storage.googleapis.com/storage/v1/b/<project_id>_cloudbuild?alt=json
DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG: https://storage.googleapis.com:443 "GET /storage/v1/b/<project_id>_cloudbuild?alt=json HTTP/1.1" 404 247

{
  "error": {
    "code": 404,
    "message": "The specified bucket does not exist.",
    "errors": [
      {
        "message": "The specified bucket does not exist.",
        "domain": "global",
        "reason": "notFound"
      }
    ]
  }
}

uri: https://storage.googleapis.com/storage/v1/b?alt=json&enableObjectRetention=False&project=<project_id>
DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG: https://storage.googleapis.com:443 "POST /storage/v1/b?alt=json&enableObjectRetention=False&project=<project_id> HTTP/1.1" 403 546
{
  "error": {
    "code": 403,
    "message": "<user>@<domain> does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist).",
    "errors": [
      {
        "message": "<user>@<domain> does not have storage.buckets.create access to the Google Cloud project. Permission 'storage.buckets.create' denied on resource (or it may not exist).",
        "domain": "global",
        "reason": "forbidden"
      }
    ]
  }
}

DEBUG: (gcloud.dataflow.flex-template.build) The user is forbidden from accessing the bucket [<project_id>_cloudbuild]. Please check your organization's policy or if the user has the "serviceusage.services.use" permission. Giving the user Owner, Editor, or Viewer roles may also fix this issue. Alternatively, use the --no-source option and access your source code via a different method.

googlecloudsdk.command_lib.builds.submit_util.BucketForbiddenError: The user is forbidden from accessing the bucket [<project_id>_cloudbuild]. Please check your organization's policy or if the user has the "serviceusage.services.use" permission. Giving the user Owner, Editor, or Viewer roles may also fix this issue. Alternatively, use the --no-source option and access your source code via a different method.
ERROR: (gcloud.dataflow.flex-template.build) The user is forbidden from accessing the bucket [<project_id>_cloudbuild]. Please check your organization's policy or if the user has the "serviceusage.services.use" permission. Giving the user Owner, Editor, or Viewer roles may also fix this issue. Alternatively, use the --no-source option and access your source code via a different method.

Run your custom flex template

Running a Dataflow job using a built flex template and Docker container

Options

Using the Run script

Like the build script, the run script will run all steps for you and do parameter replacement from the config file.

./dataflow-run.sh --env dev --name hellofruit

Dataflow UI

Using the Dataflow user interface you can use the Create a job from template and browse to Custom template JSON that was created in the build step.

Given that our metadata.json configuration contained a parameter of output_table the first optional parameter is Output table.

Results

When the DataFlow job runs successfully you should see a Job Graph like this:

Checking the target BigQuery table shows successful rows written to the table (this is a result of multiple successful runs).

Notes

I personally found the development and debugging experience for this process very atypical. Nothing seemed to run smoothly and there were many error messages that were very difficult to debug. Lack of community support and lack of stack overflow questions and answers was definitely noticeable compared to other coding frameworks.

One example I had was that I had it all working one day, and the next day it stopped working with the following error message. I found a post that suggested this could be many reasons, and one suggestion was to change the location. I changed the location to a US location and it worked. However this might not be an option for all users.

Failed to start the VM, launcher-, used for launching because of
status code: UNAVAILABLE,
reason: One or more operations had an error:
  'operation----': [UNAVAILABLE] 'HTTP_503'..

Copying a table from Teradata to Google BigQuery

Tue, 03 Oct 2023 00:00:00 GMT

Copying a table from Teradata to Google BigQuery

For this demo I will be using a Windows PC including:

Teradata Studio
Teradata ClearScape
Google BigQuery
Google DataFlow

This will demonstrate the most basic copy of a table from a Teradata ClearScape instance to a Google Cloud BigQuery table.

It will not discuss things such as:

Automation of schema creation for the target table in BigQuery
Scheduling the pipeline
Handling drop and recreate or truncate and load of the target table
Other transformations

Getting a cloud hosted Teradata instance for testing these days is easy using the free trial of Teradata ClearScape.

Check environment

Once the environment is created, let’s use the API to check a few things available with the free trail. Click on Copy API Token.

Create a file called .env and paste it in that file like this.

API_TOKEN=<token>

Create the following Python script and install the required packages with pip and then run it.

import requests
import os
import json
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()
# Get the bearer token from the environment variables
API_TOKEN = os.getenv('API_TOKEN')

url = 'https://api.clearscape.teradata.com/environments'
headers = {
    'accept': 'application/json',
    'Authorization': f'Bearer {API_TOKEN}'
}
# Handle response
response = requests.get(url, headers=headers)
data = response.json()
formatted_data = json.dumps(data, indent=4)
print(formatted_data)

The response should look something like this.

[
  {
    "state": "RUNNING",
    "region": "us-central",
    "services": [
      {
        "name": "Jupyterlab",
        "state": "RUNNING",
        "url": "https://dev-gcp-copy-<id>.env.clearscape.teradata.com:8443/lab/tree/README.md?token=<token>",
        "credentials": []
      },
      {
        "name": "Vantage",
        "state": "RUNNING",
        "url": "bteq .logon dev-gcp-copy-<id>.env.clearscape.teradata.com/demo_user",
        "credentials": [
          {
            "name": "username",
            "value": "demo_user"
          }
        ]
      },
      {
        "name": "Query Service",
        "state": "RUNNING",
        "url": "https://dev-gcp-copy-<id>.env.clearscape.teradata.com:1443/",
        "credentials": [
          {
            "name": "username",
            "value": "demo_user"
          }
        ]
      },
      {
        "name": "ModelOps",
        "state": "RUNNING",
        "url": "https://dev-gcp-copy-<id>.env.clearscape.teradata.com:8443/modelops",
        "credentials": [
          {
            "name": "username",
            "value": "demo_user"
          }
        ]
      }
    ],
    "name": "dev-gcp-copy",
    "ip": "<ip-address>",
    "dnsName": "dev-gcp-copy-<id>.env.clearscape.teradata.com",
    "owner": "christopher@mortimer.xyz",
    "type": "clearscape",
    "createDate": "2023-10-03T09:44:10.710Z"
  }
]

Connect with Teradata Studio

To test we can connect to the Teradata Database we will use Teradata Studio installed on our Windows client machine.

Create source table

Let’s create a very simple demo hello word table for copying to BigQuery.

create table demo_user.testing (
    col1 integer
  , col2 varchar(100)
)
;
insert into demo_user.testing values (1, 'hello');
insert into demo_user.testing values (2, 'world');

BigQuery Data Flow

Target table

For this example I have created a target table with the same schema as the source table in BigQuery using the GUI. For simplicity, I have also created a BigQuery Dataset with the same name as the source Teradata database.

Cloud Storage for driver

I have created a Google Cloud Storage bucket and downloaded the latest JDBC driver jar file and copied it up to cloud storage.

I have also added a temp folder for later use for the DataFlow job.

DataFlow job

Create a new Dataflow job with the following configuration options.

We will be using the Dataflow template of JDBC to BigQuery.

JDBCConnectionURL=jdbc:teradata://dev-gcp-copy-<id>.env.clearscape.teradata.com/demo_user
JDBCDriverClassName=com.teradata.jdbc.TeraDriver

For the optional parameters we will need to add the username and password for the Teradata database.

This blog post does not consider how you might store the password as a secret or in a key vault of some sort. Or at a minimal, base64 encode it so it is not visible in plain text.

Note: The two erroneous jobs were before the username and password were added to the optional parameters. The error message can be seen below.

java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 17.20.00.15] [Error 1032] [SQLState HY000] Single Sign-On NOT supported for Mechanism TD2.

Testing the results

After running the DataFlow job twice, and querying the target table we can see that the data has successfully been moved from the Teradata Database to the BigQuery Dataset.

Connecting Nifi to Teradata

Wed, 13 Sep 2023 00:00:00 GMT

Connecting Nifi to Teradata

For this demo I will be using a Windows PC including:

Docker desktop
PowerShell Core 7.3.6
VMWare Workstation Player 16
Teradata Express VM

Installing Nifi

The easiest way I found to get an instance of Nifi running is using Docker.

To get Docker working on Windows there is a little bit of extra setup required. Firstly install Docker desktop using the following installation.

https://docs.docker.com/desktop/install/windows-install/

Also, getting WSL2 setup is required for better performance.

https://hub.docker.com/r/apache/nifi

In your PowerShell terminal, pull the image.

docker pull apache/nifi

To run the Docker image (as a container), use the follow command, or run using the Docker desktop application.

docker run --name nifi -p 8443:8443 -d apache/nifi:latest

Running the container will generate a username and password.

docker logs nifi | Select-String -Pattern "Generated"

Generated Username [<username>]
Generated Password [<password>]

Browse to the service in your browser of choice and login with the Generated credentials.

https://localhost:8443/nifi/

Setting up Teradata VM

This is something for another time, and if you are reading this you probably have a large enterprise Teradata instance you want to connect to.

For this demo, the important thing is the IP address of the VM is 192.168.190.128, you will probably have a hostname for your Teradata instance.

Adding the Teradata JDBC driver to the Docker container

We need to copy the Teradata JDBC driver to the Docker container and then restart it. Firstly download the JDBC driver and change to the directory in PowerShell.

You’ll need to sign up/login to download the driver and then extract it from the Zip archive.

https://downloads.teradata.com/download/connectivity/jdbc-driver

Then list the containers running.

docker ps

Get the CONTAINER ID from the output.

CONTAINER ID IMAGE              PORTS                  NAMES
81b55e074d9a apache/nifi:latest 0.0.0.0:8443->8443/tcp nifi

Replace the CONTAINER ID in the following command to copy the driver from your local Windows host to the Docker container.

docker cp .\terajdbc4.jar 81b55e074d9a:/opt/nifi/nifi-current/lib

Setting up the FlowFile

Drag a new Processor onto the workbench.

https://localhost:8443/nifi/

We will select the ExecuteSQL type.

Now, lets configure this new Processor.

Browse to the Properties tab, and add the SQL to select from a table in your Teradata instance. In this case my demo selects from a previous loaded table from another blog post.

select * from PRD_ADS_PYTHON_NFL_DB.GAME

Then click the Arrow to configure the Database Connection Pooling Service.

Now, we have a list of Database connections. We need to add or configure one for our Teradata instance (in this case the VM).

Name=TeradataVM-Connection-Test
Id=87974be0-018a-1000-26db-a45f03ce9a88
Type=DBCPConnectionPool 1.23.2
Bundle=org.apache.nifi - nifi-dbcp-service-nar
SupportsControllerService=DBCPService 1.23.2 from org.apache.nifi - nifi-standard-services-api-nar
BulletinLevel=WARN

DatabaseConnectionURL=jdbc:teradata://192.168.190.128/dbc
DatabaseDriverClassName=com.teradata.jdbc.TeraDriver

Then click enable, and if everything is good, you should get an enabled connection.

Now you can run the Flow. I have added a PutFile Processor as well and connected them.

Right clicking on the List queue we can see the queue of resulting SQL queries.

We can see the successful query of the Teradata data in Nifi.

Hip-hop collaboration network graph

Mon, 08 May 2023 00:00:00 GMT

Introduction

As a Hip-hop fan (and other genres of music) it is obvious that certain artists are more alike than others. I was interested to see if a network of Hip-hop artists based on their collaborations with others would be able to highlight some interesting insights.

The things I needed to solve this problem were:

Data on artists, their tracks (songs) and the artists featuring on the tracks
Network graph analysis and visualisation software

This was done on a Windows 10 machine with:

Python 3
Gephi

Blog Collaboration with Herd Mentality

Speaking of collaborations, this post was done as a collaboration with a passionate group of technical data experts who are putting out their own collection of posts for the data community. See Herd Mentality

Getting the data

To have a look at the source code for getting the collaboration data, have a look at the following Github repo https://github.com/mortie23/musicbrainz-get. I looked for existing dataset but counldn’t find one, but found a fantastic free API from MusicBrainz https://musicbrainz.org/doc/MusicBrainz_API/Examples.

To get started, since it is 2023, I started with a prompt to ChatGPT that was something like this:

Write me a Python script that calls that Music Brainz API (documentaiton here https://musicbrainz.org/doc/MusicBrainz_API/Examples) that searches for an artist and returns the artists ID.

Of course it oblidged politely:

Then, with a bit of refactoring and a few more prompts to ChatGPT, I ended up with a script that could loop through a list of my favourite artists and produce a CSV file of the collaboration graph. The analysis was a bit of a throw back to the 1990’s of Hip-hop and I wanted a few East coast and West coast rappers in the list to see if the network analysis clusters them well.

# %%
artist_list = [
    "Jay-Z",
    "Nas",
    "Wu-Tang Clan",
    ...
]

Analysing and Visualizing the data

After quickly trying some network and charting packages in Python, and then trying a PowerBI markplace viz with minimal success, I remembered a desktop appliction I used years ago that could render beautiful network graphs. So I started working with Gephi https://gephi.org/users/download/.

Analysis

Here is the first look of the network graph with labels. The size of the nodes (and labels) is the number of edges in the graph (the number of collaboration artists). The color is:

Orange: Calfornia
Green: New york
Blue: Chicago

Most collaborations

The winner of this title goes to Snoop Dogg.

Clusters

The east coast and west coast artists are clustered well together based on their collaborations (as expected).

Key collaborations between clusters

It seems from the network graph that Dr Dre (from the west coast) signing 50 Cent (from the east coast) to his label became a key collaboration connection between the clusters.

Wrap up

I think there is great insights that can be gained from collaborations between artists that confirm some suspicions of fans and highlight some unknown insights.

Flask application with Postgres containerized across multiple clouds

Sun, 02 Apr 2023 12:00:00 GMT

Flask app with a Postgres database containerized across multiple clouds

Background

We are building out an application that has a decoupled RDBMS from a Front end in a multi cloud environment. We also want the front end to be containerized so that all the code and all its dependencies are packaged up and the application runs quickly and reliably regardless of the computing environment (development on client machine and on remote cloud hosting in production).

The choice architecture for this demonstration will be:

A self managed Postgres database on one cloud platform (GCP)
A self managed web server on a different cloud platform (AWS)
A local Windows client development environment

The Postgres Database (IaaS)

In this example I will be using a self managed Postgres database on a compute engine (Infrastructure as a Service). The main reason for this was that I already had this setup. It is a cheap way of testing out architectures. However, using a PaaS Postgres offering through something like these would be beneficial for a production solution as won’t need the overhead of managing upgrades and patching of the server OS:

AWS RDS https://aws.amazon.com/rds/
GCP Cloud SQL https://cloud.google.com/sql
Azure Database https://azure.microsoft.com/en-us/products/postgresql/

Firewall

To get started with local development, we need to add a firewall rule such that our client machine can connect to the remote Postgres service running on the GCP cloud compute service.

In the GCP console, we create a new firewall rule.

In this example we name it postgres and the protocol/port is tcp:5432 (the default port for Postgres). The Target (tag) will be the tag we have used for the Compute engine.

Then, after finding out what our public IP address is (you can just ask Google what is my ip address) we allow our IP address through the firewall.

You will notice there is a second IP address allowed through the firewall. This is the web server machine’s IP address which will be discussed later in the blog post.

Local development environment

I am using a Windows machine with:

WSL 2 (Ubuntu 20.04)
Python 3.8.10

To get started there is an OS dependency we need to install:

sudo apt install libpq-dev

Now we will create a Python virtual environment for the development.

cd ~/venv
python3 -m venv fpg
source fpg/bin/activate

Clone the git repository

The Github repository to follow is https://github.com/mortie23/flask-postgres.

Create a `.env` in the root of the repository

APP_SETTINGS=config.Config
DATABASE_URL="postgresql://<username>:<password>@<hostname>:<port>/<database>"

Initiate the database

If you have configured everything else correctly, and have the correct connection string in your .env file, you should be able to initialise the database.

flask db init

  Creating directory /flask-postgres/migrations ...  done
  Creating directory /flask-postgres/migrations/versions ...  done
  Generating /flask-postgres/migrations/script.py.mako ...  done
  Generating /flask-postgres/migrations/env.py ...  done
  Generating /flask-postgres/migrations/alembic.ini ...  done
  Generating /flask-postgres/migrations/README ...  done
  Please edit configuration/connection/logging settings in '/flask-postgres/migrations/alembic.ini' before proceeding.

flask db migrate

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'todo'
  Generating /flask-postgres/migrations/versions/bc12dad5cf64_.py ...  done

flask db upgrade

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 812916de9ad7 -> bc12dad5cf64, empty message

Run the Flask application in development

python3 app.py

* Serving Flask app 'app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
127.0.0.1 - - [02/Apr/2023 09:28:57] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [02/Apr/2023 09:29:10] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [02/Apr/2023 09:29:11] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [02/Apr/2023 09:30:02] "POST /add HTTP/1.1" 302 -
127.0.0.1 - - [02/Apr/2023 09:30:03] "GET / HTTP/1.1" 200 -

Test the appliction works

With a sample item in the todo, we can test that the appliction is writing to the cloud IaaS Postgres instance.

Deploy the application

Create a docker image of the Flask application

docker build -t mortimerxyz/flaskpostgrestodo:0.1 .

[+] Building 2.3s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                   0.0s
 => => transferring dockerfile: 326B                                                   0.0s
 => [internal] load .dockerignore                                                      0.0s
 => => transferring context: 2B                                                        0.0s
 => [internal] load metadata for docker.io/library/python:3.8-buster                   2.2s
 => [auth] library/python:pull token for registry-1.docker.io                          0.0s
 => [1/5] FROM docker.io/library/python:3.8-buster@sha256:0e41815385213a2be1c61f182f7  0.0s
 => [internal] load build context                                                      0.0s
 => => transferring context: 220B                                                      0.0s
 => CACHED [2/5] COPY app.py config.py database.py models.py requirements.txt /        0.0s
 => CACHED [3/5] COPY templates templates/                                             0.0s
 => CACHED [4/5] RUN pip install -r requirements.txt                                   0.0s
 => exporting to image                                                                 0.0s
 => => exporting layers                                                                0.0s
 => => writing image sha256:bd1fffc6a1154c85937d98c320e186afeb8e6a0e39bac848fb1151bdd  0.0s
 => => naming to docker.io/mortimerxyz/flaskpostgrestodo:0.1                           0.0s

To see the built image:

docker images -a

REPOSITORY                      TAG       IMAGE ID       CREATED             SIZE
mortimerxyz/flaskpostgrestodo   0.1       bd1fffc6a115   About an hour ago   986MB

This run statement passes the host machines environment variables for APP_SETTINGS and DATABASE_URL to the running Docker container. This means that to get it running, you will not be able to depend on the .env file you have for development. Since I am using a Windows host, the following PowerShell commands need to run to set the environment variables:

$Env:APP_SETTINGS='config.Config'
$Env:DATABASE_URL="postgresql://<username>:<password>@<hostname>:<port>/<database>"

docker run --env APP_SETTINGS --env DATABASE_URL -p 5000:5000 mortimerxyz/flaskpostgrestodo:0.1

[2023-04-02 01:40:51 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-04-02 01:40:51 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2023-04-02 01:40:51 +0000] [1] [INFO] Using worker: sync
[2023-04-02 01:40:51 +0000] [8] [INFO] Booting worker with pid: 8

Stop the running container

docker ps

CONTAINER ID   IMAGE                               COMMAND                  CREATED
STATUS         PORTS                    NAMES
fd611c4ff1ff   mortimerxyz/flaskpostgrestodo:0.1   "gunicorn -b 0.0.0.0…"   6 minutes ago   Up 6 minutes   0.0.0.0:5000->5000/tcp   nice_mclaren

docker stop fd611c4ff1ff

Publish the image to a container registry

We now need to push the image to a container registry so we can pull it from our deployment service.

docker push mortimerxyz/flaskpostgrestodo:0.1

Use the Docker image to deploy application

We are going to use a basic EC2 machine to host the web application. To enable us to SSH to the machine we will need to add a firewall rule to allow SSH over port 22 to our public IP address:

We will also need to add the public IP address of the EC2 machine to the firewall for the GCP machine that is hosting the Postgres database service:

First we will start with pulling the Docker image down from where we published it to Docker hub.

docker pull mortimerxyz/flaskpostgrestodo:0.1

0.1: Pulling from mortimerxyz/flaskpostgrestodo
4e2befb7f5d1: Pull complete
792af667f626: Pull complete
3e37868ebf66: Pull complete
591fe17e35dd: Pull complete
b9cba6e3073a: Pull complete
8d8bcef47b1e: Pull complete
920891d7496a: Pull complete
a2b9eb6d3f8d: Pull complete
eb27f9541dc2: Pull complete
579045be23fb: Pull complete
77f0ef020355: Pull complete
96bf4b83abf5: Pull complete
Digest: sha256:b020a3438a47ee535bb78afec1f9ab590cf73b66f74b2b94c469d84dfca93b14
Status: Downloaded newer image for mortimerxyz/flaskpostgrestodo:0.1
docker.io/mortimerxyz/flaskpostgrestodo:0.1

Since we are now using a Linux host, the following Bash commands need to run to set the environment variables:

export APP_SETTINGS=config.Config
export DATABASE_URL="postgresql://<username>:<password>@<hostname>:<port>/<database>"

We need to run the Docker container in detached mode:

docker run --detach --env APP_SETTINGS --env DATABASE_URL -p 5000:5000 mortimerxyz/flaskpostgrestodo:0.1

This EC2 machine already has Nginx installed and an associated SSL certificate and a domain name (this is outside the scope of this blog post).

We need to configure a server rule to proxy pass the service running on port 5000 to the SSL port 443 in the default Nginx configuration file (/etc/nginx/sites-available/default).

location / {
  proxy_pass http://localhost:5000;
  proxy_redirect http://localhost:5000/ $scheme://$http_host/;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_read_timeout 20d;
}

Now when we browse to the domain name we get the running Flask application.

Connecting to Teradata from Python

Sat, 25 Mar 2023 12:00:00 GMT

Reading and Writing data between Python and Teradata

This blog post is paired with the associated Github repository https://github.com/mortie23/python-teradata

Local Python environment setup

For this run through I am using:

Python 3.8.10
installed on Windows Subsystem for Linux (Ubuntu 20.04)

Create a new virtual environment and install the requirements.

# Create a new virtual environment
python3 -m venv tdsql
# Activate the environment
source tdsql/bin/activate
# Install the packages required
pip install -r requirements.txt
# Register the IPython Kernel for use (with VScode)
python3 -m ipykernel install --user --name tdsql --display-name tdsql

Teradata

I am using a Teradata Express 16.20 virtual machine running with VMWare Workstation Player 16.2.5. Our hosts file is mapping the virtual machine’s IP address to tdvm

cat /etc/hosts
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1       localhost
192.168.190.128 tdvm

Setup database structures

This step is obviously not required, however it is here for completeness sake. Running the teradata-setup.sql script to create the required database and provide access to the appropriate user.

-- Create Database
-- Assumes you have a production analytical datastore (ADS)
CREATE DATABASE PRD_ADS_PYTHON FROM PRD_ADS AS
   PERM = 0
   NO FALLBACK
   NO BEFORE JOURNAL
   NO AFTER JOURNAL
;
COMMENT ON PRD_ADS_PYTHON AS 'Python development' ;
GRANT ALL ON PRD_ADS_PYTHON TO PRDDBA WITH GRANT OPTION ;

CREATE DATABASE PRD_ADS_PYTHON_NFL_DB FROM PRD_ADS_PYTHON AS
   PERM = 0
   NO FALLBACK
   NO BEFORE JOURNAL
   NO AFTER JOURNAL
;
COMMENT ON PRD_ADS_PYTHON_NFL_DB AS 'NFL Data loaded using Python' ;
GRANT ALL ON PRD_ADS_PYTHON_NFL_DB TO PRDDBA WITH GRANT OPTION ;

-- Give Space
-- Python
CREATE DATABASE GS_TEMP FROM PRD_ADS AS
PERM = 1E9
;
GIVE GS_TEMP TO PRD_ADS_PYTHON
;
DROP DATABASE GS_TEMP
;
-- Python NFL
CREATE DATABASE GS_TEMP FROM PRD_ADS_PYTHON AS
PERM = 5E6
;
GIVE GS_TEMP TO PRD_ADS_PYTHON_NFL_DB
;
DROP DATABASE GS_TEMP
;

-- Users
CREATE USER MORTCH FROM PRD_USERS AS
  PASSWORD = MORTCH
  PERM = 0
  NO Fallback
  NO Before Journal NO After Journal
  DEFAULT ROLE = ALL
;

-- Roles
CREATE ROLE R_PYTHON_PRD;
GRANT R_PYTHON_PRD TO MORTCH;

-- Python
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, DROP TABLE ON PRD_ADS_PYTHON_NFL_DB TO R_PYTHON_PRD ;
GRANT SELECT, CREATE VIEW, DROP VIEW ON PRD_ADS_PYTHON_NFL_DB TO R_PYTHON_PRD ;

-- Test
CREATE TABLE PRD_ADS_PYTHON_NFL_DB.TEST (
  COL1 INTEGER
)
PRIMARY INDEX(COL1)
;
INSERT INTO PRD_ADS_PYTHON_NFL_DB.TEST VALUES(1);

Setup the table structures

Firstly test connecting to it from Teradata Studio IDE:

Running this (python-nfl-ddl.sql) SQL script in you favourite SQL IDE (for example Teradata Studio).

Python execution in VSCode

I’m going to using Python interactive in VSCode to run this. In this mode we can run the “Cells” (blocks of code between # %%) separately and view tabular results in a more visual way.

For example, running the python-teradata.sql script up until this point and then in the interactive window typing tables and hitting shift-enter.

# %%
query = f"""
select
  TableName
  , count(*) as ColumnCount
from
  dbc.columnsv
where
  databasename='{database}'
group by
  TableName
order by
  TableName
"""
tables = pd.read_sql(query, con)
# %%
tables

Loading data from CSV

The key function that loads data to a table from a CSV file is here:

def insert_rows_from_csv(
    tablename: str,
    columncount: int,
):
    """Insert rows from a CSV into a Teradat table

    Args:
        tablename (str): the name of the table
        columncount (int): the number of columns in the table
    """
    columncount_questionmark = ", ?" * (columncount - 1)
    exec_string = f"{{fn teradata_read_csv(./data/{tablename.lower()}.csv)}}insert into {tablename} (?{columncount_questionmark})"
    cur.execute(exec_string)

The script then has a function that loops over all the tables and runs the insert_rows_from_csv function. After the loads have completed, running a select using the Pandas read_sql method demonstrates that we have successfully loaded the tables from the CSV files.

Cleaning up

We also have a script to truncate all the rows from the tables.

# Truncate all tables
table_load = table_load.sort_values(by="order", ascending=False)
for index, row in table_load.iterrows():
    print(row["TableName"])
    cur.execute(f"delete from {row['TableName']}")

Python mono repo for Data Science with Poetry

Fri, 17 Feb 2023 12:00:00 GMT

Mono repo with Python Poetry

What we are looking to do is create a code repository that can be used as the source for many Python applications (Flask deployed with Docker) and exporitory data analysis that can share a common library of functions. Pretty much all the credit for this comes from the following article, where Dan Hipschman goes through the process that Opendoor went through to create a mono repo.

OpenDoor Labs, Our Python Monorepo

The Github repo that the remainder of the blog post is referencing is here Github

📁ml
├── 📁eda  # Each exploratory data analysis task has a directory here
│   ├── 📁eda-project-1
│   │   ├── eda-project-1.py  # Each eda project has Python scripts
│   │   ├── pyproject.toml  # Each eda project has its own dependencies
│   └── 📁eda-project-2 ...
├── 📁app  # Each application has a directory here
│   ├── 📁flask-app-1
│   │   ├── app.py  # Each application has an app entry point
│   │   ├── Dockerfile  # Each application has a Dockerfile
│   │   ├── pyproject.toml  # Each application has its own dependencies
│   └── 📁flask-app-2 ...
├── 📁lib  # Each lib is a Python package that you can install using poetry (or pip)
│   ├── 📁helloworld  #
│   │   ├── pyproject.toml  # Each lib specifies its dependencies
│   │   ├── 📁helloworld/
│   │   │   ├── __init__.py
│   │   │   ├── helloworld.py
│   │   └── 📁tests/
│   │   │   ├── test_helloworld.py
│   └── 📁lib-2 ...
└── 📁tools
    ├── 📁build  # Application build script

For this, I have tested on both ChromeOS Linux (Debian) and Windows WSL2 (Ubuntu 20.04). Docs from here are from the WSL2 run through.

Fresh Python Virtual Environment (venv)

# Make new directory in your home for virtual environments
mkdir ~/venv
# change into directory
cd ~/venv
# create a virtual environment named poetry (or any other name you choose)
python3 -m venv poetry
# activate the vitrual environment
source poetry/bin/activate
# confirm it activated
which python3

Clone this Git repo

# Make new directory in your home for git repos
mkdir ~/git
# change into directory
cd ~/git
# clone the repository
git clone https://github.com/mortie23/ml.git
# change into the clone repo
cd ml

Getting started with Poetry

Install this repositories requirements (which is just poetry). The remaining packages poetry will handle.

pip install -r requirements.txt

Using Poetry to install Python packages

Now using poetry to install the packages (including the internal one)

# change into directory
cd app/flask-app-1
# poetry to create/update lock file and install python package dependencies
poetry lock && poetry install

Updating dependencies
Resolving dependencies... (1.8s)

Writing lock file
Installing dependencies from lock file

Package operations: 12 installs, 2 updates, 0 removals

  • Installing markupsafe (2.1.2)
  • Installing click (8.1.3)
  • Installing exceptiongroup (1.1.0)
  • Updating importlib-metadata (4.13.0 -> 6.0.0)
  • Installing iniconfig (2.0.0)
  • Installing itsdangerous (2.1.2)
  • Installing jinja2 (3.1.2)
  • Installing pluggy (1.0.0)
  • Updating setuptools (44.0.0 -> 67.3.2)
  • Installing werkzeug (2.2.3)
  • Installing flask (2.2.3)
  • Installing gunicorn (20.1.0)
  • Installing pytest (7.2.1)
  • Installing helloworld (0.1.0 /mnt/c/git/github/mortie23/ml/lib/helloworld)

Run the app with poetry. This app sources from the internal package in dev mode, so changes made to the source files are reflected as they change.

# use poetry to run the app with the venv python3 binary
poetry run python3 app.py

 * Serving Flask app 'app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
127.0.0.1 - - [15/Feb/2023 19:33:46] "GET / HTTP/1.1" 200 -

Testing internal package for EDA

# change into directory
cd ../../eda/eda-project-1/
# poetry to create/update lock file and install python package dependencies
poetry lock && poetry install

Updating dependencies
Resolving dependencies... (8.2s)

Writing lock file
Installing dependencies from lock file

Package operations: 25 installs, 1 update, 0 removals

  • Installing asttokens (2.2.1)
  • Installing executing (1.2.0)
  • Installing parso (0.8.3)
  • Updating platformdirs (2.6.2 -> 3.0.0)
  • Installing pure-eval (0.2.2)
  • Installing traitlets (5.9.0)
  • Installing wcwidth (0.2.6)
  • Installing backcall (0.2.0)
  • Installing decorator (5.1.1)
  • Installing jedi (0.18.2)
  • Installing jupyter-core (5.2.0)
  • Installing matplotlib-inline (0.1.6)
  • Installing pickleshare (0.7.5)
  • Installing prompt-toolkit (3.0.36)
  • Installing pygments (2.14.0)
  • Installing python-dateutil (2.8.2)
  • Installing pyzmq (25.0.0)
  • Installing stack-data (0.6.2)
  • Installing tornado (6.2)
  • Installing comm (0.1.2)
  • Installing debugpy (1.6.6)
  • Installing ipython (8.10.0)
  • Installing jupyter-client (8.0.3)
  • Installing nest-asyncio (1.5.6)
  • Installing psutil (5.9.4)
  • Installing ipykernel (6.21.2)

python3 -m ipykernel install --user --name=poetry

Now to test running the EDA code within VScode, you need to have installed the VSCode Python extension from Microsoft. This will activate the Run Cell links above each of the # %% comment blocks.

When you do, an interactive windows will open the right side. Since you have registered your ipykernel, you will need to select it.

Now, when you run the cells, you should see that the internal package can be successfully run from the EDA project code.

Internal package unit testing

To test each function in the internal package we have included pytest and an example unit test.

# change into the package directory
cd lib/helloworld
# run all tests with pytest
pytest

=================================== test session starts ====================================
platform linux -- Python 3.8.10, pytest-7.2.1, pluggy-1.0.0
rootdir: /mnt/c/git/github/mortie23/ml/lib/helloworld
collected 1 item

tests/test_helloworld.py .                                                           [100%]

==================================== 1 passed in 0.09s =====================================

Internal package dependency

Since we are using poetry to manage the internal package as a dependency in the mono repo, it works in development mode on own local client machine, however when deploying the application using a Docker container, it will not have access to the internal package.

We can either:

deploy the package to an internal package server and then update the pyproject.toml file to source from the internal package server
copy the internal package from the lib directory it is currently in within the mono repo temporarily for the building of the docker image
There are probably other options, but the simplest option for me for this small example was the second option. the build step does this.

Build the docker container

If also testing this on Windows WSL2, you will need to install Docker desktop for windows and then configure it to use WSL2 as the kernel.

# change into build tools directory
cd tools/build/
# ensure the shell script has execute permissions
chmod +x build.sh
# run the build script with the parameter as the name of the application directory
./build.sh flask-app-1

flask-app-1
sending incremental file list

sent 630 bytes  received 17 bytes  1,294.00 bytes/sec
total size is 2,597  speedup is 4.01
[+] Building 10.3s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                    0.1s
 => => transferring dockerfile: 436B                                                    0.0s
 => [internal] load .dockerignore                                                       0.0s
 => => transferring context: 2B                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.8-buster                    2.9s
 => [auth] library/python:pull token for registry-1.docker.io                           0.0s
 => [1/7] FROM docker.io/library/python:3.8-buster@sha256:<>                            0.0s
 => [internal] load build context                                                       0.1s
 => => transferring context: 24.27kB                                                    0.1s
 => CACHED [2/7] RUN pip install "poetry==1.3.2"                                        0.0s
 => [3/7] COPY poetry.lock pyproject.toml app.py /                                      0.1s
 => [4/7] COPY lib lib/                                                                 0.1s
 => [5/7] COPY templates templates/                                                     0.0s
 => [6/7] RUN poetry config virtualenvs.create false   && poetry install --no-dev       6.6s
 => exporting to image                                                                  0.2s
 => => exporting layers                                                                 0.2s
 => => writing image sha256:<>                                                          0.0s
 => => naming to docker.io/mortimerxyz/flaskapp1:0.0.1                                  0.0s

Run the docker container

docker run -p 5000:5000 mortimerxyz/flaskapp1:0.0.1

[2023-02-15 08:59:29 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-02-15 08:59:29 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2023-02-15 08:59:29 +0000] [1] [INFO] Using worker: sync
[2023-02-15 08:59:29 +0000] [8] [INFO] Booting worker with pid: 8

Install Posit Connect on Ubuntu

Wed, 04 Jan 2023 12:00:00 GMT

Overview

We want to test out the installation and management of Posit Connect (previously RStudio Connect).

Prerequisites

A machine running Ubuntu
- I used a Windows machine with VMWare workstation 16 player and a clean install of Ubuntu 20.04 from ISO
RStudio server installation (basically because this installation process includes the installation of r-base and I was hoping to test a single server with dual installation of RStudio server and Posit Connect)
- I followed the steps on this documentation page Download RStudio server

Update: 2025-April Specific versions of r-base and RStudio

sudo apt-get install r-base
sudo apt-get install gdebi-core
wget https://download2.rstudio.org/server/jammy/amd64/rstudio-server-2024.12.1-563-amd64.deb
sudo gdebi rstudio-server-2024.12.1-563-amd64.deb

Infrastructure attempted

Windows with a WSL
- I got issues when Posit Connect used systemctl with System has not been booted with systemd as init system (PID 1). Can't operate. I’m thinking that my release of WSL didn’t support Systemd so I moved on to other options
Virtual machine running Ubuntu
- I got this to work and the remaining blog post starts with this as a basis
ChromeOS with Debian
- I actually tried this first and got issues with the service failing to start
- After trying the install on the virtual machine running Ubuntu and getting the same issue (which I resolved with a configuration change), I tried the configuration fix on ChromeOS debian and it worked

Installation

Following on this Posit documentation page Manually install Posit Connect. In my case, the Ubuntu installation I had was 20.04. (This same installation worked for Debian on ChromeOS)

sudo apt-get install gdebi-core
curl -O https://cdn.rstudio.com/connect/2022.12/rstudio-connect_2022.12.0~ubuntu20_amd64.deb
sudo gdebi rstudio-connect_2022.12.0~ubuntu20_amd64.deb

Update 2025-April I needed to build another VM recently with newer versions and to test Python applications

curl -O https://cdn.posit.co/connect/2024.09/rstudio-connect_2024.09.0~ubuntu24_amd64.deb
sudo apt install ./rstudio-connect_2024.09.0~ubuntu24_amd64.deb

Initially this failed to start the service

After running the installation, test that the service is running.

sudo systemctl status rstudio-connect

For me, unfortunately, the service failed to start.

● rstudio-connect.service - Posit Connect
     Loaded: loaded (/lib/systemd/system/rstudio-connect.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2023-01-04 01:15:02 PST; 116ms ago
    Process: 20215 ExecStart=/opt/rstudio-connect/bin/connect --config /etc/rstudio-connect/rstudio-connect.gcfg (code=exited, status=1/FAILURE)
   Main PID: 20215 (code=exited, status=1/FAILURE)

Jan 04 01:15:02 ubuntu systemd[1]: rstudio-connect.service: Scheduled restart job, restart counter is at 5.
Jan 04 01:15:02 ubuntu systemd[1]: Stopped Posit Connect.
Jan 04 01:15:02 ubuntu systemd[1]: rstudio-connect.service: Start request repeated too quickly.
Jan 04 01:15:02 ubuntu systemd[1]: rstudio-connect.service: Failed with result 'exit-code'.
Jan 04 01:15:02 ubuntu systemd[1]: Failed to start Posit Connect.

After checking the the log:

sudo cat /var/log/rstudio/rstudio-connect/rstudio-connect.log

I found the last log line showing that the reason the service cannot start is the Posit Connect cannot find the installation of R.

time="2023-01-04T09:15:02.118Z" level=fatal msg="Error: Unable to initialize local environment: Unable to use R on this system: Could not locate an R installation"

I’m not sure what default location for R that Posit Connect searches for. Let’s find out where R is installed on our system:

which R

Looks like it is installed in /usr/bin.

/usr/bin/R

Change the configuration for the R install location

sudo vim /etc/rstudio-connect/rstudio-connect.gcfg

Initially I tried a configuration of /usr/bin/ but found that Posit Connect was by default adding bin to the path and looking in /usr/bin/bin/. This was the configuration that worked for me:

[Server]
; No longer working in newer version
RVersion = "/usr/"

Update: 2025-April With the latest version of Posit Connect I used the automatic scanning config instead

[R]
ExecutableScanning = true
ExecutableSystemScanning = true

Update: 2025-April I wanted to test deploying Python based apps

; Installing Python the Posit way: https://docs.posit.co/resources/install-python.html
; Configuring Posit Connect for Python apps: https://docs.posit.co/connect/admin/python/
[Python]
Enabled = true
Executable = /opt/python/3.12.4/bin/python3

Service working

Restart the service and check the status:

sudo systemctl restart rstudio-connect
sudo systemctl status rstudio-connect

● rstudio-connect.service - Posit Connect
     Loaded: loaded (/lib/systemd/system/rstudio-connect.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-01-04 01:18:51 PST; 11s ago
   Main PID: 20580 (connect)
      Tasks: 22 (limit: 9408)
     Memory: 26.1M
     CGroup: /system.slice/rstudio-connect.service
             ├─20580 /opt/rstudio-connect/bin/connect --config /etc/rstudio-connect/rstudio-connect.gcfg
             ├─20622 /opt/rstudio-connect/ext/rserver-monitor --monitor-data-path /var/lib/rstudio-connect/metrics --monitor-interval-seconds 60 --monitor-domain-socket /tmp/connect-server/rserver-monitor.socke>
             ├─20628 /bin/sh -c '/usr/bin/rrdtool' '-'
             └─20632 /usr/bin/rrdtool -

Jan 04 01:18:51 ubuntu systemd[1]: Started Posit Connect.
Jan 04 01:18:52 ubuntu rstudio-connect[20580]: time="2023-01-04T09:18:52.476Z" level=info msg="Loaded server migration from '/etc/rstudio-connect/rstudio-connect-migration.gcfg'"
Jan 04 01:18:52 ubuntu rstudio-connect[20580]: time="2023-01-04T09:18:52.480Z" level=info msg="The migration file does not affect the configuration"
Jan 04 01:18:52 ubuntu rstudio-connect[20580]: time="2023-01-04T09:18:52.480Z" level=info msg="Found 0 environment variables starting with 'CONNECT_' that may affect configuration.\n"
Jan 04 01:18:52 ubuntu rstudio-connect[20580]: time="2023-01-04T09:18:52.482Z" level=info msg="Using file /var/log/rstudio/rstudio-connect/rstudio-connect.log to store Server Logs."

Success, we have a installation of Posit Connect up and running.

Note: it is a 45 day free trial license.

Publish an R Package to a private repository and use

Mon, 05 Dec 2022 12:00:00 GMT

Overview

Let’s say we want to manage an R code base as a package. The benefits might include:

a good way to distribute R code for other people
a good way to document R code
it keeps track R functions for reuse

What if we don’t want to share the package on the public internet (for corporate reasons), and have an R Studio server (installed on Linux) as our infrastructure.

R Studio Server

For demo purposes I have installed R Studio server on my Chromebook using the Ubuntu/Debian instructions Download RStudio Server. If you have a Windows machine this will also work Using RStudio Server in Windows WSL2

Now let’s check that the service is running.

service rstudio-server status

Since the service is running on our local machine we can access it on port 8787 like this: http://127.0.0.1:8787

Example

For this example we are going to create the most basic package we can think of. A package that contains a single function that prints out hello world. Let’s call the package helloworldr.

We want a documentation site as well.

Initial install

We will use the GUI in R Studio Server to start the template for our Hello World R package.

Select the parent directory (the New Project Wizard will create a directory with the name of the project).

In the directory it creates you will see files like this:

Now we need package down for the documentation.

usethis::use_pkgdown()

Of course Yes is the wrong answer to 1, 2 or 3.

If you have installed Node.js modules in the past, or have seen memes about installing Node.js modules, you haven’t seen nothing yet, this will take a long time.

R Environment

Since we are using renv to manage our package dependencies and versions, and we have installed a lot of package dependencies through using packagedown, we need to update the lock file.

renv::snapshot()

You should see the JSON in your renv.lock file now contains a list of all the package dependencies (if you checked it before hand you would have seen not much at all).

So this is how we got this far. But now, to follow the rest we are going to continue with something I have prepared earlier.

Hello World R

After following these steps, and making a few modifications to the function, we end up with this:

https://github.com/mortie23/helloworldr

If you were following, from here it might be easier to delete what you have and clone this one. Otherwise, the differences you will need to at least continue functionally are just the build-deploy.R script.

Deploy to private repository

Before we build and deploy the package to a private repository, we need to know what a “private repository” is and how to create one.

All it is a directory on a file system! So it can be created like this:

# Create complete path and parents as needed
mkdir -p /opt/privatePkgs/src/contrib/

Now just source the build-deploy.R script.

Check the package was built and deployed:

cd /opt/privatePkgs/src/contrib/
ll

Yep, there is the tar gzipped up pacakge and the PACKAGE repo metadata.

Permissions Size User     Date Modified Name
.rwxrwx---  1.2k mortimer 30 Nov 10:42  helloworldr_0.0.0.9000.tar.gz
.rw-r--r--   210 mortimer 30 Nov 10:42  PACKAGES
.rw-r--r--   157 mortimer 30 Nov 10:42  PACKAGES.gz
.rw-r--r--   384 mortimer 30 Nov 10:42  PACKAGES.rds

Build the docs page

pkgdown::build_site()

Preview the site built in the docs directory.

# Sorry, I don't know how to run a basic HTTP server in R, let's use Python
cd docs
python -m http.server

Hello World R function use

So we have a package deployed to a private repository, we can now use the functions in other downstream projects.

Start by cloning this Git repository down to your machine.

https://github.com/mortie23/helloworldr-fun

In the renv.lock file note the refeerence to the Repository:

{
  "Name": "private",
  "URL": "file:///opt/privatePkgs"
}

and the reference to our package:

{
  "Package": "helloworldr",
  "Version": "0.0.0.9000",
  "Source": "Repository"
}

If you have all the previous section complete correctly, you should be able to run:

renv::restore()

and our helloworldr package should be installed into the new R projects renv and the function should now be available.

Conclusion

Well, that took longer than I thought!

Databricks security model architecture

Mon, 31 Oct 2022 12:00:00 GMT

Overview

Let’s say we have a brand new Databricks resource that we have just created from Azure. It comes as an empty shell ready for use. You add yourself as the system administrator and you need to onboard your organisation in such as way that teams within the organisation can collaborate. To illustrate this, let’s take the example of a large group of Law Firms.

Organisations can have a structure where multiple “business units” are supported by a common set of enabling services. In this example the People (HR), Finance and IT teams are central and support the different groups of legal teams, Family, Criminal and Consumer law.

One of the organizations has put their hand up to be the enabling service of IT.

Databricks will be used by both the business units and the IT teams (to do bespoke analysis of IT systems, such as cost monitoring of cloud resources and monitoring usage of systems). The teams may have the need to keep their work separated and inaccessible from each other, and may also have the need to share.

Gartner coined the term Bi-model. See this article: Gartner Glossary, Bimodal.

Bimodal is the practice of managing two separate but coherent styles of work: one focused on predictability; the other on exploration. Mode 1 is optimized for areas that are more predictable and well-understood. It focuses on exploiting what is known, while renovating the legacy environment into a state that is fit for a digital world. Mode 2 is exploratory, experimenting to solve new problems and optimized for areas of uncertainty

There are many words used to separate these two modes, I have listed a few I have seen:

Mode 1
- Operational
Mode 2
- Discovery, Laboratory, Exploratory

Here we are focusing on mode 2 (some mode 1 thrown in here too) and will denote mode 2 as Discovery

Components

There are many components of a Databricks workspace that need to be considered in the security model. We will try to cover off a lot of what is included and have a default position on each component. Having this upfront will attempt to avoid business users having to ask down the track for a component that hasn’t been considered.

Workspace Settings
Users and Groups
Workspace and Repos
Compute
Storage
Secrets
Machine Learning

Workspace Settings

There are many workspace settings that can be changed depending on what your requirements for the platform are. It is difficult to prescribe a default set of modifications to make, however one recommendation would be to Enable the Web Terminal. Users are able to submit shell commands from a notebook regardless of this setting, it may just make it easier if they are comfortable using a command line.

Users and Groups

Azure Active Directory

Our Azure administrators have some nice rules and naming conventions which will help us down the track. Let’s create the organization structure in Azure Active Directory.

LawFirms-<Environment>-Databricks-<BusinessUnit>-<Role>

Component	Values	description
Organisation	LawFirms	The name of the organisation
Environment	Dev, PreProd, Prod
Resource	Databricks	The name of the resource
Business unit	Family, Consumer, Criminal etc	The name of the business unit
Role	DataScientist, DataEngineer etc	The role within the business unit

Firstly we create a series of groups that assist us in setting permissions at the group level (not user level).

Now we add the users to the groups.

This is repeated for all the users and groups in the New York law firms organizational structure.

Databricks users and groups

We will sync these users and groups from Azure Active Directory to our Databricks workspace using the following process Sync users and groups from Azure Active Directory. If that is too much overhead (which it may well be) then syncing the users manually and creating Databricks local groups manually may suffice.

Note: if we are going to do this manually, we should put careful thought on how we might migrate to the SCIM method later down the track. Following a naming convention that will make the migration easy would be beneficial. The manual method will also become cumbersome as more Law firms in New York start to see the benefits of the platform and want to jump on, or as Lawyers move between firms, which we know they do, and we need to change their membership in multiple places (Azure AD and Databricks).

By default, all users are in the users group and cannot be removed. By default the users group may have settings that you want to control separately, so lets remove all default entitlements from the users group.

Now, lets add at least Workspace access back for each of our custom groups from Azure Active Directory.

So now we have a Databricks workspace with the users and groups from our organisation, we need to setups some things to enable them to work both siloed and collaboratively.

Workspace and Repos

Workspace

Using source control is going to be the recommended way to manage notebooks (code). But in some instances, your business units data scientists may not be comfortable using Git, or maybe even the IT team hasn’t provided them with access to a remote Git service (such as Github, Gitlab, Bitbucket or Azure DevOps repos).

We create a top level folder for each business unit at both the Workspace level and the Shared level. The Shared folders are optional, and should be communicated to business that they are use at your own risk, as other teams can read/write content in them.

So now, logging in as Robert Zane from the Family Law business unit, we will only see the Family workspace folder.

Repos

What is a distributed version control system?

A distributed version control system (DVCS) brings a local copy of the complete repository to every team member’s computer, so they can commit, branch, and merge locally.

When it comes to Repos, we do not need a folder for each business unit for feature development and collaboration. Git is a DVCS, and that means that each users has a folder for themselves.

However, for running scheduled jobs or CI/CD, a location that always has the production code is required. This article discusses this topic well CI/CD workflows with Git integration and Databricks Repos. To enable this we will still create a top level folder for each business unit to enable them to operationalise their pipelines.

We have set the same Can Manage permissions to the top level Repo folder (Family), and Robert was able to log in a clone his teams repo twice:

once checked out to to the preprod branch
the next checked out to the master branch

This way, his team can create jobs that source code from either of these clones and ensure the job runs against the correct version of the code base.

He also has a personal folder that he can clone his teams repository and check out his feature branches that he creates for analysis and development.

Compute

We may/may not want to allow business units to create any cluster they want. This example assumes we do not.

Note: Going down this path will require more administration burden down the track as business units requirements change. Some restrictions appear to have more advantages than not, such as limiting the machine spec (and therefore limiting cost), however limiting, for example the Databricks runtime version, does not appear to have many advantages.

This can be done using cluster policies.

Cluster policies

A cluster policy definition is a JSON string that defines restrictions or sets parameters on cluster creation.

Cluster policies themselves require a name. The name will contain the core components required to delineate between use of the policies. They will line up with the components used to name the Active Directory Groups.

<BusinessUnit>-<Role>

Component	Values	description
Business unit	Family, Consumer, Criminal etc	The name of the business unit
Role	DataScientist, DataEngineer etc	The role within the business unit

We will create a very light touch cluster policy for our IT team members and a more restrictive policy for the members of the Family Law business unit (see Restrictive cluster policy).

The light touch cluster policy is below:

{
  "cluster_name": {
    "type": "regex",
    "pattern": "(Dev|PreProd|Prod)-(Discovery|Operational)-IT-[0-9.]*?ML?-[a-zA-Z]*-?[0-9]*?",
    "hidden": false
  },
  "custom_tags.Team": {
    "type": "fixed",
    "value": "IT"
  },
  "autotermination_minutes": {
    "type": "unlimited",
    "isOptional": true,
    "defaultValue": 60
  }
}

Our naming convention for the clusters is:

(Dev|PreProd|Prod)-(Discovery|Operational)-<BusinessUnit>-[0-9.]*?ML?-[a-zA-Z]*-?[0-9]*?

Component	Values	description
Environment	Dev, PreProd, Prod	The environment of the clusters use
Mode	Discovery, Operational	What mode the cluster is used for
Business unit	Family, Consumer, Criminal etc	The name of the business unit
Databricks runtime		The Databricks runtime version
Name	Such as a project	A name to denote the use of the cluster
Cluster number	If more than one per project	A number (optional)

Example:

Prod-Discovery-Family-11.3ML-project-01

After creation of the policy, we need to assign the policy permission to the correct group.

Testing the policy

When Robert creates a cluster he only had the option to create a cluster using the Family-DataScientist cluster policy.

When he attempted to create a cluster that breached the naming convention, he is notified at the bottom of the page what the naming convention he needs to follow is (as a regex).

Cluster use

By default, the permissions for using the cluster will be exclusive to the user that created the cluster. This is not a great idea, since the cluster might need to be used (in the case of a shared cluster), monitored, stopped or deleted by someone else within the team. We will communicate with the business units that they can self manage the permissions of their clusters and should add their team as Can Manage after the creation of any new cluster.

Package management

There are a few ways to install required packages on clusters. Databricks provides a GUI for installing packages, but this can also be done:

within notebook code using, for example, %pip install <package>
within an init script run while the cluster is starting up (see Sample init_script framework

Storage

Data lake

All storage will be kept in an Azure Data Lake Storage (ADLS) Gen 2 storage account. A single container will we created for Discovery use. A sub folder will be created for each business unit and permissions set accordingly.

To enable users from Azure Active Directory to read and write from the ADLS we need a couple of things first. We will use the group to manage the permissions:

The group will need the role of Reader at the storage account level
The ADLS folder’s ACLs will need to have the group set appropriately.

Make sure you set the default permissions too so that child items (files) in the folders inherit the permissions of the parent folder.

This is a very siloed security model and does not allow for sharing of data between business units. It is suggested to also create a shared folder within the discovery container and give each business unit access on the occasion that they need to share data.

Hive metastore

In Databricks, data can be registered to the Hive metastore (think of it as an information schema or data dictionary not the data itself). This enables SQL queries on the data (schema on read).

By default there is no security across the Hive metastore, which means that all users have permissions to create, read and drop registrations (not the data) from the metastore.

We will create a Hive metastore schema (the name schema and database are interchangeable in a Databricks metastore) for each business unit.

CREATE SCHEMA admin;
CREATE SCHEMA family;
CREATE SCHEMA consumer;
CREATE SCHEMA criminal;
CREATE SCHEMA it;

Note: Hive metastore objects (schemas, tables etc) can only be lowercase.

This will become a suggestion for business users to use, however there is nothing restricting them from creating more schemas, and dropping already existing schemas.

To make a more secure metastore layer without going down the path of Table Access Controls we could implement multiple metastores using Databricks Unity Catalog (not for this discussion).

Table Access Controls

Table access control lets you programmatically grant and revoke access to your data from Python and SQL.

By default, all users have access to all data stored in a cluster’s managed tables unless table access control is enabled for that cluster. Once table access control is enabled, users can set permissions for data objects on that cluster.

Table ACLs

This means that unless Table ACLs are enforced on all clusters and applied to all objects in the Hive metastore, then it is not an option. Since we have architected permissions at the storage level we will not go down this path.

Sample of how this may work

GRANT USAGE ON SCHEMA family TO `LawFirms-Prod-Databricks-FamilyLaw-DataScientist`;

Implementing a data governance model using Table ACLs requires clusters with Table ACLs enabled. If we attempt to run any GRANT statements with clusters that do not have Table ACLs enabled we will get the following error.

Error in SQL statement: SparkException: Trying to perform permission action on Hive Metastore /CATALOG/`hive_metastore`/DATABASE/`family` but Table Access Control is not enabled on this cluster.

After adding the following Spark config to the cluster, the Table ACL was successfully set.

spark.databricks.acl.sqlOnly true

Michael Ross in the IT team logs in tries to select from a table in the family schema and gets the following error.

Error in SQL statement: SecurityException: User does not have permission SELECT on table `family`.`game`.
User does not have permission USAGE on database `family`.

DBFS

The Databricks File System (DBFS) is a distributed file system mounted into a Databricks workspace and available on Databricks clusters.

Databricks recommends against storing any production data or sensitive information in DBFS.

In some cases it may be required, so the important thing to do is to set some standards and expectations to business units on the usage of DBFS.

directory level	directory	description
	/dbfs/FileStore	Top level DBFS folder for artifacts
	Admin	Admin directory
	init_scripts	For admin init scripts
	lib	For file based libraries or packages
	debs	Debian packages
	python	Python packages
	R	R packages
	Family	Business unit directory
	init_scripts	For business unit init scripts

We use the top level of the DBFS as the FileStore as a convention. At this level we create a folder for each business unit and provide them guidance on placing anything that is required there. There is nothing stopping them from creating more top level folders or deleting existing folders or files.

For this reason it is suggested that any files that are stored in DBFS are not the source of truth. A suggested way to manage scripts that are required in DBFS is to use a process such as the one documented here in a previous post Databricks and Azure DevOps CICD Pipeline for DBFS

Secrets

Users may/may not require secrets. However it is best to provide an Azure key vault backed secret scope upfront for each business unit and provide them a user guide on how to use it in the instance that they do.

We are going to manage the secrets using the Databricks CLI in a PowerShell terminal. Create an Azure Key Vault-backed secret scope using the Databricks CLI

# Create each secret scope
databricks secrets create-scope --scope <scope-name> --scope-backend-type AZURE_KEYVAULT --resource-id <azure-keyvault-resource-id> --dns-name <azure-keyvault-dns-name>
# List all scopes
databricks secrets list-scopes

Scope        Backend         KeyVault URL
-----------  --------------  --------------------------------------------
admin        AZURE_KEYVAULT  https://<keyvault>.vault.azure.net/
consumer     AZURE_KEYVAULT  https://<keyvault>.vault.azure.net/
criminal     AZURE_KEYVAULT  https://<keyvault>.vault.azure.net/
family       AZURE_KEYVAULT  https://<keyvault>.vault.azure.net/
it           AZURE_KEYVAULT  https://<keyvault>.vault.azure.net/

Machine Learning

Our users are going to want to register models to the inbuilt model registry in Databricks (an instance of MLFlow).

At this stage, the recommendation in Discovery is for users to self manage the permission of the model artifacts, however a user guide through an example is the best way to enable this.

To demonstrate how this could work we are going to log in as one of our IT Data Scientists (Michael Ross), and he is going to run through our Databricks NFL MLFlow example.

First he creates an appropriate cluster using the cluster policy we created for Data scientists in our IT team. He names it:

Prod-Discovery-IT-11.3ML-nfl-01

He knows the best practices because of the great users guides provided by the administrators so he goes into the cluster permissions and gives the rest of his team permissions to the shared cluster.

He clones the Git repo to his personal repo folder using the following HTTPS clone URL https://github.com/mortie23/databricks-nfl-data.git.

Firstly Michael loads the CSV files to the ADLS folder that the administrators have setup for him on abfss://discovery@lawfirm.dfs.core.windows.net/IT/ and then runs the db.sql notebook to load all the NFL data from ADLS and register them in the Hive metastore.

Then he opens up the nfl.model notebook and trains a model. Since we are using MLFlow, Michael ends up with a experiment in the model registry.

with mlflow.start_run():

Notebooks and experiments in a folder inherit all permissions settings of that folder. So only Michael will be able to see the experiment he ran. This is fine in Discovery, but when moving into Mode 1 (Operational) we will want the experiments run from the top level PreProd and Prod clones.

Michael now registers the experiment to a model for use by others in the team. He also needs to remember to set the permissions on the registered model.

Michael Ross now deicides to hand off the remainder of the work (integrating the model predictions into an operational pipeline) to the Data Engineer in his team.

Harvey Specter now logs in and since Michael has set the permission correctly he can pick up the same notebook. Since Michael used a Shared cluster, Harvey can use the cluster that Michael created.

Note: If Michael did not set the model permissions, Harvey would get the following error: RestException: PERMISSION_DENIED: User does not have any permission level assigned to the registered model.

Harvey is able to use the following code to load the latest trained model that Michael prepared for him.

model_name = "nfl-decisiontree-players"
model_udf = mlflow.pyfunc.spark_udf(
  spark,
  f"models:/{model_name}/latest"
)

Appendix

Restrictive cluster policy

{
  "spark_conf.spark.databricks.cluster.profile": {
    "type": "fixed",
    "value": "singleNode",
    "hidden": true
  },
  "num_workers": {
    "type": "fixed",
    "value": 0,
    "hidden": true
  },
  "runtime_engine": {
    "type": "fixed",
    "value": "STANDARD",
    "hidden": true
  },
  "cluster_type": {
    "type": "fixed",
    "value": "all-purpose",
    "hidden": false
  },
  "cluster_name": {
    "type": "regex",
    "pattern": "(Dev|PreProd|Prod)-(Discovery|Operational)-Family-[0-9.]*?ML?-[a-z]*-?[0-9]*?",
    "hidden": false
  },
  "data_security_mode": {
    "type": "fixed",
    "value": "LEGACY_SINGLE_USER"
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 60
  },
  "custom_tags.Team": {
    "type": "fixed",
    "value": "Family"
  },
  "spark_version": {
    "type": "allowlist",
    "values": ["11.3.x-cpu-ml-scala2.12", "11.3.x-scala2.12"],
    "defaultValue": "11.3.x-scala2.12"
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["Standard_F4"],
    "defaultValue": "Standard_F4"
  }
}

Sample init_script framework

This sample framework has a parent script that calls subsequent scripts to install packages for:

apt
python
R

clusterConfigs.sh

#!/bin/bash
## add custom prompt to bashrc
cat ${script_dir}/bashPrompt.sh >>~/.bashrc
## add custom vimrc using vim so much on the terminal
cat ${script_dir}/.vimrc >>~/.vimrc
source ${script_dir}/packageInstallTest.sh

packageInstallTest.sh

#!/bin/bash
## install python packages from a requirements file
/databricks/python/bin/pip install -r /${script_dir}/packageInstallTest.txt
## test installing jq using apt and libsodium-dev (which is a dependency for our next R one)
apt -y install jq
apt -y install libsodium-dev
## install library from R library list
cat /${script_dir}/packageInstallTest.R | R --no-save

packageInstallTest.txt

nltk
spacy
file:/dbfs/FileStore/Admin/lib/python-package/en_core_web_md-3.1.0-py3-none-any.whl
imblearn

packageInstallTest.R

## set the correct library path for Databricks spark
.libPaths('/databricks/spark/R/lib')
databricksLibPath<-.libPaths()[1]
## install two packages to this library path
install.packages('plumber', repos='https://cran.csiro.au/', lib=databricksLibPath)

Databricks and Azure DevOps CICD Pipeline for DBFS

Thu, 11 Aug 2022 12:00:00 GMT

DevOps CICD Pipeline for Databricks

This is in reference to this Git repo https://dev.azure.com/mortimer-xyz/mortie23/_git/dbfs-cicd-pieline

Much credit to Adam Paternostro

This is based on a simplified version of the repo from Adam Paternostro. Read up on this for a much more feature rich version of this.

Adam Paternostro, Azure-Databricks-Dev-Ops

I only required a small subset of what Adam was doing so I created my own how to in the process. The following is specifically about syncing Databricks init scripts from a Git repo to DBFS.

There is a whole lot of things you need to get this to work. Most of it is documented elsewhere, so instead of re-writing it, I’ll reference it.

Create a repo in Azure DevOps

Add the content of this repo.

Create a DevOps Azure Resource Manager service connection

This will create it’s own service principal which we will be using later.

Azure Resource Manager service connection

Create a Key Vault

Quick Create in Azure Portal

Add the secrets to the KeyVault

You’ll need to generate a secret for the service principal.

Create an Environment in Azure DevOps pipelines

Environment in Azure DevOps pipelines

Add an Access Policy for the KeyVault

Give the service principal associated to the Azure Resource Manager service connection (create previously) service principal Secret management on the Key Vault.

Access Policy for the KeyVault

Create a Databricks secret scope backed by KeyVault

Create and then run the pipeline

DevOps pipelines results

Databricks DBFS success

Now, checking in our Databricks DBFS we see the folder and the scripts we copied.

If we modify the scripts and then git add . git commit -m 'message' git push etc. We can configure the pipeline to run on a pull request completion to master.

Databricks using Azure ML model registry

Wed, 10 Aug 2022 12:00:00 GMT

Registering a model in AzureML from Databricks

What if we want to use AzureML as a central model repository for all models across many platforms (in this case Databricks workspaces). The AzureML model registry also uses MLFlow. I will follow these docs and show it step by step with examples.

Docs

Get things ready

There will be a few secrets we will need to maintain to get this going. The authentication we will use from Databricks to AzureML will be using an Azure Service Principal.

Create a Service Principal

Firstly lets create a Service Principal in our Azure Active Directory. Using the Azure PowerShell module:

az ad sp create-for-rbac --sdk-auth --name ml-auth --role Contributor --scopes /subscriptions/<subid>

You’ll see output like the following JSON:

{
  "clientId": "<guid>",
  "clientSecret": "<guid>",
  "subscriptionId": "<guid>",
  "tenantId": "<guid>",
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

Create the secret scope

Using this output we will put the bits we will need in a Databricks secret scope. If you don’t know how to do this using the Databricks cli you can follow this Databricks CLI. The main thing is to install it using pip and then configure it to authenticate to your workspace using databricks configure -t <databricks-pat>. Once you have done this you need to create a secret scope (I called mine azureml).

Store the secrets

databricks secrets put --scope azureml --key clientId
databricks secrets put --scope azureml --key clientSecret
databricks secrets put --scope azureml --key subscriptionId
databricks secrets put --scope azureml --key tenantId

Let’s also store some of the other info we need in the same secret scope while we are at it.

databricks secrets put --scope azureml --key subscriptionid
databricks secrets put --scope azureml --key resourcegroup
databricks secrets put --scope azureml --key workspacename

Configure the external model registry

Now to configure your Databricks cluster to use the external MLFlow model registry you need to add this extra cell in your notebook:

# Change the MLFlow tracking URI to the AzureML one
import os
from azure.ai.ml import MLClient
import azure.identity as azurei
import mlflow

# set environment variables for the service principal authentication of DefaultAzureCredential() getting details from the secret scope
os.environ["AZURE_CLIENT_ID"] = dbutils.secrets.get(scope = "azureml", key = "clientId")
os.environ["AZURE_TENANT_ID"] = dbutils.secrets.get(scope = "azureml", key = "tenantId")
os.environ["AZURE_CLIENT_SECRET"] = dbutils.secrets.get(scope = "azureml", key = "clientSecret")

# Enter details of your AzureML workspace getting details from the secret scope
subscription_id = dbutils.secrets.get(scope = "azureml", key = "subscriptionid")
resource_group = dbutils.secrets.get(scope = "azureml", key = "resourcegroup")
workspace_name = dbutils.secrets.get(scope = "azureml", key = "workspacename")

ml_client = MLClient(
    credential=azurei.DefaultAzureCredential(logging_enable=True),
    subscription_id=subscription_id,
    resource_group_name=resource_group
)

azureml_mlflow_uri = ml_client.workspaces.get(workspace_name).mlflow_tracking_uri
mlflow.set_tracking_uri(azureml_mlflow_uri)
print(azureml_mlflow_uri)

This should output the URI of your AzureML model registry.

azureml://australiaeast.api.azureml.ms/mlflow/v1.0/subscriptions/[REDACTED]/resourceGroups/[REDACTED]/providers/Microsoft.MachineLearningServices/workspaces/[REDACTED]

Train you model

Now, when we log any artifacts using MLFlow within Databricks, it will be logged against the MLFlow registry in AzureML.

mlflow.spark.log_model(spark_model=model, artifact_path="model")

Experiment list:

Specific experiment:

Model list:
Specific model:

Model scoring API as a Docker container

Thu, 04 Aug 2022 12:00:00 GMT

Use Docker to host a Python FastAPI service

In my preview blog post (Model scoring API) I went through the technical basics of a model scoring API. I finished it off at the step of having it running on you local client machine (laptop in my case). However to host such a service we need a way to deploy this somewhere. We have lots of options. The first option that comes to mind would be to:

spin up a cloud Linux virtual machine
go through the process of installing Python and the packages required
run the uvicorn service in the background

However, let’s go through the process of hosting it using Docker. The benefits are the portability of the resulting service.

Install Docker on your local computer

I’m using Windows now for this process (if you are you may need to install the Docker for Windows application). The default is for it to build Windows containers (which I wasn’t aware of at the time). This results in a Docker image that needs to be run on Windows. So if you are planning on hosting it on a Windows server, then go with this. Otherwise, like I did in the end, you will need to switch to Linux containers (assuming you have WSL2 on your machine).

Build the image

You’ll need to start by cloning the associated repo https://github.com/mortie23/fastapi-linear-model-demo.

git clone https://github.com/mortie23/fastapi-linear-model-demo.git

Then lets build the Docker image.

docker build -t mortimerxyz/linear:1.0 .

Docker files define layers. In this Docker file we start from a base image (which has Python3.9 on it) and then go through steps to build up the image by copying files we have in the repo. We also need to install the Python packages from the requirements.txt files within the image and then define the command that will run the service when the container is run.

FROM python:3.9

WORKDIR /

COPY ./requirements.txt ./

RUN pip install --no-cache-dir --upgrade -r requirements.txt

COPY ./fastapiApp /app

CMD ["uvicorn", "app.model:app", "--host", "0.0.0.0", "--port", "80"]

Windows using PowerShell

Since I first run this on Windows, I have the logs here. The Windows build took significantly longer than the Linux one below.

Sending build context to Docker daemon  3.322MB
Step 1/6 : FROM python:3.9
 ---> 05cae63731aa
Step 2/6 : WORKDIR /
 ---> Running in 1e478e292609
Removing intermediate container 1e478e292609
 ---> be030c7a42e0
Step 3/6 : COPY ./requirements.txt ./
 ---> 2487f2c14aec
Step 4/6 : RUN pip install --no-cache-dir --upgrade -r requirements.txt
 ---> Running in f8b71525170b
Collecting fastapi[all]
  Downloading fastapi-0.79.0-py3-none-any.whl (54 kB)
     ---------------------------------------- 54.6/54.6 KB 1.4 MB/s eta 0:00:00
Collecting uvicorn
  Downloading uvicorn-0.18.2-py3-none-any.whl (57 kB)
     ---------------------------------------- 57.0/57.0 KB ? eta 0:00:00
[...]
Successfully installed [...]

 ---> b901eee1947f
Step 5/6 : COPY ./fastapiApp /app
 ---> b65d2074826b
Step 6/6 : CMD ["uvicorn", "app.model:app", "--host", "0.0.0.0", "--port", "80"]
 ---> Running in 8f20e0e45191
Removing intermediate container 8f20e0e45191
 ---> 24ba8c92979c
Successfully built 24ba8c92979c
Successfully tagged mortimerxyz/linear:1.0

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

WSL on bash

After switching Docker to Linux containers I ran the build again from WSL bash shell. I’m going to push this one so I can then pull it on my Chromebook (Debian) and test using it.

[+] Building 74.9s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                     0.1s
 => => transferring dockerfile: 430B                                     0.0s
 => [internal] load .dockerignore                                        0.1s
 => => transferring context: 2B                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.9           10.3s
 => [auth] library/python:pull token for registry-1.docker.io            0.0s
 => [1/5] FROM docker.io/library/python:3.9@sha256:[...]                51.3s
 => => resolve docker.io/library/python:3.9@sha256:[...]f                0.0s
 => => sha256: [...] 2.22kB / 2.22kB                                     0.0s
 => [internal] load build context                                        0.1s
 => => transferring context: 832B                                        0.1s
 => [2/5] COPY ./requirements.txt ./                                     0.6s
 => [3/5] RUN pip install --no-cache-dir --upgrade -r requirements.txt  11.8s
 => [4/5] COPY ./fastapiApp /app                                         0.0s
 => exporting to image                                                   0.6s
 => => exporting layers                                                  0.6s
 => => writing image sha256:[...]                                        0.0s
 => => naming to docker.io/mortimerxyz/linear:1.0                        0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

Test running it

Once it is built we need to test that it runs.

docker run -p 80:80 mortimerxyz/linear:1.0

You should expect to see something like this in the terminal if you are successful.

INFO:     Started server process [1324]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
INFO:     192.168.0.19:23413 - "GET /weight/150 HTTP/1.1" 200 OK

You should be able to call the service from your browser or programmatically now (on port 80).

Publish it

To follow this process, you don’t need this step, but if you do, you will want to sign up for Docker Hub.

Sign up to Docker Hub
Create an access token

docker login -u mortimerxyz

docker push mortimerxyz/linear:1.0

The push refers to repository [docker.io/mortimerxyz/linear]
849958a0eb7c: Preparing
a7b3e42a4b41: Pushed
c00d727af360: Pushed
27a391641a6d: Pushed
6bf06ec166f0: Pushed
b2809f3ef892: Pushed
1a31622a1cf9: Pushed
b588d5e23fdf: Pushed
f22fec08a08a: Pushed
69c9ca396f64: Pushed
b55de1dfa670: Pushed
91896eadc954: Pushed
8b5e3ef66306: Pushed
63a0eaed8845: Pushed
e673d0986f90: Pushed
5e610cf6f254: Skipped foreign layer
d6fdd6832d95: Skipped foreign layer
1.0: digest: sha256:[...] size: 4232

Test it on another platform

Now we see the benefit of all of this (hopefully). To get the whole thing running on another platform (another local machine or even a cloud VM) we can just to the following:

docker pull mortimerxyz/liner:1.0
# run with the IMAGE ID or the REPOSITORY:TAG
docker run -p 80:80 mortimerxyz/liner:1.0

Success and testing calling it is all good!

curl localhost:80/weight/200

{ "weight": 90.275 }

Databricks sample NFL MLFlow

Wed, 03 Aug 2022 12:00:00 GMT

Fit a decision tree model for predicting win or loss

The code this post is relating to is here: https://github.com/mortie23/databricks-nfl-data.

This is using:

Azure Databricks
Azure Data Lake Storage (ADLS) Gen2

We are again looking at the NFL season for 2014.

In the previous blog post we looked at how to take a series of CSV files and load them into Databricks as delta tables backed by parquet files.

This time we will demonstrate using Spark and MLFlow train a model and host an model serving REST end point.

Visualizing data

Before we try to fit a model it is always a good idea to look at the distribution of the features. This example shows the distribution of the total yards.

Using MLFlow to register model

We will use MLFlow to track our “experiments”. Trained models and any other “artifacts” that you choose to log can be tracked using MLFlow. In this example we will track the RMSE and the trained model.

mlflow.sklearn.autolog()

def trainModel():
  '''
    Train a Descision tree model
    Return:
      run_id: an UUID representing the MLFLow model run
  '''
  with mlflow.start_run():
    # The input features
    inputCols = ["TOTAL_FIRST_DOWNS", "PASSING_FIRST_DOWNS","YARDS_PER_PLAY","FUMBLES","TOTAL_YARDS","INTERCEPTIONS","PUNTS"]
    va = VectorAssembler(inputCols=inputCols,outputCol="features")
    # Define the Decision tree regressor
    dt = DecisionTreeRegressor(labelCol="WIN_LOSS_BINARY", featuresCol="features", maxDepth=4)
    evaluator = RegressionEvaluator(metricName = "rmse", labelCol="WIN_LOSS_BINARY")

    grid = ParamGridBuilder().addGrid(dt.maxDepth, [3, 5, 7, 10]).build()
    cv = CrossValidator(estimator=dt, estimatorParamMaps=grid, evaluator=evaluator, numFolds = 10)
    pipeline = Pipeline(stages=[va, dt])

    # Now call fit to train the model and then use MLFlow to log the model
    model = pipeline.fit(df)
    mlflow.spark.log_model(spark_model=model, artifact_path="model")

    # Log the RMSE to MLFlow
    predictions = model.transform(df)
    rmse = evaluator.evaluate(predictions, {evaluator.metricName: "rmse"})
    mlflow.log_metric("rmse", rmse)

    # Return the run
    run_id = mlflow.active_run().info.run_id
    return run_id

run_id = trainModel()

# Register the model
model_name = "nfl-decisiontree"
artifact_path = "model"
model_uri = f"runs:/{run_id}/{artifact_path}"

model_details = mlflow.register_model(model_uri=model_uri, name=model_name)

MLFlow user interface

When this code has run, you will be able to see the experiment and the associated runs.

Adding metadata

Once we click on the model we can see the RSME we logged and any other artifacts related to the trained model.

Model lifecycle

MLFlow helps you to track the models and promote them through environments.

Checking model outputs

Vizualising the tree.

Feature importance

Feature importance is a measure of information gain. It is scaled from 0. Let’s map the features to their proper names to make them easier to read.

Feature	Importance
INTERCEPTIONS	0.392
TOTAL_YARDS	0.289
FUMBLES	0.131
YARDS_PER_PLAY	0.085
PUNTS	0.035
TOTAL_FIRST_DOWNS	0.034
PASSING_FIRST_DOWNS	0.030

Model serving

Testing the inference REST API

{
  "columns": [
    "TOTAL_FIRST_DOWNS",
    "PASSING_FIRST_DOWNS",
    "YARDS_PER_PLAY",
    "FUMBLES",
    "TOTAL_YARDS",
    "INTERCEPTIONS",
    "PUNTS"
  ],
  "data": [[28, 15, 7.6, 1, 495, 0, 3]]
}

Example of calling

Databricks provides some light weight examples of how to call it with CURL or Python. The example doesn’t provide how to call with passing data.

import os
import requests
import numpy as np
import pandas as pd

def create_tf_serving_json(data):
  return {'inputs': {name: data[name].tolist() for name in data.keys()} if isinstance(data, dict) else data.tolist()}

def score_model(dataset):
  url = 'https://<workspace>.azuredatabricks.net/model/nfl-decisiontree/1/invocations'
  headers = {'Authorization': f'Bearer {os.environ.get("DATABRICKS_TOKEN")}'}
  data_json = dataset.to_dict(orient='split') if isinstance(dataset, pd.DataFrame) else create_tf_serving_json(dataset)
  response = requests.request(method='POST', headers=headers, url=url, json=data_json)
  if response.status_code != 200:
    raise Exception(f'Request failed with status {response.status_code}, {response.text}')
  return response.json()

Databricks sample NFL dataset

Tue, 02 Aug 2022 12:00:00 GMT

Create a Databricks database with sample NFL data

The code this post is relating to is here: https://github.com/mortie23/databricks-nfl-data.

This is using:

Azure Databricks
Azure Data Lake Storage (ADLS) Gen2

Create your storage account and container

In this demo I have created a storage account called nfl and a container within the account called nfldata I have added my Azure Active Directory account as a Storage Blob Data Contributor.

Load data to ADLS

I am using Azure Storage Explorer to do this. You could also use the Azure portal, the azcopy cli or many other methods.

Clone this repo to your Databricks

Open the notebook called db.sql.py

Run the Notebook

First we will create the Hive database for the tables.

create database nfl

The notebook has two functions.
- Reading a CSV file and creating a Delta table
- Reading all files in an ADLS container and running the aforementioned function for each.

Test the results

The notebook has a sample query to test the results. This will get all players from the SF 49ers and sort them by the number rushing yards they had in each game.

select
  p.PLAYER_NAME
  , p.FIELD_POSITION
  , t.TEAM_SHORT
  , p.RUSHING_YARDS
  , p.PASSING_YARDS
from
  nfl.player p
  inner join nfl.team_lookup
on p.team_id = t.team_id
  and t.team_short = 'SF'
order by p.RUSHING_YARDS desc

For the 2014 NFL season, for the 49ers, Frank Gore and Colin Kaepernick were racking up the rushing yards.

Top 20

-	PLAYER_NAME	FIELD_POSITION	TEAM_SHORT	RUSHING_YARDS	PASSING_YARDS
1	Frank Gore	RB	SF	158	0
2	Colin Kaepernick	QB	SF	151	114
3	Frank Gore	RB	SF	144	0
4	Frank Gore	RB	SF	119	0
5	Frank Gore	RB	SF	107	0
6	Frank Gore	RB	SF	95	0
7	Frank Gore	RB	SF	81	0
8	Frank Gore	RB	SF	66	0
9	Colin Kaepernick	QB	SF	64	248
10	Frank Gore	RB	SF	63	0
11	Colin Kaepernick	QB	SF	63	204
12	Frank Gore	RB	SF	63	0
13	Colin Kaepernick	QB	SF	58	218
14	Carlos Hyde	RB	SF	55	0
15	Colin Kaepernick	QB	SF	54	245
16	Carlos Hyde	RB	SF	50	0
17	Frank Gore	RB	SF	49	0
18	Colin Kaepernick	QB	SF	46	141
19	Carlos Hyde	RB	SF	43	0
20	Frank Gore	RB	SF	38	0

Model scoring API examples

Sat, 09 Jul 2022 12:00:00 GMT

A basic example of a predictive model scoring API

The associate Github repo for this blog post is https://github.com/mortie23/fastapi-linear-model-demo

To demonstrate model scoring hosted as API

The reason for doing this is to create a very light weight and infrastructure free way to demonstrate calling a model inference API from a web front end. It doesn’t have many things you would potentially need for a real life situation like:

authentication
authorization
auto scaling

It is not meant to be used for production, but simply to demonstrate the mechanics. We are going to do this both with R (Plumber) and Python (FastAPI). This should be all good to run on Windows or Linux. I have developed it on Debian (from ChromeOS).

The model

Before we even start we need a simple example of a model. We will use linear regression to model the relationship between height and weight. No need to for an ML model framework or a model registry for this demonstration. We are demonstrating to REST API part of things.

A small sample of heights and weights was used.

The model in this case is a combination of the model function:

weight(height) = m height + b

and the model parameters:

parameter	value
m	0.5772
b	-25.165

Python

Coded in a Python function

This model is very simply to code in Python. Note the decorator code used by FastAPI before the function definition.

@app.get("/weight/{height}")
async def weight(height: int) -> dict:
    """
    This function takes a height and returns an estimated weight
        Args:
            height: user inputs the height
        Return:
            weight
        Model:
            y = mx + b
    """
    weight = 0.5772 * height - 25.165
    return {"weight": weight}

Now, how do we host this as a scoring (predicting or inference) API?

Using FastAPI

To get this up and running I will assume you have Python3 already. To get started you might want to create a new virtual environment and then install the dependencies:

pip install "fastapi[all]"
pip install uvicorn

The python script in the associated repo creates the REST service. To start it up use:

uvicorn model:app --reload

You’ll see something like the below log if it is working and you can browse to the end point using http://127.0.0.1:8000/weight/150 or for the Swagger docs using http://127.0.0.1:8000/docs:

INFO:     Will watch for changes in these directories: ['C:\\git\\fastapi']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [36128] using watchgod
INFO:     Started server process [24032]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

The part that does the “scoring” in the Python script is:

weight = 0.5772*height - 25.165

R

We will use Plumber for this. I am on a (brilliant) Chromebook at the moment which has a Debian distro on it. So I need this to get going:

sudo apt install libsodium-dev
sudo apt install libcurl4-openssl-dev

install.packages("plumber")

Using Plumber

#* Return the estimated weight
#* @param height the height
#* @get /weight
function(height) {
  0.5772 * as.numeric(height) - 25.165
}

If you look at the ./modelApi.R script in the associated repo, I have added a shebang (#!) to the script (pointing to the Rscript executable) and given the script execute permissions (chmod +x modelApi.R). We can start the Plumber REST service like this:

./modelApi.R

You will get something like this in the log if it has worked:

Running plumber API at http://127.0.0.1:8000
Running swagger Docs at http://127.0.0.1:8000/__docs__/

Adding an interface and calling the API

A basic HTML front end has been created to demonstrate calling the API with user input.

In this example we are just using some JavaScript and Jquery to get the user input, pass it to the API and then populate the front end with the score.

To test it I am just using the Live server VScode extension. If you have cloned this repo you, and have the extension, you can just right click on the index.html and click on Open with Live Server.

The JavaScript that does the calling of the Python FastAPI is:

var url = 'http://127.0.0.1:8000/weight/' + height;
$.get(url, function (data) {
  console.log(data.weight);
  weight.value = data.weight;
});

If we want to test this for the R Plumber API we would just change it to:

var url = 'http://127.0.0.1:8000/weight?height=' + height;
$.get(url, function (data) {
  console.log(data.weight);
  weight.value = data.weight;
});

Power BI PBIX Git version control

Tue, 14 Jun 2022 12:00:00 GMT

How to commit a Power BI PBIX to Git

NOTE: this blog post has been superseded by Power BI PBIP Git version control

This is in reference to the associated https://github.com/mortie23/pbix-git-version-control

What is the problem

A PBIX file, under the hood, it just a series of text files archived into a single file. For Power BI to read the file and display the dashboard in Power BI desktop or web it needs to uncompress the archive and read the files first.

Version control systems like Git are terrific at tracking text files and not designed to track files that are not text. If the file is text, then incremental changes are efficiently tracked and stored. Otherwise Git will just track versions by keeping a full history of the file.

Git will see a commit of a PBIX file and say “I can’t track the changes, so I will just keep another version of the file”. This results in:

You cannot use it to see what the changes were between versions
The size of the repo starts to blow out the more you commit

We want Git to track the underlying files in the archive and in the process we want the Power BI dashboard developers to not have to worry about any of these technical details.

Git Hooks

Git hooks are scripts that Git can run before or after most Git actions (like commit, receive, checkout etc). They are kept in the hidden .git directory and are local client files. This means they are not tracked by Git and are not synced between remotes when pushed or pulled. There are ways to get around this.

One way to do this is to create a hooks directory in the root of your repo and then run this configuration.

git config core.hooksPath './hooks'

It tells Git to look for the hooks scripts in this directory instead of the default in the .git directory.

How we solve the PBIX problem

We want Git to do the following whenever a PBIX is committed:

Extract the PBIX into its underlying files
Commit these files instead
Do not commit the PBIX file

We want Git to do the following whenever a PBIX is pulled/merged/checked out

Compress the PBIX

Pre-requisite

This will only work for a PBIX file that sources its data from a remote Power BI XMLA source. If the data (DataModel) is contained within the PBIX then it will not work.

Running on client machine with:

Windows 10
Git 2.27.0.windows.1
GNU bash, version 4.4.23(1)-release (x86_64-pc-msys)
Powershell Core 7.2.4
7-Zip 21.07 (x64)
- The 7z.exe cli needs to be in your environment path (you can do this be adding $env:Path += ";C:\Program Files\7-Zip" to your Powershell profile)

Example

First let’s create the PBIX.

Dataset

I have a cloud hosted PostGres database and have loaded some sample NFL data into it. I have subsequently loaded this to Power BI as a XMLA dataset.

Lets open up Power BI desktop and connect to this dataset.

Dashboard

We will create something very simple that includes:

Text box
Map viz

Use PBIT file type

We need to save this as a PBIT (Power BI Template) to ensure that one of the underlying files is created (DataMashup).

But then we will rename the extension back to PBIX to make things easier (i.e. double clicking the file will open it in Power BI desktop and make it easier to edit).

Ensure our configuration is set to run hooks

git config core.hooksPath './hooks'

Lets commit the file and see what happens

git add nfl.pbix
git commit -m "pbix commit"

/c/git/github/mortie23/pbix-git-version-control/hooks
nfl.pbix

Expanding pbix files...

nfl.pbix
Archive:  nfl.pbix
  inflating: ./nfl.pbix.expanded/Version
  inflating: ./nfl.pbix.expanded/[Content_Types].xml
  inflating: ./nfl.pbix.expanded/DataMashup
  inflating: ./nfl.pbix.expanded/DiagramLayout
  inflating: ./nfl.pbix.expanded/Report/Layout
  inflating: ./nfl.pbix.expanded/Settings
  inflating: ./nfl.pbix.expanded/Metadata
  inflating: ./nfl.pbix.expanded/Connections
  inflating: ./nfl.pbix.expanded/Report/StaticResources/SharedResources/BaseThemes/CY22SU03.json
  inflating: ./nfl.pbix.expanded/SecurityBindings

7-Zip 21.07 (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-12-26

Scanning the drive for archives:
1 file, 1775 bytes (2 KiB)

Extracting archive: .\nfl.pbix.expanded\DataMashup

WARNINGS:
There are data after the end of archive

--
Path = .\nfl.pbix.expanded\DataMashup
Type = zip
WARNINGS:
There are data after the end of archive
Offset = 8
Physical Size = 792
Tail Size = 975
Characteristics = Local Central

Everything is Ok

Archives with Warnings: 1

Warnings: 1
Files: 3
Size:       499
Compressed: 1775
Unstaged changes after reset:
M       README.md
[master 8e7d696] pbix commit
 11 files changed, 6 insertions(+)
 create mode 100644 nfl.pbix.expanded/Connections
 create mode 100644 nfl.pbix.expanded/DataMashup.7zexpanded/Config/Package.xml
 create mode 100644 nfl.pbix.expanded/DataMashup.7zexpanded/Formulas/Section1.m
 create mode 100644 nfl.pbix.expanded/DataMashup.7zexpanded/[Content_Types].xml
 create mode 100644 nfl.pbix.expanded/DiagramLayout
 create mode 100644 nfl.pbix.expanded/Metadata
 create mode 100644 nfl.pbix.expanded/Report/Layout
 create mode 100644 nfl.pbix.expanded/Report/StaticResources/SharedResources/BaseThemes/CY22SU03.json
 create mode 100644 nfl.pbix.expanded/Settings
 create mode 100644 nfl.pbix.expanded/Version
 create mode 100644 nfl.pbix.expanded/[Content_Types].xml

After the commit

The PBIX file itself was not committed, but instead the directory containing the files was committed.

On checkout

So now let’s delete the PBIX and then checkout and see what happens.

git checkout

/c/git/github/mortie23/pbix-git-version-control/hooks

Creating pbix files...

nfl.pbix.expanded
Removing ./nfl.pbix
rm: cannot remove './nfl.pbix': No such file or directory

7-Zip 21.07 (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-12-26

Scanning the drive:
2 folders, 3 files, 499 bytes (1 KiB)

Creating archive: DataMashup.7z

Add new data to archive: 2 folders, 3 files, 499 bytes (1 KiB)


Files read from disk: 3
Archive size: 535 bytes (1 KiB)
Everything is Ok

C:/git/github/mortie23/pbix-git-version-control nfl.pbix
Remove DataMashup C:/git/github/mortie23/pbix-git-version-control/nfl.pbix.expanded/DataMashup

The PBIX was created and successfully opens in Power BI desktop.

Testing a change

We will make one small change and then look at the diff that is created.

Let’s just change the text in the textbox.

Save the PBIX file and then commit. See the key info from the log is that only 1 file was changed:

[master e8f64a9] feat: textbox change
 1 file changed, 0 insertions(+), 0 deletions(-)
 rewrite nfl.pbix.expanded/Report/Layout (87%)

We still have the issue of Git thinking this file is binary when it is actually a text file with JSON. This may be because the JSON is minified?

Things to consider

This doesn’t test many edge cases. It will/may:

Stop you from committing any file with a PBIX extension
Crash if the structure of the expanded PBIX isn’t what I have tested
Still track some of the text files as binary

Git post-receive hook for deployment of a static site

Fri, 06 May 2022 12:00:00 GMT

Using a post receive hook for website deployment

Other deployment methods

Firstly, let’s mention that there are other potentially easier ways to do this type of deployment. A Github action or using a service such a Netlify can do the build and deployment for you.

How to do this

This approach is a lower level approach where all you need need is SSH access to the webserver you are hosting the site on.

Let’s say that we use a static site generator to build our website. The build process creates the static site in a public directory in the root of the repo.

Usually the default would be to configure the public generated site to be ignored by git. For this process we would need to include the generated site.

Directory structure on your webserver

We will use the home directory of the user account you are using to SSH (so it has permissions to access the required files).

We will create a git directory and a bare repo for our site.

# Change to home directory
cd ~
# Create a tmp directory to be used later
mkdir tmp
# Create git directory and change into it
mkdir git
cd git
# Initialise a bare repo
git init --bare

You will see a directory structure created which should contain a hooks directory.

The hook’s shell script

In the hooks directory we will create a post-receive hook script.

cd hooks
touch post-receive
chmod +x post-receive

Then paste the contents of the following script in there. Replace the <username> with the username that you will SSH to the server.

#!/bin/bash

unset GIT_DIR
set GIT_DIR .
# Change into your bare repo
cd /home/<username>/git/
# Set the work tree (the files in the repo) to the temp directory we created
git --work-tree=/home/<username>/tmp checkout -f
# Sync the files in the public directory of the repo to the root of the Nginx webserver
rsync -r /home/<username>/tmp/public/ /var/www/html

Add your webserver as a git remote

The repository on your local machine will have an origin remote (assuming you started by creating it on a remote git server like Github). You will need to add a remote which points to your webserver.

git remote add webserver ssh://<username>@<server>:<ssh-port>//home/<username>/git

Deploying change to the site

When you have changes to your site build to the public directory you can deploy them to the server using:

git push webserver master

Raspberry Pi time lapse server

Fri, 18 Mar 2022 12:00:00 GMT

Building a time lapse server with a Raspberry Pi and a webcam

I had a Raspberry Pi lying around that I use to use as a media server, and an old webcam that I used before laptops came with webcams. I had an idea to create a time lapse server.

What is does

It simply takes a photo every minute and saves it to a directory within the path of a webserver (Nginx).

Then a basic webpage takes input from the user (me) a time stamp as a parameter and starts playing the images at 1 image per second (60 times faster than they were taken).

Details on setting it up

Firstly I wiped the memory card of the Raspberry Pi that original had the OpenELEC (a Kodi distribution for Raspberry Pis) operating system and installed Rasbian.

$> lsb_release -a
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

I then installed

git
Python 3
Nginx

I created a Github repository and cloned it down to the home directory of the logged in user (just the default pi user). I created a virtual Python environment for development.

In the Nginx webserver root directory I copied the webserver files.

Running the service

To ensure the Python service is running always in the background I ran.

nohup python photo-service.py &

This will take a photo every minute and save it to the Nginx hosted webserver at /var/www/html/webcam/webcam-yyyymmddhhmm.jpg

Then, knowing the IP address of the Raspberry Pi on our network, we can use a browser to see the photos.

Time lapse results

This is a time lapse of me at my desk played back at 5 frames per second.

This is a time lapse at me buring the midnight oil, coding away at 10 frames per second.

Dynamic Images in Power BI Dashboards

Thu, 17 Mar 2022 12:00:00 GMT

Dynamic images in Power BI

Getting images to be dynamic, such as changing in reaction to an change in the selected value of a slicer seems to have limited out of the box solutions in Power BI (at the time of this blog).

The solutions I have seen involve sourcing the images from a URL (already hosted images). Which is fine if you have access to a webserver to host them, but if you don’t you may feel you are stuck.

Another way using HTML

If you don’t have access to a webserver to host your images, you can host the images as text in two ways.

The images

SVG

To demonstrate using SVGs I sourced some from Font Awesome.

Base64

To demonstrate using Base64 encoding I got a png of an emoji and used a Base64 Image converter to convert to Base64.

Database table

I will demo this using a MySQL database. SQL code from Github repository

DROP TABLE `powerbi_image`;
CREATE TABLE `powerbi_image` (
    `picture_id` int NOT NULL AUTO_INCREMENT
    , `picture_name` varchar(100) NULL DEFAULT NULL
    , `picture_type` varchar(100) NULL DEFAULT NULL
    , `picture_source_code` longtext NULL DEFAULT NULL
    , PRIMARY KEY (`picture_id`)
  )
COLLATE='latin1_swedish_ci'
ENGINE=MyISAM
;

-- The full code containing the SVG and Base64 images
-- is in this associated repo which is linked above
INSERT INTO `powerbi_image`
  (`picture_name`, `picture_type`, `picture_source_code`)
  values
  ('smile', 'svg', '<html><svg><path d=""/></svg></html>')
  , ('icecream', 'svg', '<html><svg><path d=""/></svg></html>')
  , ('avocado', 'base64', '<html><img src="data:image/png;base64,"/></html>')
;

The Power BI dashboard

I connected Power BI to the MySQL table and on the canvas I have added three visualizations:

slicer (sourcing from the picture_name)
card (sourcing the picture_type)
HTML content (sourcing the picture_source_code)

The result is an image which updates in reaction to the changing selected value in the slicer.

Integrating Azure Cognitive with Retool

Fri, 03 Dec 2021 12:00:00 GMT

What is Retool

Retool is a simple tool for creating a web application. One of the things that makes Retool great is you’re never limited by what’s available out of the box. If you can write it with JavaScript (the most popular language for web developer) and an API.

What is Azure Cognitive Services

AzureCognitive Services makes it easy to integrate AI with your web applications. All it takes is an API call to embed the ability to do cool things.

What you need to get started

A Retool subscription and an Azure subscription

First step is to sign up for the Free trial of Retool (if you don’t already have a Retool subscription) and the Free trial of Azure (if you don’t already have an Azure subscription).

Creating the App

We are going to demo how to integrate the computer vision Azure Cognitive service. So we need a basic Retool application that loads up an image from your local PC. We also need some input boxes to put the response of the API call.

We need to give the input components the correct names so the the Javascript query we are going to write later will be able to access them.

tagsInput
captionInput
confidenceInput

// relevant js code to above input boxes
tagsInput.setValue();
captionInput.setValue();
confidenceInput.setValue();

Before we start to create the Javascript that will call the API, we need to include some libraries from Microsoft that will assist in the process. Retool has our back here, and has a nice way to load libraries.

We are going to use the browser libraries from the following NPM package @azure/cognitiveservices-computervision. I simply NPM installed the package and then sourced the minified libraries from \node_modules\@azure\cognitiveservices-computervision\dist and then hosted themselves on my own webserver. You need to have them in this order because the second is dependant on the first.

If you haven’t enables CORS on your webserver you will need to do this so that the libraries can be called by Retool. If you are using NGINX that would be something like adding the following to your NGINX config add_header 'Access-Control-Allow-Origin' '*' always;

We need to create a Javascript query which will call the Azure Cognitive services API when we load the image.

// our secrets (for simplicity hard coded, I do not recommend this approach)
const computerVisionKey = '<key>';
const computerVisionEndPoint = '<endpoint>';

// this is needed because the cognitive API needs binary data not base64
function _base64ToArrayBuffer(base64) {
  var binary_string = window.atob(base64);
  var len = binary_string.length;
  var bytes = new Uint8Array(len);
  for (var i = 0; i < len; i++) {
    bytes[i] = binary_string.charCodeAt(i);
  }
  return bytes.buffer;
}

const cognitiveServiceCredentials = new msRest.ApiKeyCredentials({
  inHeader: {
    'Ocp-Apim-Subscription-Key': computerVisionKey,
  },
});
const client = new Azure.CognitiveservicesComputervision.ComputerVisionClient(
  cognitiveServiceCredentials,
  computerVisionEndPoint,
);

// get the base64 image provided by browse to file from retool
const imageBase64 = cognativeImageButton.value[0];

const options = {
  maxCandidates: 5,
  language: 'en',
};

// make the call and write results to input box
client
  .describeImageInStream(_base64ToArrayBuffer(imageBase64), options)
  .then(result => {
    tagsInput.setValue(result.tags.join());
    captionInput.setValue(result.captions[0].text);
    confidenceInput.setValue(result.captions[0].confidence);
  })
  .catch(err => {
    console.log('An error occurred:');
    console.error(err);
  });

Alright, this missing piece to all this now is the Azure cognitive services.

Getting the Azure services

Firstly, after you sign up for your Azure subscription, you will need to create a Computer Visions resource.

Now that the endpoint has been created, lets get a key and the enpoint and paste them into the JS query in Retool.

// our secrets (for simplicity hard coded, I do not recommend this approach)
const computerVisionKey = '<key>';
const computerVisionEndPoint = '<endpoint>';

Testing it

Uploading this close up picture of an Eagle, and the computer vision API returns with a confidence of around 50% that it is a close up of an eagle. Not too bad.

Installing Prefect Server on Azure

Mon, 22 Nov 2021 12:00:00 GMT

Reading

Firstly I came across the following docs on how to install prefect server

Prefect Server 101 Deploying to GCP
Deploying to a Single Node Prefect Server

None of these outlined fully in detail what I needed, which was how to install from scratch on Azure.

The Azure VM

We are going to set ourselves up with a Linux VM with 4 GB of RAM and 60 GB of attached disk. By default, we will not be allowing any inbound ports, because we will be opening them with firewall rules later.

The machine will be based on the basic Ubuntu 20.04 image from Canonical.

The machine we will choose is the Standard_B2s machine which will have the resources that we need.

Set the public inbound ports to none.

Create an attached disk that is Standard SSD and 64 GB in size (E6 performance tier).

For the rest of the image, set to defaults for Networking, Management, Advanced, Tags.

Key file

Firstly we are going to move the keyfile from where ever your browser downloaded it to your .ssh directory in your home directory. I am going to use Powershell on Windows for this example (but you could just as easily use any other shell or OS).

ls | Where-Object {$_.Name -eq 'prefect_key.pem'}

        Directory: C:\Users\<username>\.ssh


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a---        24/11/2021   9:35 AM           2498   prefect_key.pem

Now we can go to our resource and connect.

Networking

To enable yourself to connect (using Secure Shell to connect to your remote computer) you must create a networking rule to allow your public IP address to access inbound port 22.

The easiest way to find your public IP address is to ask Google.

We are going to add three inbound port rule so that we can:

Connect to the server (Port 22)
Enable access to the UI (Port 8080)
Enable the UI to access the GraphQL API (Port 4200)

For now we are going to only enable these ports to be open for us.

This shows how to enable TCP through port 22 for only our public IP address.

We need to repeat the steps for ports 8080 and 4200.

Connecting

ssh -i C:\Users\<username>\.ssh\prefect_key.pem azureuser@<azure-vm-public-ip>

You’ll be prompted that the authenticity of host can’t be established and asked if you want to continue.
Respond yes and the host will be saved to your know_hosts file so you will not have to be prompted again.

The authenticity of host '<azure-vm-public-ip> (<azure-vm-public-ip>)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghij/12345678901234567+g/abcdefghijkl.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<azure-vm-public-ip>' (ECDSA) to the list of known hosts.
azueruser@<azure-vm-public-ip>: Permission denied (publickey).

Installing

Following the steps

This is where I am now going to basically follow the steps in Prefect Server 101 Deploying to GCP. However, a few modifications have been made as some commands seemed to fail on my attempt.

Installing Python (and Pip)

Update your package information:

sudo apt update

Check Python3 is installed:

It was already installed on my Azure Standard_B2s machine.

python3 --version

Install Pip:

sudo apt install python3-pip

Install the Python 3 virtual environment manager:

sudo apt install python3-venv

Installing Docker

sudo apt install docker.io

Add your user to the docker group

sudo usermod -aG docker ${USER}

After this you will need to disconnect your SSH session and reconnect to pickup the new group membership.

# From remote
exit
# From local
ssh -i C:\Users\<username>\.ssh\prefect_key.pem azureuser@<azure-vm-public-ip>

Get the Docker Compose repository:

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Apply executable permissions:

sudo chmod +x /usr/local/bin/docker-compose

Installing Prefect

Now let’s install Prefect. We are going to do this from a virtual Python environment. Firstly, create and activate your virtual environment.

python3 -m venv prefect
source prefect/bin/activate

pip install prefect

Starting Prefect Server

Before we start the server infrastructure, we need to modify the UI endpoint so it knows how to access the GraphQL server correctly (note that this requires Prefect version 0.10.3+). We can do this by creating a user config.toml and adding the endpoint of our VM:

mkdir ~/.prefect
touch ~/.prefect/config.toml

Add the following to the file we just created (replacing the text in bold with your own IP). Note, I used vim to edit the file.

[server]
  [server.ui]
    apollo_url = "http://<azure-vm-public-ip>:4200/graphql"

Start the prefect server!

prefect server start --expose

Once the output looks like the following, go to this URL in your browser http://azure-vm-public-ip:8080.

 WELCOME TO

   _____  _____  ______ ______ ______ _____ _______    _____ ______ _______      ________ _____
  |  __ \|  __ \|  ____|  ____|  ____/ ____|__   __|  / ____|  ____|  __ \ \    / /  ____|  __ \
  | |__) | |__) | |__  | |__  | |__ | |       | |    | (___ | |__  | |__) \ \  / /| |__  | |__) |
  |  ___/|  _  /|  __| |  __| |  __|| |       | |     \___ \|  __| |  _  / \ \/ / |  __| |  _  /
  | |    | | \ \| |____| |    | |___| |____   | |     ____) | |____| | \ \  \  /  | |____| | \ \
  |_|    |_|  \_\______|_|    |______\_____|  |_|    |_____/|______|_|  \_\  \/   |______|_|  \_\


   Visit http://localhost:8080 to get started

For the remaining documentation please refer to Prefect Server 101 Deploying to GCP.

Secret Santa with Node.js and Gmail

Sat, 16 Oct 2021 12:00:00 GMT

What is Secret Santa

an arrangement by which a group of friends or colleagues exchange Christmas presents anonymously, each member of the group being assigned another member for whom to provide a small gift, typically costing no more than a set amount.

We have an extra requirement in our family:

No person should get their partner in the random anonymous allocation of Secret Santa.

Other options

A quick google search shows that there are much easier ways to get a random Secret Santa emailing system going. I have listed the top two Google search results here. But what is the fun in that. Let’s learn something about sending email with Node.js.

Forking a Git repository

I had assumed I wasn’t the first person to want to, or have the need to give this a try. My first step was to see what Git repositories already existed. I found one that did 90% of what I wanted (credit to Dmitri Kunin).

https://github.com/DKunin/secret-santa-module

So I forked the repository and started making changes.

Stop partners from getting each other

The repository I forked used another package called secret-santa-shuffler that was also created by Dmitri Kunin. I took the index.js from this module and created it as a separate module in my forked repository so I could modify it.

I added a function (partner()) to get the partner of a sender (s). Then added to the already existing condition so that I had both of:

(sender (s) is not equal to receiver (receivers[j]))
(receiver (receivers[j]) is not equal to sender’s partner (partner()))

// If condition prior to change
if (s !== receivers[j]) {
  ...
}
// if condition after change
if (s !== receivers[j] && receivers[j] != partner()) {
  ...
}

Use Gmail as the email service

Converting to use Gmail was quite simple. I registered a new Gmail address for the purpose of sending these emails. The extra step is to allow Less secure apps.

Since I am using this once a year, and only using for a fleeting moment, I can turn this feature on https://myaccount.google.com/lesssecureapps, and then turn it off when I am finished.

// Using mailgun prior to change
var mg = require('nodemailer-mailgun-transport');
var transporter = nodemailer.createTransport(
  mg({
    from: 'secret@santa.com',
    to: '',
    subject: 'Secret Santa',
    html: '',
  }),
);
// Using gmail after change
var transporter = nodemailer.createTransport({
  service: 'gmail',
  auth: {
    user: useremail,
    pass: userpassword,
  },
});

Sending the emails

My modified fork of Dmitri Kunin’s repository is here https://github.com/mortie23/secret-santa-module.

Installation

git clone
npm install

Configure your environment variables

useremail=hollywoodsecretsanta@gmail.com
userpassword=qwerty1234
secretsantaname=Hollywood

Debug mode by default

By default both scripts confirmemail.js and secretsanta.js scripts have debug: true. Suggest you run it in this mode first. In this mode no emails are sent, only results logged to console.

// Example
santaModule(list, {
  debug: true,
  template: `<div><h2>It's the ${secretsantaname} Secret Santa!</h2><div>You should prepare for <h3>{%=o.to%}</h3></div></div>`,
}).then(console.log);

When you are happy, turn this flag to false.

# Run this first if you want to confirm that the emails are reaching people
npm run confirmemail
# Then run this to send them out
npm run secretsanta

List Example

This program will not allow partners to have each other in the Secret Santa.

Name,email,partner
Brad Pitt,brad.pitt@email.com,angelina.jolie@email.com
Angelina Jolie,angelina.jolie@email.com,brad.pitt@email.com
Justin Beiber,justin.beiber@email.com,hailey.bieber@email.com
Hailey Bieber,hailey.bieber@email.com,justin.beiber@email.com
Jake Gyllenhaal,jake.gyllenhaal@email.com

confirmemail Sent message

It’s the Hollywood Secret Santa!
Please reply to confirm your email

secretsanta Sent message

It’s the Hollywood Secret Santa!
You should prepare a present for: Brad Pitt (brad.pitt@email.com)

Debug mode on output

[
  {
    "giver": "brad.pitt@email.com",
    "string": "<div><h2>It's the Mortimer Secret Santa!</h2><div>You should prepare for <h3>undefined (hailey.bieber@email.com)</h3></div></div>"
  },
  {
    "giver": "angelina.jolie@email.com",
    "string": "<div><h2>It's the Mortimer Secret Santa!</h2><div>You should prepare for <h3>undefined (jake.gyllenhaal@email.com)</h3></div></div>"
  },
  {
    "giver": "justin.beiber@email.com",
    "string": "<div><h2>It's the Mortimer Secret Santa!</h2><div>You should prepare for <h3>undefined (brad.pitt@email.com)</h3></div></div>"
  },
  {
    "giver": "hailey.bieber@email.com",
    "string": "<div><h2>It's the Mortimer Secret Santa!</h2><div>You should prepare for <h3>undefined (angelina.jolie@email.com)</h3></div></div>"
  },
  {
    "giver": "jake.gyllenhaal@email.com",
    "string": "<div><h2>It's the Mortimer Secret Santa!</h2><div>You should prepare for <h3>undefined (justin.beiber@email.com)</h3></div></div>"
  }
]

How I develop Python in VSCode

Sat, 28 Aug 2021 12:00:00 GMT

Why

Working on Javascript code in Visual Studio Code (VSCode) for the best part of last year got me use to all the good stuff. Autoformatting, linting etc. Coming back to a role where Python and SQL are the primary languages and I had the need to find out the best way to develop in Python.

Autoformatting

I’ll use this example valid Python code to demo my settings. This code isn’t nice. There is a lot of bad spacing, and some arguments being passed to a function of the wrong type.

import pandas as pd

def addcolumn(df:pd.DataFrame)->pd.DataFrame:
    '''
    Add a generic column to a dataframe
      :param df: the source dataframe
      :return: a dataframe with a new column populated with new value
    '''
    df['newcol']='new value'
    return df
df = pd.DataFrame({'numbers':[1,2,3],'colors':['red','white','blue']})
print('Test add column')
print(addcolumn(df))

def addition(x:int,y:int)->int:
    '''
    Add two numbers function using mypy types
      :param x: the first number
      :param y: the second number
      :return: the sum of the two arguments
    '''
    return x+y
print('Test typing')
print(addition(1,2))
print(addition('1','2'))

As Donald Rumsfield said “But there are also unknown unknowns—the ones we don’t know we don’t know”. And if you didn’t know, we don’t have to manually enforce styles by hitting space bar a lot of times.

To mirror the way I setup my VSCode, follow along. Add this setting to you settings.json

{
  "editor.formatOnSave": true
}

When you save this setting you will get this toaster modal:

Click Yes and your terminal should load up a pip install command. If you have issues pip install, that is a story for another time.

Now when we save the file, all that hideous formatting is gone. We don’t depend on ourselves to have to remember to hit Alt-Shift-F to format, or even worse, have to space bar our way through code to manually format it.

import pandas as pd


def addcolumn(df: pd.DataFrame) -> pd.DataFrame:
    '''
    Add a generic column to a dataframe
      :param df: the source dataframe
      :return: a dataframe with a new column populated with new value
    '''
    df['newcol'] = 'new value'
    return df


df = pd.DataFrame({'numbers': [1, 2, 3], 'colors': ['red', 'white', 'blue']})
print('Test add column')
print(addcolumn(df))


def addition(x: int, y: int) -> int:
    '''
    Add two numbers function using mypy types
      :param x: the first number
      :param y: the second number
      :return: the sum of the two arguments
    '''
    return x+y


print('Test typing')
print(addition(1, 2))
print(addition('1', '2'))

Typing in Python

These are type hints and not static typing.

Add these to your settings.json

{
  "python.linting.mypyEnabled": true,
  "python.linting.mypyPath": "~/python-venv/myvenv/bin/mypy"
}

You will need to install mypy using pip install mypy.

Ok, now this will not break the program or stop it from running. The issue with the code, you may notice is with calling the addition function with str arguments. It doesn’t cause an issue, but you can imagine another case where using the wrong type as arguments would cause an error, and we want our development environment to let us know this.

We can run mypy from the command line against a python file to see what it outputs.

# Windows
mypy.exe .\addition.py

The Windows one will return this:

This info will be surfaced right into VScode with the options we enabled in the settings.json file.

Docstrings

Last but not least, using standard docstrings for functions helps to surface other info as well.

Git Cleanup

Thu, 29 Jul 2021 12:00:00 GMT

What is the issue?

Collaborating on a repo and someone accedientally commits a very large file that is not very compressable. It makes the repo size way too large and needs to be reomved. We’ll use the repo here as an example. Firstly let’s look at the size of the directory, including the .git database. Before things go wrong.

# Check the complete size of the directory
du -hcsb .
# 44,924 .

Ok, so it is only about 45 K in size.

git checkout master
mkdir lotsoffiles
touch ./lotsoffiles/lotsoffiles-gen.sh

This is the lotsoffiles-gen.sh script:

#! /bin/bash
for n in {1..20}; do
  truncate -s 5K file$n.txt
done
## this is a big file that is not very compressable
wget https://chromedriver.storage.googleapis.com/93.0.4577.15/chromedriver_mac64.zip

Run the script to generate the files:

cd lotsoffiles
bash ./lotsoffiles-gen.sh

Now lets create the mess in git and on the remote

git add .
git commit -m "big commit lots of files one big bad file"
git push

What is our disk size now?

du -hsb .
# 16,479,850        .

Ok, committing a 7 MB file has blown out the entire size of the directory to 16.5 MB and that big commit with lots of files is hard to separate. The one big file we want gone and it is now in master branch on the remote. How do we not only get rid of it, but ensure that the .git database is clean?

Oh no, wait, what has happened?

Someone on another branch has pulled master and made some changes

git checkout working
git pull origin master

Some other files were added.

mkdir working
touch ./working/test.js

git add .
git commit -m "a working commit with the big bad file"
git push origin working

Someone pulled from master and kept working merrily on their branch. They made commits on top of the one we don’t want and pushed to the remote.

The last thing to fix

In an attempt to remove the big file, the user working on master deletes the file and makes another commit push. The big file is gone, but not from history.

git checkout master
rm ./lotsoffiles/chromedriver_mac64.zip

Now lets commit this.

git add .
git commit -m "reversing big file"
git push

But what is the size now? It looks like the deleting of the file doesn’t remove the size from the .git database.

du -hsb .
# 8,321,042 .

Now the cleanup

git checkout master
git log

Find the last commit on master that we want:

commit 777e7a7ecaef0dfef8f2cf919cca2681296304b0 (HEAD -> master
Author: Christopher Mortimer <christopher@mortimer.xyz>
Date:   Thu Jul 29 21:45:00 2021 +1000

    reversing big file

commit 431bd1123da15c597e496c413116919e0d9ae1bb
Author: Christopher Mortimer <christopher@mortimer.xyz>
Date:   Thu Jul 29 21:41:02 2021 +1000

    big commit lots of files one big bad file

commit 6b62247468340f8ec9ee61a2aee16bd5d43f038b
Author: Christopher Mortimer <christopher@mortimer.xyz>
Date:   Thu Jul 29 21:39:32 2021 +1000

    before things went wrong

Now lets reset the head back to that commit. Noting that before we go this, retain the files you actually want somewhere else on your machine. You’ll need to recommit that work.

git reset --hard 6b62247468340f8ec9ee61a2aee16bd5d43f038b
git push -f

Great, now lets do the same for the working branch:

git checkout working
git log

Find the last commit on master that we want:

commit a3e751174204b778e1f86228014d1bdb929d57c2 (HEAD -> workin
Author: Christopher Mortimer <christopher@mortimer.xyz>
Date:   Thu Jul 29 21:42:19 2021 +1000

    a working commit with the big bad file

commit 431bd1123da15c597e496c413116919e0d9ae1bb
Author: Christopher Mortimer <christopher@mortimer.xyz>
Date:   Thu Jul 29 21:41:02 2021 +1000

    big commit lots of files one big bad file

commit 6b62247468340f8ec9ee61a2aee16bd5d43f038b (origin/master,
Author: Christopher Mortimer <christopher@mortimer.xyz>
Date:   Thu Jul 29 21:39:32 2021 +1000

    before things went wrong

Noting that before we go this, retain the files and changes from the working commit you actually want somewhere else on your machine. You’ll need to recommit that work.

git reset --hard 6b62247468340f8ec9ee61a2aee16bd5d43f038b
git push origin working -f

du -hsb .
# 8,218,212 .

Is the git database clean?

Unfortunately we aren’t free yet. The .git/objects/pack pack file is still huge (tracking the size of the big file).

git reflog expire --expire=now --all && git gc --prune=now --aggressive

Now we are back to a clean repo.

du -hsb .
# 40,916   .

RStudio with SSL proxy

Wed, 26 May 2021 12:00:00 GMT

What are we doing

We have a server hosting RStudio, and we want encrypt the traffic too and from the RStudio service. We also do not want to have to open up port 8787 on the server’s network firewall.

Why would we want this

RStudio server doesn’t come with encryption by default, you need an SSL certificate. This means all input in transit is not encrypted… this includes credentials.

How did we do this

This server already had an SSL certification configured on the Nginx HTTP server running a static site: (see Lets Encrypt).

We want to use this certificate for the RStudio server running on the same machine. We also want it to be available by a <server>/rstudio route.

This documentation provided the solution on how to setup a reverse proxy at the route <server>/rstudio (see RStudio with a proxy).

The complete configuration

The configuration file to edit is /etc/nginx/sites-available/default.conf.

The contents of this configuration file which includes the default Nginx configurations as well as both the Let’s Encrypt and the Rstudio reverse proxy.

server {
  listen 80 default_server;
  listen [::]:80 default_server;

  root /var/www/html;

  index index.html index.htm index.nginx-debian.html;
  server_name _;

  location / {
    # First attempt to serve request as file, then
    # as directory, then fall back to displaying a 404.
    try_files $uri $uri/ =404;
  }
}

server {
  root /var/www/html;

  # Add index.php to the list if you are using PHP
  index index.html index.htm index.nginx-debian.html;
  server_name <domain.name>; # managed by Certbot

  location / {
    # First attempt to serve request as file, then
    # as directory, then fall back to displaying a 404.
    try_files $uri $uri/ =404;
  }

  rewrite ^/rstudio$ $scheme://$http_host/rstudio/ permanent;

  location /rstudio/ {
    rewrite /rstudio/(.*) /$1 break;
    proxy_pass http://localhost:8787;
    proxy_redirect http://localhost:8787 $scheme://$host/rstudio/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_read_timeout 20d;
  }

  listen [::]:443 ssl ipv6only=on; # managed by Certbot
  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/<domain.name>/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/<domain.name>/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
  if ($host = <domain.name>) {
    return 301 https://$host$request_uri;
  } # managed by Certbot

  listen 80 ;
  listen [::]:80 ;
  server_name <domain.name>;
  return 404; # managed by Certbot
}

Tableau Web Data Connector Node Generator

Sat, 17 Apr 2021 12:00:00 GMT

Using Tableau Web Data Connector (WDC) as the data source for dashboard.
This post is in reference to the associated Github repo at https://github.com/mortie23/tableau-wdc-gen.

Reason:

The data we are sourcing is in SAS.
If using Tableau Public the only server sources are Google Sheets and Tableau WDC.
Tableau WDC is the easiest way to have a Tableau Public dashboard sourcing from a server that can be updated independently of the dashboard.

Background

If you are interested in the details of how this package works it is built from knowledge gained through this tutorial.
https://tableau.github.io/webdataconnector/docs/wdc_tutorial.html

Generate the data

To generate the JSON that this package uses we will use the SASjs macro core library. Directly using proc json adds a high level array element to the JSON output that we don’t want: ['SASTableData+<filename>'].
This means that we create a clean JSON output and we would be able to use different tools in the future and not have legacy SAS generated objects.

Documentation on the SASjs mp_jsonout macro can be found at: https://core.sasjs.io/mp__jsonout_8sas.html

The key steps are to create two file handles (one for the data, and one for the metadata) and use the SASjs macro to write the files out.

* Create filename handle for files to write to;
filename json "&outdir./datasourcename.json";
filename jsonmeta "&outdir./datasourcename_meta.json";
* Export to JSON;
%mp_jsonout(OBJ,datasourcename,jref=json)
%mp_jsonout(OBJ,datasourcename_meta,jref=jsonmeta)

Create the WDC

A WDC needs the following 4 files:

	Description
filename.html	WDC page
filename.js	WDC script
filename.json	The dataset as JSON array

We will used a node script to generate these files from the metadata.

Run the node script

This has been tested on Node.js 12.19.0 on for Windows 10 run using Powershell.

node .\script\tableauwdc.js --filename <filename (no extension)> --title "<a title for the dataset with quotes>" [--geojson]
## example of flat tabular data from SAS
node .\script\tableauwdc.js --filename classdata --title "Class Data"
## example GeoJSON data
node .\script\tableauwdc.js --filename geo_earthquake_4.5_week --title "Geo Earthquake 4.5 week" --geojson

You should see a quick summary of some validations and a prompt to continue.

Do not continue if your validation fails.

When you pass the validations, then continue, your WDC should be built.

Agent Based Modelling

Fri, 16 Apr 2021 12:00:00 GMT

I was experimenting with Agent based models and came across a really great talk Jackie Kazil | Agent based modelling in Python. This lead me to look at the Python module that was being discussed Mesa | Agent-based modelling in Python 3+.

General Practitioner (GP) Workforce Model

The Mesa project has many examples which I thought were a great place to get started. I decided to try to build a model that was similar to the Wolf Sheep model in this example: https://github.com/projectmesa/mesa/tree/main/examples/wolf_sheep.

So I cloned the repo and the rest of this blog post is based on my associated github repo.

I decided to use the concepts and framework for a simple workforce model, consisting of three agent types:

GP Fellows (fully qualified and practicing)
GP trainees (working towards qualification)
Patients

I made a few assumptions which were:

The GP Fellows train trainees over 5 years
The Trainees then become Fellows after 5 years.
The GP Fellow may or may not take on another trainee.

The model tests and demonstrates several Mesa concepts and features:

MultiGrid
Multiple agent types (GP Fellow, Trainee and Patient)
Overlay arbitrary text (Age and Sex of the Provider) on agent’s shapes while drawing on CanvasGrid
Agents inheriting a behaviour (random movement) from an abstract parent
Writing a model composed of multiple files.
Dynamically adding and removing agents from the schedule

Installation

To install the dependencies use pip and the requirements.txt in this directory. e.g.

pip install -r requirements.txt

How to Run

To run the model interactively, run mesa runserver in this directory. e.g.

mesa runserver

Then open your browser to http://127.0.0.1:8521/ and press Reset, then Run.

Files

This lists the key files that were modified from the base Wolf and Sheep model that this is inherited from.

level	file	description
	workforce	Workforce components
	random_walker.py	This defines the RandomWalker agent.
	agents.py	Defines the GP Fellow, Trainee, and Patient agent classes.
	model.py	Defines the Workforce model itself
	run.py	Launches a model visualization server.

Geospatial shapes for vizualisation

Fri, 12 Mar 2021 12:00:00 GMT

Putting analysis on a map

People always seem to like seeing data visualised on a map. Something about looking at a gradient of colour on a map makes people feel like they are empowered to understand.

But high-resolution shapefiles can be massive. While they are great for accurately allocating coordinates (latitude, longitude) to geospatial areas (polygons), they are terrible for rendering lower-resolution dynamic maps in dashboards.

I was looking at the Australian Remoteness Areas published by Australia’s Bureau of statistics. The raw shapefile and subsequent raw GeoJSON are way too large (~110 MB) to download over HTTP for a simple data visualisation on a webpage.

To solve this, I built a geo-processing pipeline to convert, simplify, and dissolve the boundaries at Github repository. Here is how it works.

Setup

First, let’s get our environment ready. We are using uv for fast package management. Running the following command will sync all dependencies:

uv sync

Configuration

Paths and parameters are managed in config.yaml. Update these to point to your local data before running the scripts:

paths:
  raw_shapefile: "data/to/RA_2016_AUST.shp"
  geojson_output: "data/to/RA_2016_AUST.geojson"
  simplified_geojson: "data/RA_2016_AUST-simple.geojson"
  dissolved_geojson: "data/RA_2016_AUST_all.geojson"

You can download the source shapefile directly from the ABS. Or any shape file of your choice.

The Pipeline

The process consists of three main steps.

1. Convert Shapefile to GeoJSON

First, convert the ABS shapefile format to GeoJSON:

uv run convert

2. Simplify the Boundaries

To reduce the file size, we simplify the complex shapes of map borders. Dashboard users viewing a national map at low resolution don’t need to load every single nook and cranny of the coastline.

Simplification uses the Douglas-Peucker algorithm. You can adjust the simplify_tolerance parameter in config.yaml to balance the file size vs. boundary details (default is 0.001 degrees, which is roughly 100 meters).

uv run simplify

The difference is clear. Take a look at the detail around Sydney:

Original (High Detail):

Reduced (Simplified):

This simple optimization easily drops the file size to a fraction of the original!

3. Dissolve State Boundaries

By default, the ABS remoteness boundaries are provided per state. So “Inner Regional Australia” has separate shapes for NSW, Victoria, Queensland, etc. For a national dashboard, we don’t care about the state borders. We just want a single national feature for each remoteness category.

A sample of the input properties looks like this:

{
  "properties": {
    "RA_CODE16": "11",
    "RA_NAME16": "Inner Regional Australia",
    "STE_CODE16": "1",
    "STE_NAME16": "New South Wales",
    "AREASQKM16": 87424.8418
  }
}

To remove these state boundaries, we run:

uv run dissolve

Here is what it looks like before and after dissolving the state borders:

With state boundaries:

Without state boundaries:

Visualising Geospatial Data

If you are working with GeoJSON files in VS Code, I highly recommend checking out the VSCode Geo Data Viewer extension. It makes it super easy to view the maps, filter features, and colour them.

Working Project

As mentioned, the complete code for this geo-processing tool is available in the geoshape-reduce-size repository. Feel free to check it out!

Setting up a cloud webserver for hosting a static site

Fri, 05 Feb 2021 12:00:00 GMT

Why would I want to do this?

Since there are many easier (options) for hosting a static website (of which I recommend and also use Github pages) then why would I want to go through this process. For me, I need to use the server for a few other things as well, but the main reason was just to test putting a few things together quickly to see what was involved. The TL;DR version is:

Getting a cloud linux server
Installing NGINX for the webserver
Getting a domain name and configuring the DNS
Getting an SSL certificate
Putting some static content up to test

The tools

Linode for the Cloud server
Ubuntu for the operating system
Nginx for the webserver
Fastcomet for the domain name
Let’s Encrypt for the SSL certificate
React renders the UI
MDX converts markdown into React components
Navi handles routing and creates static HTML files for each route
create-react-app builds the app’s assets and provides the dev server

1. Getting a cloud linux server

There are lots of options out there these days. I went with Linode for this one. The smallest shared CPU option will be totally fine for hosting a static site and any of the LTS Ubuntu images will be fine too.

Once you have created the machine you will need to get the machines IP address to access it.

2. Installing NGINX for the webserver

There is no point in writing the steps to install Nginx as it has already been written many times. I’ll link to the tutorial from Ubuntu. Install and configure Nginx

But before you can run the shell commands you’ll need to access your machine via SSH. If you are using Windows, I’ll give a shout out to the Windows Terminal.

Using a Powershell session in Windows Terminal we can connect to our server using:

ssh root@<ip-address>

3. Getting a domain name and configuring the DNS

Since I have other hosting and domains with Fastcomet it was easy enough for me to get another domain through them. I found a domain that cost around $10 USD a year.

Once you have purchased a domain name, you’ll need to find out how to configure the DNS. Once you’ve found this you’ll need to configure an A Record and add your machines IP address.

4. Getting an SSL certificate

The last thing I wanted to do was to get an SSL certificate. After listing to this SyntaxFM podcast, and reading up the benefits for even static sites to have SSL certificated, I was convinced that it was easy enough to do using Let’s encrypt.

Just select NGINX and Ubuntu from the Let’s encrypt page and follow the instructions.

Now you’ll have HTTPS. But you probably want a redirect for all HTTP traffic. You can add a basic NGINX configuration change to achieve this:

server {
    listen 80;
    server_name mystaticstie.net www.mystaticsite.net;
    return 301 https://mystaticstie.net$request_uri;
}

5. Putting some static content up

I previously have used Gatsby (this blog site is built using Gatsby), which is great, but I wanted to try a basic Create React App site this time. I chose to use create-react-blog a build a basic static blog.

Results

Check out the results at https://thefog.xyz/

Tableau Web Data Connector

Thu, 17 Dec 2020 12:00:00 GMT

This post is in reference to the associated Github repo at https://github.com/mortie23/tableau-wdc.

Tableau WDC

Quick example of a Tableau Web Data Connector (WDC) pointing to the output of a SAS proc json.
Reason: For example, your source data is in SAS, and your BI tool is Tableau. You want a way to publish the data to Tableau and you have access to a webserver.

Automate the generation of the WDC package

Before we start, if you have lots of datasets or need to make lots of changes to a dataset, then creating or modifying the javascript becomes tedious. I’ve built a simple Node.js package to automate this. See Tableau Web Data Connector Node Generator post if you are interested.

Tutorial

This is based on this tutorial.
https://tableau.github.io/webdataconnector/docs/wdc_tutorial.html

Generate the data

To generate the JSON I installed WPS Community edition.

libname this "./data";
proc json out="./json/class.json";
  export this.class;
run;

Deploy the HTML/JS/JSON

To get this demo going, copy the files in the Github repo to a location on your webserver. In this example I’ll be using my mortie23.github.io/

Test the WDC

Open up Tableau (in this case I’m just using Tableau Public Desktop).

Add a data connection

Browse the the published WDC HTML page

Get the data

Click Get Class Data! and you should be able to build a dashboard from the JSON data now.

Embedded BI

Wed, 04 Nov 2020 12:00:00 GMT

What is Embedded BI

When people talk about Embedded Business Intelligence (BI) they are usually referring to putting things such as interactive dashboards and/or analytics within another business application. If that other business application is web based, and is customizable, then you might be able to achieve some of this yourself.

It seems that the popularity of data journalism and organisations want to publish data is increasing. Or maybe it’s just my perspective.

A sample of options

I’ve recently explored the use of multiple different BI tools and what capability they have to embed their content in a website. I’ll run through the few I’ve looked into, specifically the ones which you can use for free.

Tableau

Tableau was my first choice as it is a product I had used before. It has a free tier and is nice enough to allow you to publish content and embed it on other sites. You can sign up for an account here https://public.tableau.com/s/

You will need to download the free Tableau Public Desktop application to develop you first dashboard and publish it to your Tableau public profile. Once you do, the Tableau public dashboard will have a share icon which will provide you with the embed code. It might look something like this:

<!-- Start Tableau Embed -->
<section class="brands-area">
  <div class="container">
    <div class="tableauPlaceholder" id="vizSOMEID" style="position: relative">
      <noscript>
        <a href="#">
          <img
            alt=" "
            src="https://public.tableau.com/static/images/nh/YOURDASHBOARDNAME/Dashboard1/1_rss.png"
            style="border: none"
          />
        </a>
      </noscript>
      <object class="tableauViz" style="display:none;">
        <param name="host_url" value="https%3A%2F%2Fpublic.tableau.com%2F" />
        <param name="embed_code_version" value="3" />
        <param name="site_root" value="" />
        <param name="name" value="YOURDASHBOARDNAME/Dashboard1" />
        <param name="tabs" value="no" />
        <param name="toolbar" value="yes" />
        <param
          name="static_image"
          value="https://public.tableau.com/static/images/nh/YOURDASHBOARDNAME/Dashboard1/1.png"
        />
        <param name="animate_transition" value="yes" />
        <param name="display_static_image" value="yes" />
        <param name="display_spinner" value="yes" />
        <param name="display_overlay" value="yes" />
        <param name="display_count" value="yes" />
        <param name="language" value="en" />
      </object>
    </div>
  </div>
</section>
<!-- End Tableau Embed -->

You also get a script tag that you need to include in your site as well.

Or you can directly embed with an iframe:

<div class="container">
  <iframe
    src="https://public.tableau.com/views/YOURDASHBOARDNAME/YOURDASHBOARDNAME?:showVizHome=no&:embed=true"
    width="1000"
    height="625"
  ></iframe>
</div>

Power BI

Power BI was the other option that I thought would be a good one to demo. It has a free tier, but it isn’t made as easily accessible. You can sign up for an account here https://powerbi.microsoft.com/en-us/get-started/, but you’ll need a corporate work email address (Gmail, Outlook etc won’t cut it).

You’ll also need to download the Power BI desktop application, I’d recommend getting this from the Microsoft Store (if you’re using a Windows computer).

Use Power BI desktop application to develop a quick dashboard and publish it to the Power BI service (that is https://powerbi.com).

There are a few administration hoops to jump through, but once you do, you can get the embed code from your dashboard in the Power BI service by doing the following:

It may look something like this:

<!-- Start Power BI Embed -->
<section class="brands-area">
  <div class="container">
    <iframe
      width="1000"
      height="625"
      src="https://app.powerbi.com/view?r=YOURAPPID"
      frameborder="0"
      allowfullscreen="true"
    >
    </iframe>
  </div>
</section>
<!-- End Power BI Embed-->

Other products that get a mention

Yellowfin
Qlik Sense
SAS Visual Analytics
Looker

I should also mention, that there are also a lot of open source and paid javascript charting and graphing frameworks that could be used to achieve the same thing.

Wrapping it up, it’s never been a better time to embed charts, dashboards and/or analytics into your business application or site.

Speed Up My Site

Sun, 20 Sep 2020 12:00:00 GMT

Originally I built my personal site by using a free responsive template and then I just modified the source code. The page loads were slow and I didn’t really learn much in the process.

I got interested in static site generators as they seemed to meet what I needed for most of my use cases. I’d heard about GatsbyJS from SyntaxFM and then watched a great conference talk from Kyle Mathews (Talk) about the framework. So I decided to give it a go.

Template

The starter templates were a great place to start. I chose the Hero Blog by Greg Lobinski and started modifying it.

Quick and Fun

I found the experience great and learnt a lot while having heaps of fun. The final site is much faster. I recommend it.

Linux Users Groups

Wed, 02 Sep 2020 12:00:00 GMT

This post is in reference to the associated Github repo at https://github.com/mortie23/linux-users-groups.

Deploy a security model (users, groups and shared directory structure) for Linix system for enterprise data

This package was designed to run on Ubuntu 20.04. This package has some dependencies including ACLs and JQ which will be outlined in the setup.sh.

Description

This code repository will create everything required to implement the security model from start to finish.

The structure of the security model is based on a two level organisation structure. The organisation has agencies which comprise of business units. This is represented by:

Agency NFL
- Business unit 49ers
- Business unit patriots
Agency NBA
- Business unit Bulls
- Business unit Warriors

Each agency has a shared directory that each business unit can read and write to for files that are shared between business units. Then each business unit has a directory that only they can read and write from.

Getting Setup

Firstly clone the repository to your Linux machine.

Follow the manual steps in setup.sh. The administrator account and group is nixadmin. If you choose to change this you will need to update the ACL files that use this user and group as well.

The example has the root directory for setting up the shared directory structure in /u01/data. This is a parameter that can be changed before deployment by editing param-files/global.json.

{
  "dataRoot": "/u01/data"
}

Once you have run the manual steps in setup.sh you need to make the following scrips executable.

sudo chmod +x users-groups.sh
sudo chmod +x dir-agency.sh
sudo chmod +x dir-businessunit.sh

Firstly run this script to create the users and groups. Modify this to suit your organisation, and also update the corresponding ACL files. If you have users and groups from an identity provider such as Active Directory then you will skip this step.

## Create the users and groups
sudo ./users-groups.sh

Usage

Run order of the scripts there after is following:

## Only run this command once for each agency (example agency NFL)
sudo ./dir-agency.sh -p ./param-files/global.json -l a -a nfl
## Only run this command once for each business unit (example business unit Patriots)
sudo ./dir-businessunit.sh -p ./param-files/global.json -l b -a nfl -b patriots

Success

The directories have been deployed as per the security model:

The example user MichaelJordan in {agency: nba, businessunit: bulls} can change into directories allowed and write files where allowed.