System Administration,  Cloud Infrastructure,  Data Engineering

Installing Prefect Server on Azure

Installing Prefect Server on Azure


Firstly I came across the following docs on how to install prefect server

Prefect Server 101 Deploying to GCP
Deploying to a Single Node Prefect Server

None of these outlined fully in detail what I needed, which was how to install from scratch on Azure.

The Azure VM

We are going to set ourselves up with a Linux VM with 4 GB of RAM and 60 GB of attached disk. By default, we will not be allowing any inbound ports, because we will be opening them with firewall rules later.

The machine will be based on the basic Ubuntu 20.04 image from Canonical. Azure Ubuntu 20.04 VM

The machine we will choose is the Standard_B2s machine which will have the resources that we need. Azure Machine

Set the public inbound ports to none. Azure No Ports

Create an attached disk that is Standard SSD and 64 GB in size (E6 performance tier). Azure Disk

For the rest of the image, set to defaults for Networking, Management, Advanced, Tags. Azure Disk

Azure Disk

Key file

Azure Disk

Firstly we are going to move the keyfile from where ever your browser downloaded it to your .ssh directory in your home directory. I am going to use Powershell on Windows for this example (but you could just as easily use any other shell or OS).

ls | Where-Object {$_.Name -eq 'prefect_key.pem'}
        Directory: C:\Users\<username>\.ssh

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a---        24/11/2021   9:35 AM           2498   prefect_key.pem

Now we can go to our resource and connect. Azure Disk


To enable yourself to connect (using Secure Shell to connect to your remote computer) you must create a networking rule to allow your public IP address to access inbound port 22.

The easiest way to find your public IP address is to ask Google.

Azure Disk

We are going to add three inbound port rule so that we can:

  1. Connect to the server (Port 22)
  2. Enable access to the UI (Port 8080)
  3. Enable the UI to access the GraphQL API (Port 4200)

Azure Disk

For now we are going to only enable these ports to be open for us.

This shows how to enable TCP through port 22 for only our public IP address.

Azure Disk

We need to repeat the steps for ports 8080 and 4200.


ssh -i C:\Users\<username>\.ssh\prefect_key.pem azureuser@<azure-vm-public-ip>

You’ll be prompted that the authenticity of host can’t be established and asked if you want to continue.
Respond yes and the host will be saved to your know_hosts file so you will not have to be prompted again.

The authenticity of host '<azure-vm-public-ip> (<azure-vm-public-ip>)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghij/12345678901234567+g/abcdefghijkl.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<azure-vm-public-ip>' (ECDSA) to the list of known hosts.
azueruser@<azure-vm-public-ip>: Permission denied (publickey).


Following the steps

This is where I am now going to basically follow the steps in Prefect Server 101 Deploying to GCP. However, a few modifications have been made as some commands seemed to fail on my attempt.

Installing Python (and Pip)

Update your package information:

sudo apt update

Check Python3 is installed:

It was already installed on my Azure Standard_B2s machine.

python3 --version

Install Pip:

sudo apt install python3-pip

Install the Python 3 virtual environment manager:

sudo apt install python3-venv

Installing Docker

sudo apt install

Add your user to the docker group

sudo usermod -aG docker ${USER}

After this you will need to disconnect your SSH session and reconnect to pickup the new group membership.

# From remote
# From local
ssh -i C:\Users\<username>\.ssh\prefect_key.pem azureuser@<azure-vm-public-ip>

Get the Docker Compose repository:

sudo curl -L "$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Apply executable permissions:

sudo chmod +x /usr/local/bin/docker-compose

Installing Prefect

Now let’s install Prefect. We are going to do this from a virtual Python environment. Firstly, create and activate your virtual environment.

python3 -m venv prefect
source prefect/bin/activate
pip install prefect

Starting Prefect Server

Before we start the server infrastructure, we need to modify the UI endpoint so it knows how to access the GraphQL server correctly (note that this requires Prefect version 0.10.3+). We can do this by creating a user config.toml and adding the endpoint of our VM:

mkdir ~/.prefect
touch ~/.prefect/config.toml

Add the following to the file we just created (replacing the text in bold with your own IP). Note, I used vim to edit the file.

    apollo_url = "http://<azure-vm-public-ip>:4200/graphql"

Start the prefect server!

prefect server start --expose

Once the output looks like the following, go to this URL in your browser http://azure-vm-public-ip:8080.


   _____  _____  ______ ______ ______ _____ _______    _____ ______ _______      ________ _____
  |  __ \|  __ \|  ____|  ____|  ____/ ____|__   __|  / ____|  ____|  __ \ \    / /  ____|  __ \
  | |__) | |__) | |__  | |__  | |__ | |       | |    | (___ | |__  | |__) \ \  / /| |__  | |__) |
  |  ___/|  _  /|  __| |  __| |  __|| |       | |     \___ \|  __| |  _  / \ \/ / |  __| |  _  /
  | |    | | \ \| |____| |    | |___| |____   | |     ____) | |____| | \ \  \  /  | |____| | \ \
  |_|    |_|  \_\______|_|    |______\_____|  |_|    |_____/|______|_|  \_\  \/   |______|_|  \_\

   Visit http://localhost:8080 to get started

prefect 01 started

For the remaining documentation please refer to Prefect Server 101 Deploying to GCP.