Installing Prefect Server on Azure
Reading
Firstly I came across the following docs on how to install prefect server
Prefect Server 101 Deploying to GCP
Deploying to a Single Node Prefect Server
None of these outlined fully in detail what I needed, which was how to install from scratch on Azure.
The Azure VM
We are going to set ourselves up with a Linux VM with 4 GB of RAM and 60 GB of attached disk. By default, we will not be allowing any inbound ports, because we will be opening them with firewall rules later.
The machine will be based on the basic Ubuntu 20.04 image from Canonical.
The machine we will choose is the Standard_B2s machine which will have the resources that we need.
Set the public inbound ports to none.
Create an attached disk that is Standard SSD and 64 GB in size (E6 performance tier).
For the rest of the image, set to defaults for Networking, Management, Advanced, Tags.
Key file
Firstly we are going to move the keyfile from where ever your browser downloaded it to your .ssh
directory in your home directory.
I am going to use Powershell on Windows for this example (but you could just as easily use any other shell or OS).
ls | Where-Object {$_.Name -eq 'prefect_key.pem'}
Directory: C:\Users\<username>\.ssh
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 24/11/2021 9:35 AM 2498 prefect_key.pem
Now we can go to our resource and connect.
Networking
To enable yourself to connect (using Secure Shell to connect to your remote computer) you must create a networking rule to allow your public IP address to access inbound port 22.
The easiest way to find your public IP address is to ask Google.
We are going to add three inbound port rule so that we can:
- Connect to the server (Port 22)
- Enable access to the UI (Port 8080)
- Enable the UI to access the GraphQL API (Port 4200)
For now we are going to only enable these ports to be open for us.
This shows how to enable TCP through port 22 for only our public IP address.
We need to repeat the steps for ports 8080 and 4200.
Connecting
ssh -i C:\Users\<username>\.ssh\prefect_key.pem azureuser@<azure-vm-public-ip>
You’ll be prompted that the authenticity of host can’t be established and asked if you want to continue.
Respond yes
and the host will be saved to your know_hosts
file so you will not have to be prompted again.
The authenticity of host '<azure-vm-public-ip> (<azure-vm-public-ip>)' can't be established.
ECDSA key fingerprint is SHA256:abcdefghij/12345678901234567+g/abcdefghijkl.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<azure-vm-public-ip>' (ECDSA) to the list of known hosts.
azueruser@<azure-vm-public-ip>: Permission denied (publickey).
Installing
Following the steps
This is where I am now going to basically follow the steps in Prefect Server 101 Deploying to GCP. However, a few modifications have been made as some commands seemed to fail on my attempt.
Installing Python (and Pip)
Update your package information:
sudo apt update
Check Python3 is installed:
It was already installed on my Azure Standard_B2s machine.
python3 --version
Install Pip:
sudo apt install python3-pip
Install the Python 3 virtual environment manager:
sudo apt install python3-venv
Installing Docker
sudo apt install docker.io
Add your user to the docker group
sudo usermod -aG docker ${USER}
After this you will need to disconnect your SSH session and reconnect to pickup the new group membership.
# From remote
exit
# From local
ssh -i C:\Users\<username>\.ssh\prefect_key.pem azureuser@<azure-vm-public-ip>
Get the Docker Compose repository:
sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
Apply executable permissions:
sudo chmod +x /usr/local/bin/docker-compose
Installing Prefect
Now let’s install Prefect. We are going to do this from a virtual Python environment. Firstly, create and activate your virtual environment.
python3 -m venv prefect
source prefect/bin/activate
pip install prefect
Starting Prefect Server
Before we start the server infrastructure, we need to modify the UI endpoint so it knows how to access the GraphQL server correctly (note that this requires Prefect version 0.10.3+). We can do this by creating a user config.toml
and adding the endpoint of our VM:
mkdir ~/.prefect
touch ~/.prefect/config.toml
Add the following to the file we just created (replacing the text in bold with your own IP). Note, I used vim to edit the file.
[server]
[server.ui]
apollo_url = "http://<azure-vm-public-ip>:4200/graphql"
Start the prefect server!
prefect server start --expose
Once the output looks like the following, go to this URL in your browser http://azure-vm-public-ip:8080.
WELCOME TO
_____ _____ ______ ______ ______ _____ _______ _____ ______ _______ ________ _____
| __ \| __ \| ____| ____| ____/ ____|__ __| / ____| ____| __ \ \ / / ____| __ \
| |__) | |__) | |__ | |__ | |__ | | | | | (___ | |__ | |__) \ \ / /| |__ | |__) |
| ___/| _ /| __| | __| | __|| | | | \___ \| __| | _ / \ \/ / | __| | _ /
| | | | \ \| |____| | | |___| |____ | | ____) | |____| | \ \ \ / | |____| | \ \
|_| |_| \_\______|_| |______\_____| |_| |_____/|______|_| \_\ \/ |______|_| \_\
Visit http://localhost:8080 to get started
For the remaining documentation please refer to Prefect Server 101 Deploying to GCP.