Integrating Azure Cognitive with Retool

What is Retool

Retool is a simple tool for creating a web application. One of the things that makes Retool great is you’re never limited by what’s available out of the box. If you can write it with JavaScript (the most popular language for web developer) and an API.

What is Azure Cognitive Services

AzureCognitive Services makes it easy to integrate AI with your web applications. All it takes is an API call to embed the ability to do cool things.

What you need to get started

A Retool subscription and an Azure subscription

First step is to sign up for the Free trial of Retool (if you don’t already have a Retool subscription) and the Free trial of Azure (if you don’t already have an Azure subscription).

Creating the App

We are going to demo how to integrate the computer vision Azure Cognitive service. So we need a basic Retool application that loads up an image from your local PC. We also need some input boxes to put the response of the API call.

We need to give the input components the correct names so the the Javascript query we are going to write later will be able to access them.

• tagsInput
• captionInput
• confidenceInput

// relevant js code to above input boxes
tagsInput.setValue();
captionInput.setValue();
confidenceInput.setValue();

Before we start to create the Javascript that will call the API, we need to include some libraries from Microsoft that will assist in the process. Retool has our back here, and has a nice way to load libraries.

We are going to use the browser libraries from the following NPM package @azure/cognitiveservices-computervision. I simply NPM installed the package and then sourced the minified libraries from \node_modules\@azure\cognitiveservices-computervision\dist and then hosted themselves on my own webserver. You need to have them in this order because the second is dependant on the first.

If you haven’t enables CORS on your webserver you will need to do this so that the libraries can be called by Retool. If you are using NGINX that would be something like adding the following to your NGINX config add_header 'Access-Control-Allow-Origin' '*' always;

We need to create a Javascript query which will call the Azure Cognitive services API when we load the image.

// our secrets (for simplicity hard coded, I do not recommend this approach)
const computerVisionKey = '<key>';
const computerVisionEndPoint = '<endpoint>';

// this is needed because the cognitive API needs binary data not base64
function _base64ToArrayBuffer(base64) {
  var binary_string = window.atob(base64);
  var len = binary_string.length;
  var bytes = new Uint8Array(len);
  for (var i = 0; i < len; i++) {
    bytes[i] = binary_string.charCodeAt(i);
  }
  return bytes.buffer;
}

const cognitiveServiceCredentials = new msRest.ApiKeyCredentials({
  inHeader: {
    'Ocp-Apim-Subscription-Key': computerVisionKey,
  },
});
const client = new Azure.CognitiveservicesComputervision.ComputerVisionClient(
  cognitiveServiceCredentials,
  computerVisionEndPoint,
);

// get the base64 image provided by browse to file from retool
const imageBase64 = cognativeImageButton.value[0];

const options = {
  maxCandidates: 5,
  language: 'en',
};

// make the call and write results to input box
client
  .describeImageInStream(_base64ToArrayBuffer(imageBase64), options)
  .then(result => {
    tagsInput.setValue(result.tags.join());
    captionInput.setValue(result.captions[0].text);
    confidenceInput.setValue(result.captions[0].confidence);
  })
  .catch(err => {
    console.log('An error occurred:');
    console.error(err);
  });

Alright, this missing piece to all this now is the Azure cognitive services.

Getting the Azure services

Firstly, after you sign up for your Azure subscription, you will need to create a Computer Visions resource.

Now that the endpoint has been created, lets get a key and the enpoint and paste them into the JS query in Retool.

// our secrets (for simplicity hard coded, I do not recommend this approach)
const computerVisionKey = '<key>';
const computerVisionEndPoint = '<endpoint>';

Testing it

Uploading this close up picture of an Eagle, and the computer vision API returns with a confidence of around 50% that it is a close up of an eagle. Not too bad.