This is a simple demonstration using GoogleCloud Functions and Flows to do image recognition on an image sent on Telegram.

Flows and Beyond

As a developer of Flows I often think who our users are, why they use Flows, and what they need to accomplish their goals; and then, which features we need to implement to best serve those users.

Flows is a powerful drag-and-drop automation engine for creating communication flows. We initially conceived it as a no-code solution, but we found many users could get really powerful behavior writing some code for specific use-cases.   These bits of code can be inside Flows, or they can be 3rd party cloud functions like AWS Lambda functions or Google Cloud Functions. 

An interesting use case: Image Recognition 

For a short and funny example, I will show you how to implement an app that recognizes hot dogs. We will set up a flow in Flows, which will receive images from users and decide, whether they sent a hotdog or not.

For many of our customers, this type of image recognition can be very powerful. Imagine you run a delivery service and you wanted to verify successful deliveries automatically. Similar to what I’m going to show, you could use location data, photos of parcels, and even recipient signatures to create a verification flow in Flows. 

A plan for success

First, we will set up a cloud function, which receives a request with a URL to an image, then it uses an image recognition API to process the image,  and responds whether there is a hotdog in the image or not.

Then we will build a flow, which receives a message from a user via a messaging channel (Telegram in this case), executes the cloud function above, and responds to the user whether there is a hotdog in a picture he sent.

Setting up the Google Cloud Function

First,  we will need to set up a cloud function. To get started quickly, create a cloud function using this tutorial: https://cloud.google.com/functions/docs/quickstart-console. As a ‘Trigger’ choose HTTP trigger, execution environment: Node.js 10, and in the source code field insert the code snippet. It’s simple code, which checks whether the request contains JSON code and answers yes or no. 

Next, you’ll need to deploy this function. To test it inside Google Cloud Platform, follow steps from the tutorial. 

‍

To test from your browser, go to the following URL inserting the specific address for your function:

https://your-function-address.cloudfunctions.net/HotDogOrNot/?url=hello should return {“isHotDog”: true} and the address https://your-function-address.cloudfunctions.net/HotDogOrNot should return {“isHotDog”: false}.

Nice job! You set up a google cloud function. Now we need to make our cloud function smarter.

Setting up the Google Vision API

To make it smarter let’s add image recognition. For this purpose we will use the Google Vision API. To get started, follow steps 1-4 of this tutorial: https://cloud.google.com/vision/docs/quickstart-client-libraries. In the tutorial you’ll activate the Vision API and create a service account to use it.

Now return to the cloud function you created. Toggle “Environment variables, networking, timeouts and more” and in file “Service account” choose the VisionAPI service account you just created. Now we can access the Vision API inside our function.

Now let’s change the code. On a “Package.json” tab, insert this code. It will add Google Vision API library as a dependency to your function, and on "Index.js" tab update existing code with the following code snippet.

What’s the difference compared to the previous version? We added a request to VisionAPI, which returns the ‘labels’ it found on the image.  Then we filter these labels by description: if it contains “hot dog” and if it has greater than 60% confidence in that label. If there is at least 1 label remaining after filtering, that means we found a hotdog on the image.

To understand how Google VisionAPI works and how the response looks like check their documentation, https://cloud.google.com/vision/docs

After that, deploy the new version of our function. To test it from your browser, find any image of a hotdog and save it's URL. Now go to URL of your function (inserting the correct address for your function) https://your-function-address.cloudfunctions.net/HotDogOrNot?url=url_to_image and replace the  “url_to_image” with a URL to the found image. If there is a hotdog in the image, the page will return {“isHotDog”: true}.

Now let’s connect this function to Flow Builder.

Creating a flow in Flows

Log in into the Bird Dashboard or sign up for an account if you don’t have one.

If you are new to Flows and you don’t have any channels set up, you’ll need to go to the Channel setup page, and choose to set up the Telegram channel. I chose Telegram for this demo because it’s easy and fast to set up.

Now you have a channel we can use in Flows. Go to the Flows page, create a new custom flow, and choose the “Telegram” channel trigger.

You’ll be redirected to a flow page, where you should choose your Telegram channel as trigger, in our case it’s “Hotdog”. Please add 2 steps: “Fetch variables” and “Reply to channel message”.

Inside the “Fetch variables” step we will call our cloud function and retrieve response into variable “isHotDog” which will contain “true” or “false” as a response from the GoogleClound function. In URL field please insert URL to your function https://your-function-address.cloudfunctions.net/HotDogOrNot and fill all other fields as on the "Fetch variable step content" picture.

And inside the “Reply to channel message” step we will respond to the customer with a message containing the yes or no response. For that insert in "Reply with message" field the following text "Hotdog on the image? {{isHotDog}}".

‍

If you have any trouble building the flow, you can export it from file: simple-hotdog-flow.json.

To test it, send an image to your Telegram bot.

So far, it looks cool! We created a small chat bot, which checks images customers sent. To make it prettier, let’s add some more steps as shown below: 

If you have any trouble building the flow, you can export it from file: advanced-hotdog-flow.json.

Results

While this is a fun example, we believe this type of functionality can be very useful for our users. 

If you want more features like this built-in in Flows, write to our support team to let us know.