Simple Intelligent Bot using Microsoft Bot Framework & Cognitive Services

One of the main reasons I love programming is you continue learning new and exciting things. Your job is never monotonous if you are doing what you love! Big software companies like Microsoft, Google, Facebook and many more are all trying to put Innovation at the forefront and working on releasing interesting products, frameworks and APIs.

This article is published from the DNC Magazine for Developers and Architects. Download this magazine from here [Zip PDF] or Subscribe to this magazine for FREE and download all previous and current editions

As a Microsoft developer, I find the Build conference to be one of the most exciting places to learn about these innovations. This year’s Build held in March 2016 was even better, with announcements about conversational intelligence that clubs natural human language, with advanced machine intelligence. New frameworks like the Microsoft Bot Framework and Skype Bot Platform were announced and enhancements to the Cortana Intelligence Suite powered with Microsoft Cognitive Services were made which exposes Intelligence APIs that allow systems to see, hear, speak, understand and interpret our needs with natural communication.

In this article we will take a look at two of my favourites, Microsoft Bot Framework and Microsoft Cognitive Services.

Microsoft Bot Framework

As a SciFi fan, Robots and Artificial Intelligence has always fascinated me. Bots are nothing but software applications which run automated tasks. They can accept commands and perform tasks which are structured.

Microsoft Bot Framework lets you easily build Bots and connect them to various channels like Skype, Slack, Office 365 Email and many more. If you have an existing Bot, you can also connect them to various channels. This means you don’t have to rewrite or build new Bots for every channel (Skype, Slack, Twitter etc) if you want to target a wider audience. You can concentrate on building better bots and let Bot Framework take care of connecting to other 3^rd party services. Here is a screenshot of one of my Bot admin page.

As you can see, my bot, Knowledge Guru, is connected to Skype and Web Chat. This means my Bot has a face and any one can interact with it. Any Skype User can send message to this bot and I can also embed this Bot in any website. You can ask Knowledge Guru about any Academic topic and retrieve papers published by various Authors and Affiliations within seconds. The chat control you see in this page is provided by the Bot Framework. This is very useful and saves us some time.

Note: This is just a starting point. There are bugs in the app and I will be eliminating them and making the conversation smarter as I build on it.

Bots are not new and Microsoft is not the first player in the new Bot race! Many enthusiasts including me have created automated services and have used them in our day to day life. In 2008, I created my first Bot, RemindMeAbout, a Twitter Bot which accepts commands as tweets and adds reminders to my calendar. This bot was simple and accepted only SPECIFIC commands. It was not intelligent to understand spelling mistakes or different sentences. This is where Microsoft Cognitive services comes into play.

Microsoft Cognitive services

Microsoft Cognitive services lets you add intelligence to your Apps. They are a set of services which can be used to add different intelligent capabilities like Language Understanding, Vision, Speech etc. with very little code. This means you can interact with your bot with natural language and talk to it like a human.

I encourage everyone to visit https://www.microsoft.com/cognitive-services/ and read through all the services. Microsoft has provided a good explanation and some helpful videos to understand each service.

If you are aware of Azure Machine Learning Services or Cortana Analytics Suit, then this service from Microsoft is not entirely new to you. It was possible to setup a Machine Learning Experiment, build a Model, run Experiments, publish it as a Service and use it in your app. But this was always a little challenging for me as I was not a Data Scientist and Machine Learning is not exactly my area of expertise. Cognitive Services makes this process easy for an everyday developer to concentrate more on building better apps with intelligent services and not to worry about setting it up.

I will demonstrate this with a simple fun Bot and let's call it Emoji Bot. Make sure you read the documentation at https://dev.botframework.com/ and https://www.microsoft.com/cognitive-services/en-us/emotion-api - Emotion API of Microsoft Cognitive Services. Emoji bot will accept any photo as a message and return emojis based on the emotion detected by Emotion API. Here is a screenshot

Make sure your development environment is ready for Bot Framework development. We will be using the Free Visual Studio 2015 Community Edition. Visit Bot Framework getting started page, http://docs.botframework.com/connector/getstarted/#getting-started-in-net, and download and install the necessary project template as detailed in the “Getting started in .NET” section. Make sure all Visual Studio extensions are updated before starting the project.

1. Create a new Bot Application using Visual Studio 2015

2. Replace the auto generated code for Post method in MessageController.cs with the following code

if (message.Type == "Message")
{

    //Get attachment
    if (message.Attachments.Count > 0)
    {
        string imageUri = message.Attachments[0].ContentUrl;
        return message.CreateReplyMessage(await Utilities.CheckEmotion(imageUri));
    }
    else
    {
        return message.CreateReplyMessage("Send me a photo!");
    }

}
else
{
    return HandleSystemMessage(message);
}

In the code, we check the number of attachments. Images sent in a message are retrievable through Attachments property. We then use this image to detect faces and their respective emotions in step 3.

3. Add the following code to MessageController.cs file:

public static class Utilities
{
    public static async Task CheckEmotion(string query)     {          var client = new HttpClient();         var queryString = HttpUtility.ParseQueryString(string.Empty);         string responseMessage = string.Empty;         // Request headers         client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "YOUR SUBSCRIPTION KEY");          // Request parameters         var uri = "https://api.projectoxford.ai/emotion/v1.0/recognize";         HttpResponseMessage response = null;         byte[] byteData = Encoding.UTF8.GetBytes("{ 'url': '" + query + "' }");         using (var content = new ByteArrayContent(byteData))         {             content.Headers.ContentType = new MediaTypeHeaderValue("application/json");             response = await client.PostAsync(uri, content).ConfigureAwait(false);         }          string responseString = await response.Content.ReadAsStringAsync();          EmotionResult[] faces = JsonConvert.DeserializeObject

In this code, we pass the image to Emotion API to detect faces. If faces are found, then their emotional scores are sorted. We can then use this value to reply with an appropriate emoji. Make sure you get your subscription key by visiting https://www.microsoft.com/cognitive-services/en-US/subscriptions

4.  Add the following classes to MessageController.cs

public class Scores
{
    public double anger { get; set; }
    public double contempt { get; set; }
    public double disgust { get; set; }
    public double fear { get; set; }
    public double happiness { get; set; }
    public double neutral { get; set; }
    public double sadness { get; set; }
    public double surprise { get; set; }
}

public class EmotionResult
{
    public Scores scores { get; set; }
}

Make sure you also update your AppId and Secret in app.config after creating a bot in https://dev.botframework.com/#/bots .

Once the above steps are completed, Enable Web channel and test your bot by adding an iframe to the default.html file.

https://webchat.botframework.com/embed/emojibot?s=YOUR%20SECRET%20HERE

This article has been editorially reviewed by Suprotim Agarwal.

C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.

We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).

Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.

Click here to Explore the Table of Contents or Download Sample Chapters!

Author

ShobanKumar is an ex-Microsoft MVP in SharePoint who currently works as a SharePoint Consultant. You can read more about his projects at http://shobankumar.com. You can also follow him on twitter @shobankr