Microsoft Cognitive Services with UWP

Microsoft Cognitive Services with UWP

UWPCognitive1

Short introduction

Did you hear about Microsoft Cognitive Services? Have you ever had an idea to detect emotions or recognize language using your app? Now you can!

Microsoft Cognitive Services are intelligent APIs that enables you to add just a few lines of code to your app to detect which emotions are visible in the picture or check the spelling. Let’s see how it works with Universal Windows 10 Apps! We will try to detect and mark face on the image.

The whole source code is available on my Git Hub account.

What do I need to start?

1) Visual Studio 2015 Community (for free) or higher

 

Let’s start

1) Open Microsoft Cognitive Services website

2) Click “My account” in the right corner and sign in with the Microsoft account:

UWPCognitive2

3) Select “Face – Preview” from the list:

UWPCognitive3

4) Accept the terms and click “Subscribe”:

UWPCognitive4

5) Once you subscribe you should see below information about available subscriptions:

UWPCognitive5

We will use Key 1 in this tutorial so copy it to the safe place.

 

Let’ s create some UWP app to test Cognitive Services!

 

1) Open Visual Studio and create new Universal Windows App project:

UWPCognitive6

2) We need to add some NuGet packages to make it work. Paste below in the Package Manager Console:

Install-Package Microsoft.ProjectOxford.Face

You should see the information about success:

UWPCognitive8

One more nuget:

Install-Package Newtonsoft.Json

The last one to enable drawing rectangles on images:

Install-Package Win2D.uwp

We can start using API!

UWPCognitive9_1

 

3) Add below code to the “MainPage.xaml.cs” file to enable connections with the API:

Paste it just under class declaration:

private readonly IFaceServiceClient faceServiceClient = new FaceServiceClient("<<your API Key here>>");

UWPCognitive10_1

4) Firstly we nedd to upload image to detect faces:

Before we do it we have to add some image to the “Assets” folder.
I added myself but of course feel free to change image.

UWPCognitive11

5) Now xaml code for “MainPage.xaml” should look like below:

 

<Page x:Class="UwpCognitiveServices.MainPage" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:local="using:UwpCognitiveServices" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:customCanvas="using:Microsoft.Graphics.Canvas.UI.Xaml" mc:Ignorable="d">

 <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
 <Image Source="Assets/Me.jpg" Stretch="None"/>
 <customCanvas:CanvasControl Draw="canvasControl_Draw" x:Name="CustomCanvas" />
 </Grid>
</Page>

 

Please note that we are using Win2d library to draw rectangle on canvas.
Now in the code behind we need to add some code to handle face recognition:

1) Firstly we need to upload image to the Api to recognize where the face is. To achieve that use below method:

async void UploadAndDetectFaces(string imageFilePath)
    {
     try
      {
         StorageFolder appInstalledFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;
         StorageFolder assets = await appInstalledFolder.GetFolderAsync("Assets");
         var storageFile = await assets.GetFileAsync("Me.jpg");
         var randomAccessStream = await storageFile.OpenReadAsync();

      using (Stream stream = randomAccessStream.AsStreamForRead())
       {
         //this is the fragment where face is recognized:
         var faces = await faceServiceClient.DetectAsync(stream);
         var faceRects = faces.Select(face => face.FaceRectangle);
         _faceRectangles = faceRects.ToArray();
         CustomCanvas.Invalidate();
       }
     }
     catch (Exception ex)
      {
        System.Diagnostics.Debug.WriteLine(ex.Message);
      }
 }

Once the face is recognized and we have information about the location of it we are able to draw rectangle to select it in the picture:

 void canvasControl_Draw(CanvasControl sender, CanvasDrawEventArgs args)
   {
     if (_faceRectangles != null)
      if (_faceRectangles.Length > 0)
       {
         foreach (var faceRect in _faceRectangles)
         {
          args.DrawingSession.DrawRectangle(faceRect.Left, faceRect.Top, faceRect.Width, faceRect.Height, Colors.Red);
         }
       }
 }

Whole code of “MainPage.xaml.cs” file should look like below:

  public sealed partial class MainPage : Page
 {
   private readonly IFaceServiceClient faceServiceClient = new FaceServiceClient("<<Your API Key here>>");
   FaceRectangle[] _faceRectangles;
   public MainPage()
    {
      this.InitializeComponent();
      UploadAndDetectFaces("ms-appx:///Assets/StoreLogo.png");
     }

   async void UploadAndDetectFaces(string imageFilePath)
    {
      try
       {
        StorageFolder appInstalledFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;
        StorageFolder assets = await appInstalledFolder.GetFolderAsync("Assets");
        var storageFile = await assets.GetFileAsync("Me.jpg");
        var randomAccessStream = await storageFile.OpenReadAsync();

        using (Stream stream = randomAccessStream.AsStreamForRead())
         {
          var faces = await faceServiceClient.DetectAsync(stream);
          var faceRects = faces.Select(face => face.FaceRectangle);
          _faceRectangles = faceRects.ToArray();
          CustomCanvas.Invalidate();
         }
       }
      catch (Exception ex)
       {
         System.Diagnostics.Debug.WriteLine(ex.Message);
       }
   }

   void canvasControl_Draw(CanvasControl sender, CanvasDrawEventArgs args)
    {
      if (_faceRectangles != null)
       if (_faceRectangles.Length > 0)
        {
          foreach (var faceRect in _faceRectangles)
           {
             args.DrawingSession.DrawRectangle(faceRect.Left, faceRect.Top, faceRect.Width, faceRect.Height, Colors.Red);
           }
        }
    }
 }

6) Run the app and check the result:

UWPCognitive12

Sum up

In this post I wanted to show how to use Microsoft Cognitive Services with Universal Windows 10 Apps. Please remember that this is only one example. There are more APIs to use like:

Computer Vision API

Emotion API

Speaker Recognition API

Text Analytics API

Go, register and try some of them because it is worth a try!

Advertisements