written by
Ashish Dubey

How to build a custom model using AI builder for Object Detection

Power Platform 4 min read , August 27, 2021

In this article, I would like to show you how to build an AI Object Detector model using Power Platform, and then use it in a PowerApp.

What is Object Detection?

As per Microsoft docs,

Object detection can help expedite or automate business processes. In retail, it can help streamline inventory management, allowing retail leaders to focus on onsite customer relationships. In manufacturing, technicians can use it to speed the repair process by quickly accessing the manual for a piece of machinery for which the UPC/serial number isn't obvious.

Essentially, Object detection refers to the capability of computer and software systems to detect images, locate objects in an image and identify each object. Object detection can be used to expedite or automate business processes.

Image Guidelines

The details around image quality and quantity are available on Microsoft Docs. However, following are some of the highlight points:

Format

  • JPG
  • PNG
  • BMP

Size

  • 6 MB maximum for training
  • minimum width / height of 256 pixels x 256 pixels

Model output

The most interesting form processing model output from an input document or image is as follows:

Results

A list of values extracted from the input image by AI the model. These include values like TagName, TagId and Confidence.

GroupedResults

A list of values extracted from the input image by AI the model. These include values like TagName, TagId and ObjectCount.

What are we going to build?

We will build a custom model for object detection. We will then add this custom object detection component into a new canvas PowerApp. Here is the result of our trained and tested model in PowerApp for a given test image:

Building the custom Object Detection Model

Please follow the steps below to first create and train your custom object detection model:

1) Sign in to PowerApps and in the left navigation pane, select AI Builder > Build > Object Detection.

2) Give a name to your model and click Create.

3) Select domain – Here, you can see different options for selecting your model’s domain. Presently, there are 3 domains available for selection:

  • Brand Logos - This model is optimized for identifying corporate logos.
  • Objects on retail shelves - This model is typically used for taking a physical inventory.
  • Common Objects – This model is for common objects.

Select Common objects and click on Next button.

4) Choose Objects – Select the object names by selecting an entity from Common Data Service or add them manually. Create 3 new objects as shown below and click Next button.

5) Select Images - Click on “Add Images” button to upload images of objects required for detection and identification. Upload 15+ images of each object.

6) Add images – Upload the sample images from your local machine

7) Upload images – Click on “Upload Images” button once all images have been uploaded.

8) Click “Close” button when all images have been uploaded.

9) Click on “Next” button to continue.

10) Tag objects in Images – In this step, we teach the model to identify objects in images by tagging and labelling them. Click on the first image to open it. In order to tag an object in image, hover over the image, near an object that needs to be tagged. A dotted-lined box should appear around the objects. This means that it has been detected as a single item that can be tagged. If the predefined selector is not accurate, drag the container to draw it to accurately tag the objects. Repeat this step for all the images.

11) Click on” Done tagging” once all the images have been tagged.

12) Click on” Next” button to continue.

13) Train Model – Click on “Train” button to train the model. This step may take several minutes.

14) Quick Test – Once your model is trained, it is always a good idea to test the model’s accuracy before Publishing the model for consumption it in PowerApp or Power Automate.

15) Click on the information icon next to the label “Performance”. As you can see, Performance is a measure of model’s effectiveness, or in other words, the accuracy of the model in detecting object. In our case, this value is 94, which indicates we have a good model.

16) Click on “Quick Test” button and select an image for testing purpose.

17) Look at my test results in the screen shot below. In my case, the image I tested with the model shows me a confidence level of 98% that the object in the image is Green Tea Cinnamon.

18) Here are the results for another quick test of the model with a different image. Check the confidence of the model in detecting objects.

19) Once we are satisfied with testing that our model is good at detecting our objects, click on the “Publish” button to publish the model.

Congratulations…! You have successfully built your Object Detection AI model.

Object Detection PowerApp

Download the sample Power App from Modern Work OnDemand site import the attached PowerApp into your environment. Edit the PowerApp in designer to explore the controls and code associated with this simple one screen app.

Conclusion

In this article we saw how to use Object Detection AI model of AI builder to process our images and detect the required objects from them. We can consume the resultsn, for example Microsoft Power Automate, to further fulfil our business requirements.

While the possibilities are endless, I hope this article encouraged you to start exploring AI builder capabilities. This solution can be easily extended using Power Platform and other Office 365 services, and I would like to hear your thoughts on this.

Hope you liked this article! Please share your valuable comments and feedback with me on LinkedIn.

Stay well and happy PowerApping...!

PowerApps AI Builder Power Platform