BUSINESS IMPACT

May 29, 2019

Knowledge Mining Showcase: Azure Search

Kevin Jackson Posted by Kevin Jackson

Welcome to the first installment of an in-depth look at Knowledge Mining –  the ability to use Microsoft Azure’s advanced AI Search capabilities to comb through all of your data (PDFs, emails, scanned documents, images, etc.) to glean insight. In this series, I’m going to take you through Microsoft’s Azure Cognitive Search (Azure Search with human-like reasoning capabilities) and show you the ins and outs of effectively using this awesome tool to uncover insight from enterprise data, whether structured or raw.

Let’s begin by digging in to Azure Search.

knowledge-mining

Azure Search

Managed by Microsoft, the intelligent Azure Search cloud solution-as-a-service has built-in cognitive abilities. Recognizing and extracting text and identity from images; highlighting key talking points from text; and the power to recognize and classify people, places, and things from text and images, are among its innovative capabilities.

This expansive offering can help dig deep into your organization’s data, often uncovering rich insight. BlueGranite is currently working on one such project for a private global energy exploration and engineering company. Leaders here needed a way look back at decades of data, recorded in paper files, without a team of archivists. We’re working with the worldwide energy giant, using cloud-scale technology, to digitize, store, and offer deep search capabilities on more than six million documents. By combining Azure Search with Knowledge Mining, we will ultimately provide nearly instantaneous access to information that once would have taken a team of people potentially months (if ever) to uncover.

So, what is Knowledge Mining?

Knowledge Mining

Knowledge Mining is a cognitive search-based technique of extracting facts from unstructured data. It’s like having a crew of experts comb through your most important documents to discover and leverage data to drive your enterprise. This content comprehension capability can be used to create in-depth search resources that inform an organization’s employees and enrich its clients and customers.

Knowledge_Mining_Solution

In today’s tutorial, I’ll offer step-by-step instructions on creating a search service and an index.

Creating an Azure Search Service and Index

The first step in creating an effective search is providing the data that we are going to search. Azure Search can be used against several data sources, both structured (Azure SQL Database and Cosmos DB) and unstructured, in the form of Azure Blob Storage. Blob indexers can extract text from major file formats such as Microsoft Office, PDF, and HTML documents.

In this how-to series on Knowledge Mining, we’ll be focusing on unstructured data, so let’s create some blob storage.

First, we need to create a storage account to hold our blobs. From the Azure Portal, select the Create a resource option and then type storage account into the search box. Select Storage account from the drop-down under the search box.

Azure_Marketplace 

Click on the Create button.

 Storage_Account

And now add the details:

  1. Select your Subscription and Resource group that you are using for your search.
  2. Enter a Storage account name. Note that this name must be unique across all storage account names.
  3. Select a Location that you will use for creating all of your resources throughout this exercise.
  4. Make the following selections:
    1. Performance: Standard
    2. Account kind: Storage (general purpose v1)
    3. Replication: Locally-redundant storage (LRS)
  5. Click on the Next: Advanced> button.
Create_Storage_Account_Basics

On the next page, make the selections shown below and then click on the Review + create button.

 Create_Storage_Account_Advanced

Now press the Create button and wait for the deployment of your new storage account to complete.

 Create_Storage_Account_Review

When your deployment is complete, you will see a link to your new storage account at the bottom of the page. Click on it to move to the next step.

 Deployment_Complete

You’ll now see the Storage Account Overview page. We will create our searchable blob storage in this storage account by clicking on the Blobs link in the middle of the page.

Storage_Account_Overview

Click on the big plus sign + Container in the upper left of your screen and enter a name for your container. Think of a container as a folder that will hold all of our documents.

Select Container (anonymous read access for containers and blobs) for your Public access level and click on the OK button.

 New_Container

Click on the name of your new container.

New_Container_Name

You should now see the Overview of your blob container. Now we need to add the files that we are going to be searching. We do this by clicking on the Upload option in the upper left of the screen.

 Blob_Overview

You’ll see this on the right side of your screen. For this series, we’ll be working with a common data set used in Microsoft’s Knowledge Mining Bootcamp, which is an excellent introduction to Azure Search as well. You can go to the Bootcamp here. To get the dataset, go to the GitHub repository at https://github.com/Azure/LearnAI-KnowledgeMiningBootcamp.git and clone the repository. You’ll find the sample data in the resources/dataset/ folder.

 Select_Dataset_Files

From the Azure Portal, click on the selection button and select all of the files in the dataset folder.

 Upload_Blob

Now we are ready to create a search service and index all of the documents we just uploaded.

Create a new Azure Search Service by clicking on the Create a resource menu item in Azure Portal and then type Azure Search into the search box. Select Azure Search from the drop-down under the search box.

New_Azure_Search_Service 

Click on the Create button.

Azure_Search_Create 

Enter a name for your search service, the subscription and resource group you want your searching to be done in, and the location. You will want these to all match the setting on the storage account that you created earlier.  You can use the Free pricing tier for these exercises but understand that it is very limited and should only be used for dev/test.

Click the Create button and wait for your new Search Service to be created.

New_Search_Service

Now that the search service is created, you should see something similar to this:

Dashboard_Overview

Next, we’ll create the index that will hold all of the information for our search. Click on the Import data link on the top of the page. On the next screen, select Azure Blob Storage from the Data Source drop-down list.

Import_Data

Name your data source and then click on Choose an existing connection and select the storage account that you created earlier. You will then select the container that you created and press the Select button. Leave all other fields at their default values and press the Next: Add cognitive search (Optional) button on the bottom of the screen. Azure Search will try to infer index fields from the files in your storage account. Since we are working with unstructured data, it will only come back with standard search fields.

Connect_Data

We’re going to skip the cognitive search settings for now (much more on this in future posts). Press the Skip to: Customize target index button.

Add_Cognitive_Search

We’ll leave all of the settings at their default value for now. Press on the Next: Create an indexer button.

Customize_Target_Index

Change the Schedule value to Once. You can leave all of the other fields at their default value and press on the Submit button at the bottom of the screen.

Create_Indexer

When the index creation is complete, it will take you back to the Search Service Overview page. Notice that the Index, Indexers, and Data sources menu items in the middle of the page all have a (1) next to them showing the number of elements in each area.

Congratulations! You’ve just built your first searchable index.

Click on the Indexers(1) link in the middle of the page. You’ll see that the status is set to Warning. Let’s look into that by clicking on the line for your indexer.

Indexers

Click on the line in the Execution details section for the indexer we just ran. In the new blade, you’ll see several entries in the Warnings section. There are two kinds of warnings: 1) Document has unsupported content type, and 2) Truncated extracted text to 32768 characters. The first warning is because several of our files contain images. The standard Indexer does index images. We have to add a cognitive service for image cracking. We’ll do this in our next session. The second warning is because we selected the Free pricing tier when we created our search service. The Free pricing tier only allows a maximum of 32,768 characters to be extracted out of a document.

Indexer_Summary

Go back to the Search Service Overview page and click on the Indexes(1) link and then again on the line with the index that you just created. From the Index Screen we can query a selected index and test it out. Try it out. Type “Microsoft” in the Query String field and click Search. You’ll see the results returned in JSON. Using the dataset from the Knowledge Mining Bootcamp, you’ll see 10 documents returned.

Search_Explorer

More to Come

In the coming weeks I’ll be exploring the many ways to use Azure Cognitive Search to more easily unearth knowledge from once-difficult-to-mine data sources. Be sure to subscribe to our blog so that you don't miss a tutorial, or contact us today to discover the many ways BlueGranite can make the most of your data!

step-by-step guide to getting started with AI
Kevin Jackson

About The Author

Kevin Jackson

Kevin Jackson has worked in software development for almost 30 years, specializing in architecture and technical leadership. He has been part of numerous successful projects working with teams that varied in size from just a few people to hundreds working across different time zones and continents. He enjoys working closely with clients to design and deliver world-class systems on time and on budget.

Latest Posts

AI Strategy for Business