What is Microsoft Azure Cognitive Search?
Azure Cognitive Search is a Search-as-a-Service cloud solution in Microsoft Azure. This service provides tools and APIs for developers to add rich search experiences to their application.
The Azure Cognitive Search Service can build indexing on the Data Source so that it can be queried from application code using end user’s inputs. As defined in the official documentation, in Azure Cognitive Search, an index is a persistent store of documents and other constructs used for filtered and full text search.
Azure Cognitive Search Service uses AI enrichment capability on indexing, which can be used to extract text from images, blobs and other unstructured data sources, and make the search richer. The text extraction is implemented through cognitive skills that is attached to an indexing. The cognitive skills use Natural Language Processing (NLP) and Image Processing.
NLP includes language detection, key phrase extraction, text manipulation, etc. The Image processing skills use Optical Character Recognition (OCR). Using AI and NLP is beyond the scope for this Azure Cognitive Search Service tutorial, but those interested can read more about it from this link.
Azure Cognitive Search Service functionality is provided using .NET SDK and REST API. Since the Azure Cognitive Search Service runs in the cloud, we need not worry about the availability and infrastructure, as its managed by Microsoft.
Figure 1 gives an idea of the Azure Cognitive Search Service in the application.
Figure 1: Application development approach using Azure Cognitive Search Service
Inspiration for this Azure Cognitive Service REST APIs tutorial
An organization where I am working for as an Azure consultant, had a requirement to integrate enterprise search features into their existing application. They wanted to provide a high-responsive search experience to the end-users.
The requirement was to consolidate data across various data stores on Microsoft Azure and provide search functionality on it based on end-user’s input. The UX requirement expectation was a highly responsive UI with fast search. After doing a rigorous research and adopting complex querying mechanisms, as well as NoSQL solutions, we decided to use Azure Cognitive Search service. We soon found that it had the capability to address most of my client’s requirements. We tried Azure Cognitive Search Service on various complex data structures and also on a high volume of data – with success!
Having delved into a relatively new piece of technology (I am no search expert) and getting a good experience of Azure Cognitive Search Service, I persuaded Suprotim to let me write on it for this month’s magazine edition.
Scenarios of using Microsoft Azure Cognitive Search Service
– While handling search operations on a high volume of data from different data sources (e.g. Azure SQL Database, SQL Server on Azure VMs, Cosmos DB, Azure BLOB Storage, Azure Table Storage, etc.), writing complex join queries increases complexity of the operations.
In such cases, you can consolidate the data from various data sources in JSON documents formats and use Azure Cognitive Search Service Indexers to load the data in index to ease search operations. For example, Let’s assume you want to search ‘Order details’ across countries, regions, cities, customers, categories, etc, then consolidating data into a single store and searching this store using Azure Cognitive Search Service, will be helpful.
– If search is using Filters, AutoComplete, Suggestions, etc., then Azure Cognitive Search Service is a good option.
– Indexing unstructured text or extracting text from image. Please note that we need to use Azure Cognitive Search AI features over here.
Azure Cognitive Search Service provides the following features for search
- Free-form text search
- Full-Text Search
- Simple Query Syntax
- Lucene Query Syntax
- Relevance
- Geo-Search
- Filters and facets
- Faceted Navigation
- Filters
- User Experience Features
- Autocomplete
- Search Suggestions
- Sorting
- Paging
- Hit Highlighting
- Synonyms
In this tutorial, we will implement the Azure Cognitive Search Feature using REST API along with Simple Query syntax and Lucene Query syntax. We will implement the application using the following steps:
1. Creating the Data Source using Azure CosmosDB
2. Creating Azure Cognitive Search Service
3. Creating Angular Client Application to access Azure Cognitive Search Service REST API
Creating the Data Source using Azure CosmosDB
As we have discussed in Figure 1, we need a Data Source for consolidating the data to perform search. I have used the simple Northwind database and the Orders Table in it. But since some of the columns from this table are reference columns, I have read data from the Order’s table using the following query:
Select OrderID, Customers.ContactName as CustomerName, Employees.FirstName + ' ' + Employees.LastName as EmployeeName,
OrderDate, RequiredDate, ShippedDate, Shippers.CompanyName as ShipperName,
Freight, ShipName, ShipAddress, ShipCity,ShipPostalCode, ShipCountry
from Orders, Customers, Employees, Shippers
where
Customers.CustomerID=Orders.CustomerID and Employees.EmployeeID =Orders.EmployeeID and Shippers.ShipperID=Orders.ShipVia
Create a Cosmos DB Account. If you are new to CosmosDB, follow this link to read steps for creating Cosmos DB account.
Editorial Note: For those who want to delve deeper into CosmosDB, I recommend reading Azure Cosmos DB – Deep Dive by Tim Sommer.
Now create an OrdersDb database and two containers named as AllOrders and SuppliersData. To add data in Cosmos DB Containers, download Data Migration tool for CosmosDB.
If you are not aware about this tool, you can visit this tutorial to read more about it, and how to use it. Once you migrate data from Orders table to the AllOrders container, the JSON document data will be displayed in the AllOrders container. We will also read data from SuppliersData table (38000 rows) and migrate to the SuppliersData container. The query for reading data from the SuppliersData is as follows:
select SupplierID, CompanyName,ContactName,ContactTitle,Address,City,Country,Phone from SuppliersData
Creating Azure Cognitive Search Service
Let’s create an Azure Cognitive Search Service. Open the Azure portal using https://portal.azure.com. From the Azure portal, using Create a Resource we can create Azure Cognitive Search Service as shown in the following figure:
Figure 2: Creating a new Azure Cognitive Search Service
To create a service, provide the following information as shown in Figure 3.
Figure 3: New Search Service
One important point to note is the Pricing tier. Azure Cognitive Search Service configuration is completely based on the pricing tSipliire. We can see the configuration details by clicking on the Change Pricing Tier link as shown in Figure 4.
Figure 4: The Pricing tier showing the Azure Service Configure details
There are offerings for configuring Azure Cognitive Search Service. Service Indexes, Indexers count and Storage size are defined based on the offering.
Once the service is created, its details will be shown as seen in Figure 5.
Figure 5: The Azure Service details
We need keys for accessing Search Service in the client application. These keys will be used to authorize the client application to access Azure Service. We can see these keys as shown in Figure 6.
Figure 6: Azure Cognitive Search Service Keys
To configure the data source for the search service, click on the import data link as shown in Figure 5. The Connect to your data tab option will be displayed, where you can select the data source as shown in Figure 7.
Figure 7: Connect to data source
Select CosmosDB from the dropdown. The Connect to your data tab will bring up a UI where we can enter data source information like Cosmos DB Account Key, Database name and Container name from the Cosmos DB as shown in Figure 8.
Figure 8: Data Source Configuration
Leave the Query TextBox as empty, we will be selecting all the columns from the source collection. Click on the Next button, here we can attach Cognitive Service with the Azure Cognitive Search Service. This is an optional step and we will keep it as-is without any changes. See Figure 9.
Figure 9: Azure Cognitive Service Configuration
Click on the Skip to: Customized target index button.
Settings index configuration is the most important step for the Azure Cognitive Search Service. The Index is like a database table. This holds data from the source so that queries on the data can be accepted. We can configure the index by setting following configuration values:
1. Index name – is the unique index name. The index name can contain lower case characters, alphanumeric characters, digits.
2. Key – the index can have only one key field. This must be a string. This key represents the unique identifier for each document for every document stored in the index.
3. Suggester name – is required for Auto-Complete suggestions.
4. Search mode – is used for strategy used to search
5. We can select fields from the index so that we can perform following types of queries on it
1. Retrievable
2. Filterable
3. Sortable
4. Facetable
5. Searchable
6. Suggester
Set values for the Index as shown in Figure 10:
Figure 10: Index Configuration
Note that although we are making all fields as Retrievable, Filterable, Sortable, Facetable, Search can only be possible on string fields. These will be used for text-based search.
The Analyzer column provides a drop down for the query analysis syntax. Here we will select English -Microsoft. We are doing this for plain text-based search. There are several other Analyzer values which can be set as per your business needs.
Click on the Next: Create an Indexer button to create an index as shown in Figure 11.
Figure 11: Create an Indexer
Here we are keeping the defaults as-is. The indexer will streamline and automate data indexes for all data source connection, data read and serialization operations.
Indexers are available for Azure CosmosDB, Azure SQL Database, Azure BLOB Storage and SQL Server on Azure VM. Click on the Submit button to complete the service configuration.
Once the Azure Cognitive Search Service is configured, we can view the details as shown in Figure 12.
Figure 12: The Azure Service Configuration success details
As shown in Figure 12, we can see all the indexer, indexes, storage, data source, etc. for the Azure Cognitive Search service.
Click on the Indexes tab and then click on the Refresh tab, the page will display the fetched documents from the data source. See Figure 13.
Figure 13: The Indexes with data loaded
It shows all the 830 documents (in this case) received from the data source. Now click on the Indexers tab and see the status. The status shows the Indexers has run successfully to load the data as seen in Figure 14.
Figure 14: Indexer success
The other important details of the service e.g. URL will be displayed as shown in Figure 15.
Figure 15: Azure Cognitive Search Service other details
We can query using the Search Explorer link. This link will navigate to the search explorer as seen in Figure 16.
Figure 16: Search Explorer
Figure 16 shows the Index name, the API Version, Query String and Request URL. After clicking the Search button, by default, 50 records will be returned. The Request URL will contain the Query string values entered into the Query String box. In the Query String box enter ‘Roland Mendel’ and click on the Search button to get all records with name Roland Mendel.
Azure Cognitive Search Service Limits
While creating Azure Cognitive Search Service, be aware of the maximum limits for storage, number of indexes, workloads, documents etc. These limits are helpful to provide an access of Azure Cognitive Search Service to the client application. These limits vary based on the Provisioning of Azure Cognitive Search E.g. Free, Basic, Standard, etc. More information about these maximum limits can be read from this link.
As seen in Figure 4, we have selected the Standard Provisioning for the Search Service. This means that we can create a total of 50 Indexes. To create a new container in Cosmos DB, follow all the steps discussed in the Create Data Source Sections.
From the Northwind Database and its SuppliersData table, execute the following query”
select SupplierID, CompanyName,ContactName,ContactTitle,Address,City,Country,Phone from SuppliersData
As already explained in the steps for creating a data source, migrate the data from this table to the new Cosmos DB container. Import this data in the Azure Cognitive Search Service and create a new Index for reading the Suppliers Data. So now we have two indexes in Azure Cognitive Search Service.
Creating Angular Client Application to access Azure Cognitive Search Service REST API
Azure Cognitive Service provides a feature for generating a React.js application with HTML views. This can be done by clicking on “Indexes” from Search Dashboard and then clicking on “Create Search App (Preview)) as shown in Figure 17.
Figure 17: Sample Preview app
Since the Create Search Sapp is in preview, in this step, we will create a new Angular client application and use Azure Cognitive Search REST API to perform search operations.
We will implement a Simple Query Search in the client application. We require the Search Service Admin keys as explained in Figure 6. The admin key is used to authorize the client application against the Search Service.
Create the Angular Application using angular CLI and name this application as azuresearch-client.
Step 1: In the app sub-folder of the src folder, add a new file and name this file as servicedetails.ts. Add the following code to it:
export class ServerDetails {
// 1. the Service name
public static searchServiceName = 'allorderssearch';
// 2. The Admin Key
public static searchServiceAdminApiKey = '<MY ADMIN KEY>';
// 3. The Index Name
public static searchIndexName = 'cosmosdb-index-allordersdata';
public static searchIndexNameLucene = 'cosmosdb-index-suppliersdata';
// 4. The API Version
public static apiVersion = '2019-05-06';
// 5. The Search URLS
// tslint:disable-next-line: max-line-length
public static searchUri = `https://${ServerDetails.searchServiceName}.search.windows.net/indexes/${ServerDetails.searchIndexName}/docs/search?api-version=${ServerDetails.apiVersion}`;
// tslint:disable-next-line: max-line-length
public static searchUriLucene = `https://${ServerDetails.searchServiceName}.search.windows.net/indexes/${ServerDetails.searchIndexNameLucene}/docs/search?api-version=${ServerDetails.apiVersion}`;
}
Listing 1: The Azure Cognitive Search Service Details
You can see Azure Cognitive Search service details in the code. The important thing here is the URIs, as these URIs are used for performing query-based search operations.
Step 2: Add the following two code files in the app folder. These files will be used for Orders class and SuppliersData classes
export class Orders {
constructor(public OrderID: string,
public CustomerName: string,
public EmployeeName: string,
public OrderDate: string,
public RequiredDate: string,
public ShippedDate: string,
public ShipperName: string,
public Freight: number,
public ShipName: string,
public ShipAddress: string,
public ShipCity: string,
public ShipPostalCode: string,
public ShipCountry: string,
public id: string) {
}
}
Listing 2: The Orders class
export class SuppliersData {
constructor(
public SupplierID: string,
public CompanyName: string,
public ContactName: string,
public ContactTitle: string,
public Address: string,
public City: string,
public Country: string,
public Phone: string,
public id: string
) {
}
}
Listing 3: The SuppliersData class
Step 3: In the app folder, add a new file and name it as azuresearchservice.ts. This file contains an Angular service. This service will use the HttpClient class to make a call to Azure Cognitive Search Service to perform Search operations. Add the following code in the file:
import { Injectable } from '@angular/core';
import { HttpClient, HttpHeaders } from '@angular/common/http';
import { Observable } from 'rxjs';
import { ServerDetails } from './serverdetails';
@Injectable({
providedIn: 'root'
})
export class AzureSerchService {
constructor(private http: HttpClient) {
}
searchData(query: string, pageSize: number): Observable {
let result: Observable = null;
const options = {
headers: new HttpHeaders({
'api-key': ServerDetails.searchServiceAdminApiKey,
'Content-Type': 'application/json'
})
};
result = this.http.post(ServerDetails.searchUri, JSON.stringify({
search: query,
top: pageSize,
}), options);
R1eturn result;
}
searchSuppliersLuceneData(query: string): Observable {
let result: Observable = null;
const options = {
headers: new HttpHeaders({
'api-key': ServerDetails.searchServiceAdminApiKey,
'Content-Type': 'application/json'
})
};
result = this.http.post(ServerDetails.searchUriLucene, JSON.stringify({
search: query
}), options);
return result;
}
}
Listing 4: The Service class
In the Service class, we have searchData() and searchSuppliersLuceneData() methods. These methods will be used to perform simple search and Lucene Text search respectively.
In the searchData() method, we are making a POST request by passing api-key and Content-Type in header. The api-key will authenticate the call against the Azure Cognitive Search Service. We are sending the search and top parameters in the request body. The search parameter will post the query for search operations and the top parameter will define how many results can we expect in response. The default response records are 50.
Similarly in the searchSuppliersLuceneData() method we are passing search query to perform search operations. You can read more about Simple Query Syntax from this link and the Lucene Query Syntax from this link.
Step 4: In the app folder, add a new file and name it as app.english.search.component.ts. Add the following code in it:
import { Component, OnInit } from '@angular/core';
import { AzureSerchService } from './azuresearchservice';
import { Orders } from './orders';
@Component({
selector: 'app-search-component',
templateUrl: './app.component.view.html'
})
export class AppEnglishSearchComponent implements OnInit {
query: string;
orders: Array;
headers: Array;
private order: Orders;
recordCount: number;
pageSizeArray: Array;
pageSize: number;
constructor(private serv: AzureSerchService) {
this.query = '';
this.orders = new Array();
this.headers = new Array();
this.recordCount = 0;
// the page size array
this.pageSizeArray = [100, 200, 300, 400, 500, 600, 700, 800, 900, 1000];
// the defaule page size
this.pageSize = 100;
this.order = new Orders('', '', '', '', '', '', '', 0, '', '', '', '', '','');
}
ngOnInit(): void {
// tslint:disable-next-line: forin
for (const p in this.order) {
this.headers.push(p);
}
this.onChangeQuery();
}
// method for search query
onChangeQuery(): void {
this.serv.searchData(this.query, this.pageSize).subscribe(resp => {
const response: any = resp.value;
this.getOrders(response);
});
}
// receive the search data
private getOrders(data: []): void {
this.orders = new Array();
for (const ord of data) {
this.orders.push(ord);
}
this.recordCount = this.orders.length;
}
}
Listing 5: The Simple Query Component class
The Simple Query Component class defines page size array so that the end-user can select the page size to define expected records in the response against the search query. The method onChangeQuery() will call the searchData() method from the Angular service by passing the search query and the page size parameters to it. The getOrders() method will update the orders array by pushing the search records in it.
Step 5: In the app folder, add a new html file and name it as app.component.view.html with following markup in it:
<div>
<h2>The Standard Search Using 'English- Microsoft'</h2>
<div class="form-group">
<label>Enter Search Value</label>
<input type="text" class="form-control" placeholder="Enter your search here" (keyup)="onChangeQuery()" [(ngModel)]="query">
<label>Select Page Size</label>
<select class="form-control" (change)=" onChangeQuery()" [(ngModel)]="pageSize">
<option *ngFor="let size of pageSizeArray" value="{{size}}">{{size}}</option>
</select>
</div>
<hr>
<label>Total Match Found: {{recordCount}}</label>
<hr>
<h2>Orders Details</h2>
<table class="table table-bordered table-striped">
<thead>
<tr>
<td *ngFor="let h of headers">{{h}}</td>
</tr>
</thead>
<tbody>
<tr *ngFor="let ord of orders">
<td *ngFor="let h of headers">{{ord[h]}}</td>
</tr>
</tbody>
</table>
</div>
Listing 6: The Html markup for the English search
The html contains the required angular binding for invoking onChangeQuery() method on keyup event of an input element. apgeSizeArray is bound with select element to display page size in the option element of the select element. The table element will display received orders after search service returns a response.
Step 6: Similar to Simple Query Search, the following code shows the logic for performing Lucene Query Search in the app.suppliers.component.lucene.ts file in the app folder:
import { Component, OnInit } from '@angular/core';
import { AzureSerchService } from './azuresearchservice';
import { SuppliersData } from './suppliersdata';
@Component({
selector: 'app-suppliers-search-lucene-component',
templateUrl: './app.suppliers.component.lucene.view.html'
})
export class AppSuppliersSearchLuceneComponent implements OnInit {
query: string;
suppliers: Array;
headers: Array;
private supplier: SuppliersData;
recordCount: number;
constructor(private serv: AzureSerchService) {
this.query = '';
this.suppliers = new Array();
this.headers = new Array();
this.recordCount = 0;
this.supplier = new SuppliersData('', '', '', '', '', '', '', '', '');
}
ngOnInit(): void {
// tslint:disable-next-line: forin
for (const p in this.supplier) {
this.headers.push(p);
}
this.onChangeQuery();
}
onChangeQuery(): void {
this.serv.searchSuppliersLuceneData(this.query).subscribe(resp => {
const response: any = resp.value;
this.getSuppliers(response);
});
}
private getSuppliers(data: []): void {
this.suppliers = new Array();
for (const sup of data) {
this.suppliers.push(sup);
}
this.recordCount = this.suppliers.length;
}
}
Listing 7: app.suppliers.component.lucene.ts for accessing the the searchSuppliersLuceneData method from Angular Service
We will also have the app.suppliers.component.lucene.view.html file for Search UI for Lucene Search as shown in Listing 8.
<h2>The Standard Search Using 'English- Lucene'</h2>
<div class="form-row">
Note that you can query for Fields as FieldName:Value <br/> You can using AND/OR Conditions e.g FieldName1:Value1 AND FieldName2:Value2
</div>
<div class="form-group">
<label>Enter Search Value</label>
<input type="text" class="form-control" placeholder="Enter your search here" (keyup)="onChangeQuery()" [(ngModel)]="query">
</div>
<hr>
<label>Total Match Found: {{recordCount}}</label>
<hr>
<h2>Suppliers Details</h2>
<table class="table table-bordered table-striped">
<thead>
<tr>
<td *ngFor="let h of headers">{{h}}</td>
</tr>
</thead>
<tbody>
<tr *ngFor="let sup of suppliers">
<td *ngFor="let h of headers">{{sup[h]}}</td>
</tr>
</tbody>
</table>
Listing 8: The app.suppliers.component.lucene.view.html file
Listing 8 contains binding with the properties and methods from the AppSuppliersSearchLuceneComponent for performing search operations based on the Lucene Query Syntax.
Step 7: In the app folder, add a new file and name it as app.main.component.ts. This file will contain routing configuration across components added in Step 5 and 6. Add the following code in the file:
import { Component, OnInit } from '@angular/core';
@Component({
selector: 'app-main-component',
template: `
<table class="table table-bordered table-striped">
<tr>
<td>
<a [routerLink]="['']">English Search</a>
</td>
<td>
<a [routerLink]="['lucene']">Lucene Search</a>
</td>
</tr>
</table>
<br/>
<router-outlet></router-outlet>
`
})
export class MainComponent implements OnInit {
constructor() { }
ngOnInit(): void { }
}
Listing 9: The MainComponent with the Routing
Step 8: Modify app.module.ts file in the project with the following code. This code defines routing for the components we added in Step 5 and Step 6.
import { BrowserModule } from '@angular/platform-browser';
import { NgModule, Component } from '@angular/core';
import { AppEnglishSearchComponent } from './app.english.search.component';
import { HttpClientModule } from '@angular/common/http';
import { FormsModule } from '@angular/forms';
import { AppSuppliersSearchLuceneComponent } from './app.suppliers.component.lucene';
import { RouterModule } from '@angular/router';
import { MainComponent } from './app.main.component';
@NgModule({
declarations: [
AppEnglishSearchComponent,
AppSuppliersSearchLuceneComponent,
MainComponent
],
imports: [
BrowserModule, HttpClientModule, FormsModule,
RouterModule.forRoot([
{ path: '', component: AppEnglishSearchComponent },
{ path: 'lucene', component: AppSuppliersSearchLuceneComponent }
])
],
providers: [],
bootstrap: [MainComponent]
})
export class AppModule { }
Listing 10: The AppModule
Run the application from the Command Prompt using npm run start command – this will run the Angular Application. Open the browser and enter http://localhost:4200 url in the address bar. This will render a view in the browser as shown in the Figure 18.
This will load the first 100 records because the record size is 100. In the Enter Search Value, enter the value as Hari, it will immediately perform search operations. The search result will be shown as in the following figure:
Figure 19: Simple Search based on query as Hari
We can use AND Operator for search using + operator e.g. enter “Margaret Peacock” + “Sierras de Granada 9993” in the search textbox (without the double quotes), and the search results will be displayed accordingly.
On the page, click on the Lucene Search. This page will show the Suppliers data with the first 50 records. In the Lucene Query Syntax, we can use Boolean operators, Regular Expressions, WildCards, Fuzze search, etc. For example, in the Enter Search Value text box, enter Charlotte Cooper || London. A the search will take place on these values and result will be displayed as shown in Figure 20.
Figure 20: The Lucene Query syntax
In the Search Text box, enter Admin*. A search occurs for the word starting with Admin and it will search Admin, Administrator, etc. This makes it easy to perform search.
Conclusion
Microsoft Azure Cognitive Search is a search-as-a-service solution with optimized and high-performance search capabilities. It allows developers to incorporate great search experiences into applications that needs enterprise search across various data sources.
Download the entire source code of this article on Github
This article was technically reviewed by Vikram Pendse
This article has been editorially reviewed by Suprotim Agarwal.
C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.
We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).
Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.
Click here to Explore the Table of Contents or Download Sample Chapters!
Was this article worth reading? Share it with fellow developers too. Thanks!
Mahesh Sabnis is a DotNetCurry author and a Microsoft MVP having over two decades of experience in IT education and development. He is a Microsoft Certified Trainer (MCT) since 2005 and has conducted various Corporate Training programs for .NET Technologies (all versions), and Front-end technologies like Angular and React. Follow him on twitter @
maheshdotnet or connect with him on
LinkedIn