Monitoring an ASP.NET Web Application with Automated Actions using Azure Monitor

Operations is an integral part of DevOps.

One of the main tasks of Operations is to monitor the health of the application and take corrective actions if it is not up to the mark. These activities are usually done by a set of people in the organization called the Operations group.

As per the guidelines of DevOps, the responsibility of “operations” should no longer be on an external group (like the Operations group), but should be a part of the responsibilities given to the DevOps team.

To make Operations more reliable, we can automate the monitoring of activities and appropriate actions. The automation of operations may involve the following:

• monitor health of the application and the environment in which it is hosted.
• If any activity crosses a threshold (set for a condition of performance or quality), raise an alert. These conditions can be availability of application or quality of the application or performance of the hardware that is supporting it.
• Once such an alert is raised, an automated action should take place to bring the “out of bound” parameter back to normal.

Case Study – Monitoring an ASP.NET Web Application using Azure Monitor

We are going to study an application hosted on IIS on a VM in Azure. It is an ASP.NET web application that is already deployed and working. It has certain flaws that we need to find out and take appropriate actions until a bug-fix is found for those. We are going to make use of certain services in Azure to achieve the desired results.

The Azure services that we will use are:

a) Azure Monitor

• Log Analytics
• Application Insights
• Alerts

b) Azure Automation

c) Azure Storage

In this case study, we are going to do the following:

1. Setup Azure Monitor – Log Analytics to observe the performance counters of a VM in Azure. This VM is hosting the application that we are monitoring.

2. Setup Azure Application Insights, which is a part of Azure Monitor, to observe availability of application.

3. Setup Azure Automation and a VM in Azure for running automation scripts.

4. Create query in Azure Monitor – Log Analytics to get the records where performance of hardware (e.g. %Processor Time) for the VM hosting our application, is below expected.

5. Create Azure automation runbooks (scripts) in Azure Automation which are hooked to alerts in Azure Monitor. These scripts will take the corrective action on the VM when a hooked alert is sent.

6. Create alerts in Azure Monitor to notify runbook so that appropriate actions are taken.

Figure: Case of integration of Azure Monitor with Azure Automation

Azure Monitor

Azure Monitor is a service offered by Azure to monitor and report various parameters related to health of the resources in Azure and on-premises. Set of services offered by Azure Monitor include Application Insights, Log Analytics, Alerts and predefined Metrics in a graphical form.

Azure Monitor can target various resources in Azure like Application Services, Virtual Machines, Storage Accounts, Containers, Networks and Cosmos DB.

Figure: Azure monitor overview

Major components of Azure Monitor are:

Note: I have already written in details about Application Insights and alerts in my earlier article.

Let us now study Log Analytics in this article.

Log Analytics

Log analytics is a feature of Azure Monitor to query and display filtered results of various logs collected by Azure Monitor service. Application Insights data too can be queried with Log Analytics.

In this example, we will setup and query the performance counters of a virtual machine that is part of IaaS service of Azure. On this VM, I have installed an ASP.NET Web Application. What we want is when the application runs, if the performance counter of “% Processor Time” exceeds 90%, the VM should be restarted.

Setup Azure Monitor Log Analytics

To use the Log Analytics, we need to create a Log Analytics Workspace. This is easily done by searching for Log Analytics Workspaces in the search box (top ) in the Azure portal and then use +Add to create a new Log Analytics Workspace.

Figure: Create log analytics workspace

Once the Log Analytics Workspace is created, go to the section of “Get started with Log Analytics” and select Azure virtual machines (VMs) in it. This section shows a list of VMs in our account. We can select a VM from the list and connect to it.

Figure: Get started with log analytics

Select a VM from the list and connect to it.

Figure: Connect log analytics to vm

When we click the connect button for a VM, a Log Analytics Agent is installed on the machine. Log analytics agent is a component of Log Analytics which resides on infrastructure hosting the application, like a VM. The agent starts collecting the logs from that machine and starts sending those to Log Analytics of Azure Monitor.

Log Analytics Agent does not automatically collect the performance counters of the VM. We have to configure it to do so. Let’s see how we can do that.

1. To start collection of performance counters of VM, let’s open the “Advanced Settings” of the Log Analytics Workspace that we created earlier.

2. On the page that opens, create a Data section. We will select the Windows Performance Counters from the list.

3. From the next list of various performance counters, we can retain the checks in the checkboxes of those performance counters that we want to collect.

4. Finally, by clicking the button of “Add the selected performance counters” we enable the agent to collect those performance counters.

Figure: Add performance counters to log analytics workspace

Now that we have configured Log Analytics and the VM to collect the desired logs of performance counters of that VM, we can create a query that will filter and return the results. We want that only those log records should be considered where the “% Processor Time” is beyond 90%. This is the condition where the performance of the application is below expected and if that is true, we want to restart the VM.

Setup Azure Monitor – Application Insights

Application Insights is part of Azure Monitor that monitors the application that is deployed. One of the parameters that it can monitor is the Quality of Application.

We can measure quality of application in different ways. We can monitor exceptions thrown, unhandled exceptions, logical errors, usability and many more such pointers to quality of application.

Azure Automation Setup to Access Azure Resources

Setup Azure Automation to create PowerShell Runbooks under it. First step in it is to create an Azure Automation Account from the Azure Portal. This is a simple step and does not need any parameters to be selected.

Once our Azure Automation account is created, configure the modules under it to run scripts that will allow us to authenticate to Azure Account, and to access resources under our Azure Subscription. For this, the following modules need to be present in our Azure Automation account:

1. Az.Accounts

2. Az.Computing

3. Az.Resources

4. Az.Storage

To check that these modules are present, click the Modules under Shared Resources section of Azure Account in the Azure portal. You will see that unfortunately and surprisingly, none of the Az modules are added by default to the Azure Automation Account (this is confirmed to be true until mid-March 2020 when I wrote this article). To install these modules, click the Browse Gallery link at the top and then search for the required module, e.g. Az.Accounts.

Figure: Add selected Az module

Ensure that you are importing the stable version of the module and not a Preview version. Preview versions of these modules do not have features that we require. Click the Import button, agree to update dependent modules and then confirm by clicking the OK button. Import all the modules that are required (mentioned above).

Figure: Modules installed in azure automation

Another configuration change to do is to set the access policy. We need to access various Azure resources like storage, VMs etc. in the PowerShell runbooks that we will be creating. To give access of those resources to the identity of the PowerShell runbook, create a Service Principal. In Azure Automation, it is called “Run as Account”. Click the “Run as Account” under the Account Settings section and then click “+ Azure Run As Account”.

Figure: Create Run as Account

This creates the Service Principal and sets it in the “Contributor” role of the RBAC (Role Based Access Control) of the subscription. By doing so, we are allowing access to all Azure resources in our subscription.

We now have all the necessary components setup and configured. These components are:

1. Azure Monitor – Log Analytics

2. Azure Monitor – Application Insights

3. Azure Automation Account

Let’s now create the tests in Application Insights and a query in Log Analytics that will be used for monitoring the application and the hardware hosting that application.

Create the Monitoring Conditions

Create the Availability Test

In this case study, I have taken the example of availability of all the pages of application as the indicator of quality of the application. Availability of pages of application are possible to be monitored using Availability Tests that are created under Application Insights.

I have already written a guide to create such Availability Tests in my earlier tutorial on Application Insights. Please refer to it and create at the least one such availability test for the application that we have hosted on our VM.

Figure: Application Insights Availability Test

If the availability test failure goes above 10%, we want to take an action. This action is to reset IIS so that if the application change has not got initiated, then it will be. Resetting IIS is an action that is subtle compared to restarting the whole VM. It is also more complex because we need to go inside the VM, find the IIS process and restart it using the tool iisreset.exe. We will need to write an Azure Automation PowerShell Runbook to do that.

Create the Log Analytics Query

The queries of Log Analytics are written in a language that is known as Log Analytics Query Language. In its earlier form, it was known as Kusto.

Log Analytics Query Language targets the tables of data that store the collected logs. Some of these tables are event, operation, perf (Performance Counters), syslog etc.

Note: Learn more about Log Analytics Query Language.

Providing just the table name, say Perf, will return all the records (or top 10000, whichever is less) of that table.

We can filter the logs using “where” clauses in the query. To set a filter on the results, we will use the | (pipe) operator and give a where clause. If we want that only records of “% Processor Time” performance counter should be returned, then we will add the where clause as –

| where CounterName == "% Processor Time" 

We can further narrow the results for getting records where value of the "% Processor Time" is more than 90%. To do so, add another where clause as –

| where CounterValue > 90

Now the entire query will look like this –

Perf
| where CounterName == "% Processor Time" 
| where CounterValue > 90

We can also set the time limit for the records to be considered.

When we run this query, it will return only those records that fulfill the filter criteria. I have modified the query for a threshold of 30% to get some results quickly. The query and the results are shown here:

Figure: Log Analytics query with results

We can also sort the results and take only a few records from it by using “sort” and “top” clauses. Aggregation of the results is done with the “Summarize” clause and grouping is done with the “by” clause.

So far, we have used Log Analytics to create a query that filters the logs to give us only the records where “% Processor Time” is beyond 90%. Let us save that query with a name “Pct Proc Load GT 90”.

We can now create an Alert based upon that query. Alert under Azure Monitor needs a condition of a signal logic to be true, and the action to be taken, if it becomes true. The action we need to take is to restart the VM. This action can be configured in Azure Automation as a PowerShell Runbook.

Create Actions against Notifications

Create Azure Automation Runbooks to Take Action When Notified

Once the Azure Automation Account is configured in this way, we can start creating the runbooks. The first runbook that we will create is to restart the VM.

On the Azure Automation account, click the Runbooks under the Process Automation section. It shows a list of three tutorial runbooks. Click on the “Create Runbook” button. Give the name RestartVM to this runbook and select PowerShell as the type of runbook.

Figure: Create a PowerShell Runbook

In the runbook that is created, add the following code:

param (
    [Parameter(Mandatory=$true)]  
    [String] $VMName,
    [Parameter(Mandatory=$true)]  
    [String] $RGName,
) 
## Authentication
try
{
    # Ensures you do not inherit an AzContext in your runbook
    $null = Disable-AzContextAutosave -Scope Process
    $connectionName = "AzureRunAsConnection"
    $Conn = Get-AutomationConnection -Name $connectionName 
    Connect-AzAccount `
-ServicePrincipal `
-Tenant $Conn.TenantID ` 
-ApplicationId $Conn.ApplicationID `
-CertificateThumbprint $Conn.CertificateThumbprint
    Write-Output "Successfully logged into Azure." 
} 
catch
{
    if (!$Conn)
    {
        $ErrorMessage = "Service principal not found."
        throw $ErrorMessage
    } 
    else
    {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}
## End of authentication
Write-Output "Trying to restart virtual machines ..."
try
{
        Restart-AzVM -ResourceGroupName $RGName -Name $VMName      
}
catch
{
    Write-Error -Message $_.Exception
    throw $_.Exception    
}

This script first authenticates to Azure and then runs the Restart-AzVM cmdlet to restart the VM that is identified by the Resource Group Name and the VM Name. We will save this runbook and publish it so that it is available from outside, before creating the second runbook.

The next runbook that we are going to create is for execution of iisreset on the VM. We need to run a PowerShell script on the VM to do that.

Initially we will keep that PowerShell script in an Azure Storage as a blob. This PowerShell script will only have the command of “iisreset”, and nothing else.

Create the PowerShell script in your favorite editor and save it locally as a file “ResetIIS.ps1”. Then in Azure Portal, create an Azure Storage account and a blob container in it named “psscripts”. Set the Container level access permission. Upload the ResetIIS.ps1 file in this container.

Figure: Reset IIS PowerShell script

Once it is saved, get the URL of that file and store it somewhere to use it later in another script.

Figure: URL of PowerShell script blob

Now we will create a second runbook under the Azure Automation Account created earlier. This runbook will also be a PowerShell type runbook. Let’s call it as “ResetIISOnVM”.

The code in this runbook is very similar to the code in our previous one. The only change in it is that instead of Restart command, there will be two different commands. First one is to download the ResetIIS.ps1 PowerShell script from the Azure Storage using wget tool. The second one is the Invoke- AzVMRunCommand cmdlet to call ResetIIS.ps1.

wget "https://ssgsemsstore.blob.core.windows.net/psscripts/ResetIIS.ps1" -outfile ((Get-Location).path + "\ResetIIS.ps1") -UseBasicParsing
Invoke-AzVMRunCommand -ResourceGroupName $RGName -VMName $VMName -ScriptPath ((Get-Location).path + "\ResetIIS.ps1") -CommandId 'RunPowerShellScript'

Parameters of Resource Group Name and VM Name will be provided at the time of creating the alert that calls this runbook. We will now save this runbook.

Now the Log Analytics Queries representing the condition for sending alerts are ready. The Azure Automation Scripts that represent the action are ready too. We only need to connect each script to the appropriate query. This is done with the help of Azure Monitor Alerts.

Creating Azure Monitor Alerts

Alerts is a mechanism in Azure Monitor by which notification of some condition is passed to the receptors. This notification can be as simple as sending an email or a SMS, or it can be complex as a call to an Azure Function or a Logic App. We are going to create the alerts that invoke Azure Automation Runbooks.

Create an Alert for Hardware Overload

To create an alert, we need to specify the condition and the action. We will start the wizard to create a new Alert Rule. Under the Azure Monitor, select the Alerts and click the “+ New alert rule” button to start the alert rule creation wizard. Let us first select the Azure Subscription and Log Analytics Workspace to monitor as the target of monitoring.

To define the logic to raise an alert, click the Add button to select the condition. From the custom saved queries, select the query that we had created to check if “% Processor Time” goes beyond 30. The value 30 is for demonstration purpose only, in the real-life conditions you may want it to be somewhere beyond 80.

Figure: Select query as alert signal logic

The search query that we had created is already there. In addition to the query, we can also configure that alert should be raised if the alert logic threshold exceeds more than three times (3 is for example, we can change that value if desired). We can also change the frequency of evaluation if needed.

Figure: Alert signal logic

The next step is to set the action to be taken when this condition is met. In the section of Action Group, we will create a new action group and a new action under that. In this action, let’s select action of the type Automation Runbook. In the details selection, we will select “User” defined runbook and then drill down to the runbook that we had created to restart the VM. Provide necessary parameters like VM Name and Resource Group Name in which the VM exists.

Figure: Add action to restart VM

Give a name to the Alert Rule and save it by clicking the button of “Create Alert Rule”.

Figure: Add rule restart VM

This way, we have linked the Log Analytics – Azure Monitor Alert – Azure Automation Runbook, so that if the load on the processor exceeds beyond the set limit consistently, the VM will be restarted without any manual intervention.

Create an Alert for Test Failures

We will now create the combination of Application Insights – Azure Monitor Alert – Azure Automation Runbook, so that IIS on the VM will be reset if the availability of the application goes below set limit.

We will start the wizard to create a new alert. Let’s select the Application Insights resource of the application as the resource to be monitored. In the Signal Logic (condition), select the availability test that was created. Set a logic that the Test passed at a value smaller than 95%.

Figure: Signal logic for availability tests condition

In the action, select the Azure Automation Runbook to reset the IIS on VM.

Figure: Add action to reset IIS

Save the alert by clicking the “Create new alert” button. This way, we have linked the Application Insights – Azure Monitor Alert – Azure Automation so that if availability test fails more than a set limit, the IIS on the VM will be reset without any manual intervention.

Conclusion

In this tutorial, we have seen how to automate some of the important parts of Operations, an integral part of DevOps.

We have taken a case of an ASP.NET application hosted on IIS that is on a VM in Azure. We used services offered by Azure Monitor – Log Analytics, Application Insights and Alerts to monitor the application and the hardware hosting it. We used Azure Automation Runbooks to take the desired actions – restart the VM if the hardware is overloaded or to reset IIS, if the application has some problems.

These examples can be extended to monitor other parameters and automate many of the actions that are desired to be taken. Lesser dependency on manual intervention makes these operations more reliable.

This article was technically reviewed by Mahesh Sabnis.

This article has been editorially reviewed by Suprotim Agarwal.

C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.

We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).

Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.

Click here to Explore the Table of Contents or Download Sample Chapters!

Author

Subodh is a Trainer and consultant on Azure DevOps and Scrum. He has an experience of over 33 years in team management, training, consulting, sales, production, software development and deployment. He is an engineer from Pune University and has done his post-graduation from IIT, Madras. He is a Microsoft Most Valuable Professional (MVP) - Developer Technologies (Azure DevOps), Microsoft Certified Trainer (MCT), Microsoft Certified Azure DevOps Engineer Expert, Professional Scrum Developer and Professional Scrum Master (II). He has conducted more than 300 corporate trainings on Microsoft technologies in India, USA, Malaysia, Australia, New Zealand, Singapore, UAE, Philippines and Sri Lanka. He has also completed over 50 consulting assignments - some of which included entire Azure DevOps implementation for the organizations.

He has authored more than 85 tutorials on Azure DevOps, Scrum, TFS and VS ALM which are published on www.dotnetcurry.com.Subodh is a regular speaker at Microsoft events including Partner Leadership Conclave.You can connect with him on LinkedIn .