Microsoft was recently announced as the leader in Cloud for PaaS, IaaS and SaaS by Gartner in their recent report . Although there is a big traction in the adoption of PaaS services, IaaS is still a key offering of Azure. Additionally, there is a huge traction in Open Source workload adoption as well. The current stats reveals that one out of three VMs in Azure is Open Source / Linux. Enterprises are not only adopting Azure as Cloud offerings, but they are also doing large scale implementations in a Hybrid Cloud environment.
This article is published from the DNC Magazine for Developers and Architects. Download this magazine from here [PDF] or Subscribe to this magazine for FREE and download all previous and current editions
Considering all these aspects, Infrastructure monitoring is one of the core activities for all IT Administrators and enterprises, as it is the backbone of the ongoing business. Log Analytics (OMS) [formerly known as “Operational Insights”] in Azure caters to all these requirements in one single service. OMS stands for Operational Management Suite. It takes care of Log Analytics, Automation, Availability and Security at one single place. The new enhanced Log Analytics also extends itself to On Premise infrastructure, Amazon (AWS) workload and Open Stack besides traditional Windows and Linux virtual infrastructure in Azure.
This article provides insights on Log Analytics in Azure.
What is Log Analytics (OMS) and who needs it?
All enterprises or Azure customers who are looking for one single dashboard to monitor logs, security and other infrastructure related crucial information, can leverage Log Analytics. The most important and unique feature of Log Analytics is that it helps you to see all details in one place not only for your Virtual Machines on Azure, but also On-Premise Machines, as well as Amazon-AWS Open Stack workloads. When we say Virtual Machines in Azure, it is in no way limited to only Windows VMs, but you can use this for Linux VMs as well. Another good part is, now you can leverage this for your Docker Containers as well. Note that Amazon-AWS and Container Log analytics is in Preview mode, and not generally available.
In this article, we will cover some important features of Log Analytics and the scenarios where it will be useful.
Introduction to Log Analytics (OMS)
What is Log Analytics (OMS)?
Log Analytics (OMS) is an Azure based service which gives you real time operational intelligence and Visualization from your Windows and Linux servers, irrespective of their location and format.
Is Log Analytics (OMS) free in Azure?
There are multiple price slabs available for Log Analytics as below (given in USD):
1. Free (Price: FREE, Daily Limit: 500 MB, Retention Period: 7 days)
2. Standard (Price: $2.30/GB, Daily Limit: None, Retention Period: 1 Month)
3. Premium (Price: $3.50/GB, Daily Limit: None, Retention Period: 12 Months)
How to create Log Analytics (OMS) instance in Azure Portal?
As a prerequisite you require Azure Subscription. You can click on “Browse > Log Analytics (OMS)” or can use the Global Search box at the top of Azure portal and search from there.
While creating OMS workspace, you need to give the following details:
1. OMS Workspace Name (You can also link to an already created Workspace)
3. Resource Group (You can either create new Resource Group or use an existing one)
4. Location (Datacenter Location)
5. Pricing Tier (By default FREE tier, you can select Standard or Premium as per your requirement)
What is the “OMS Portal” option?
Although a similar functionality like OMS was already available in a different format, it is now Cloud based (Azure) with a new name as “Log Analytics”.
You can certainly perform Log Search and other various configurations from the Azure portal, however there is a different dedicated “OMS” portal which has more functionalities and options, than what you can get on the Azure portal. Hence by clicking on “OMS Portal”, an altogether different portal opens up.
Note that the Azure dashboard has limited capabilities, so once you register your VMs (data source); for all other configurations, you need to rely on the OMS portal.
Adding your VMs and Machines
Users need to add existing VMs or Machines to the workspace created. Go to Settings and you can see the Virtual Machine option which enlists the Virtual Machines in your subscription. These VMs can be from different Resource Groups. Here the Quick Start dialog will show you options to connect to:
1. Azure Virtual Machines (VMs) – These VMs can be Windows or Linux as well
2. Storage Account Logs – Storage Account (Can be classic or Resource Manager)
3. Computers (On-Premise machines)
4. System Center Operations Manager (SCOM)
These components are basically treated as “Data Source” for your Log Analytics (OMS) workspace.
You need to select an individual VM, and need to click on “Connect”
The dashboard will give you a view of all connected VMs. To connect to your local machines or System Center Operations Manager, you need to download and install Agent setup (It is available for x64 and x86 Architecture).
Post installation, the agent will start pushing data to the workspace. Note that while installing the Agent, it will ask you for a Workspace ID and Primary Key. You can get this info from the OMS Portal’s Setting page as shown here:
You can also see the other data source options like Storage (including AWS storage which is in Preview as of this writing) and Office 365 (which is also in Preview). Once you set your Data sources, you need to start adding Solutions.
Adding Solutions to your OMS Workspace
Beside Logs and Update Status Visualization, OMS gives you many useful add-in/pluggable solutions which you can add to your workspace to get every minute details of your infrastructure. Not only that, you can also customize and automate it as well, as per your requirements. Generating “Alerts” and building Power BI data sources from the collected data are some value added features.
There is no specific limit on the number of solutions you can add to your workspace. There are multiple solutions available in the Gallery. Some of the commonly used solutions are as below:
1. Active Directory Assessment
4. Alert Management
5. System Updates
6. Change Tracking
7. SQL Assessment
There are some more solutions which you can see in the Solution Gallery on OMS portal, and you can add the same to your workspace. Some of these solutions are in Preview (Not recommended for Production usage) and covers some value added solutions for Docker, Network Performance etc. Some solutions like Service Fabric and Application Dependency monitor will be made available soon. Currently it shows the status as “Coming”. However, for Capacity Management, Configuration Assessment and SCOM alerts, you need SCOM Agent.
From your Settings page on OMS portal, you can see the solutions you have added, and if you want, you can remove them as well. The Solution Setting panel too has a link to the Solution Gallery.
Setting up Dashboard
Once you create a Log Analytics instance from the Azure portal and come to the OMS portal, by default all the widgets are empty. Once you configure Data tab in Settings, you will start getting updated Visuals on the main dashboard, and your customized dashboard as well. You can configure the log data matrix and types of logs depending on your logging and monitoring requirements. This includes Windows and Linux VMs, and Windows machines as well. You can also specify performance counter criteria/filters.
Once you configure Data (Log Matrix) in settings, you can see the visual output on the Dashboard, and can get a more granular level information from the graphs.
Personal Customized Dashboard View
In Settings, Computer Group tab allows you to collect SCCM collection Membership. You can import from Active Directory and WSUS (Windows Server Update Service) as well.
Value Add features of Log Analytics (OMS)
There are multiple features of OMS which in practice does a lot of logging, monitoring and automation. However, here are some key features which are heavily used, and are very useful in large enterprise scenarios.
Over a period of time, huge logs get created in OMS, and most of the time, Administrators need them handy. They also wish to get a drill down information and search facility over these massive amount of logs. OMS solves all these issues. Log Search is also available in Azure Portal along with OMS portal. However, OMS gives additional options and operations over Log Search compared to the Microsoft Azure Portal. Since the data source has multiple VMs and Machines, and there is no separate classification for Windows and Linux, hence all the Logs are at one place. However with filters, you can get more details.
You can also export the result of Log Search to a CSV file from the New Portal. You can also Save the template and can mark it as a favorite to access it later.
Security and Audit
Security and Audit gives you Threat Intelligence which is very helpful to understand valid and invalid attempts to login. You can see a variety of user logins including some from the Operational Team in Microsoft. Thus it helps you to keep track of Valid/Successful Logins as well as Login failures/Invalid attempts to login. It also keeps track of Policy changes, Change or Reset Password Attempts as well as Remote Procedure Calls (RPC) attempts.
Threat Intelligence which is currently in Preview, gives you a Map based Graphical information on the Intrusion attempts, and malicious incoming and outgoing traffic. By clicking on the push pin on the Map, you will get further drill down information like source and geography of malicious incoming traffic, Host name and IP address from where malicious incoming traffic is detected.
Windows Updates or patches is one of the most common routine, but is an important and sensitive activity done by IT team on a day to day basis. However, in a large infrastructure of VMs / Machines, it becomes a difficult task to check the status of updates.
Updates provide all the information related to Windows updates in a single dashboard, and thus you can see information like Number of updates, which ones are critical, Classification of updates along with update age (update history).
Since the entire system should be interactive and efficient, just having all the visualization and log search is not enough. While we have seen various features of OMS till now, and the in-depth information we can get, store and utilize from it; it is also important to maintain the OMS activities running smoothly. Hence Alerts comes into the picture.
You will also see an “Alert” icon in Log Search and other options like Security and Audit as well. Here is a simple way to Add Alert for a particular Log item or value
You can also add One or more Email recipients (usually Administrators and Support Team members) to get Alerts with necessary metadata over email.
Based on the frequency set, and matrix for alert generation, you will get an alert in your inbox as shown below. You then need to do further analysis and take necessary action. You can also manage Alert Rules and do further actions/customizations from Setting page as shown below
Power BI and Preview features
Although Log Analytics (OMS) is GA (Generally available), some of the services and functionalities are still in Preview. We saw how rich the Log Analytics (OMS) tool is, and how it gathers and orchestrates all the infrastructure information into a single workspace.
Although OMS dashboard shows Data visualization for the selective Matrix, Power BI is a generic offering for Data Visualization. Power BI provides Windows, Web and Mobile client. Log Analytics (OMS) portal helps you to generate Power BI data source, so that it can be used by many for rich data visualization.
Based on your Search query over Log data which is pushed to the Power BI data source, you can now visualize that data in your Power BI, and do further filtering and customization if required. You can enable or disable the Preview features from Preview Features tab as shown here:
As we saw there are multiple slabs available in Log Analytics (OMS) as mentioned earlier. For FREE, Standard or Premium, we can see the utilization from OMS portal as well, as seen here:
Microsoft Log Analytics (OMS) is a cloud based service which helps you gain insights into details of infrastructure hosted on-premises and cloud. Log Analytics (OMS) helps you get maximum level of details of your running, ongoing infrastructure irrespective of the datacenter location availability and public clouds (including Azure and Amazon). It delivers deep level insights across datacenter and public cloud (AWS) and VMs, which consists of Windows and Linux VMs as well.