Configuration Management of Non-Azure VM’s with Azure Automation – Part 2

In part 1 of this series, I discussed the Inventory tool that is a part of Azure Automation’s config management and how to on-board an AWS VM for management.  In this article, I will cover Change Tracking.  With Inventory, you get a report on the Windows files, registry and services, as well installed software for the machines being monitored.  However, Change Tracking takes it a step further and provides a notification whenever there is a change to anything that’s being tracked on the machine.  It also provides the capability to perform queries against the change logs.  Let’s take a look and see how it works.

Since I previously enabled Inventory, there’s nothing that needs to be done to enable Change Tracking.  If you go to your Automation account in the Azure Portal, look under the Configuration Management section and click on “Change Tracking“.  On the main screen, there’s a graphical layout of changes made to Windows services, registry, files, Linux daemons and software being tracked on the AWS VM that was previously on-boarded.  In this case, there was 1 software change and a large number of changes to Windows services.

Below the graph, there’s a tabular layout indicating the resource name, resource type, source machine and time of change.

You can click on one of the resources to see more details about how it was modified.  If I select one of the Windows Update changes, more details are provided about exactly what changed.  Below it shows that the service state changed from “Running” to “Stopped”.

Configure Change Tracking
Change Tracking is customizable by allowing you to configure which Registry keys, services or files are tracked.  To configure Change Tracking, click on the “Edit Settings” button at the top of the main screen.  This will bring you to the Workspace Configuration page for Change Tracking.

Here you will see sections for each type of resource that is able to be tracked.  Under “Windows Registry” in the below screenshot, you will see a list of recommended keys to track.  They’re disabled by default.  To enable a key, just click on it and set enabled to “true”  You can also add a registry key by clicking on the “Add” button.

Under “Windows Files” you can view the files being tracked.  These have to be added manually.  For example, I have added the “c:\windows\system32\drivers\etc” folder path.  If a change is made to the Hosts file or any other file in this location, it will be recorded in Change Tracking.  Also, the process to enable tracking for Linux files is the same.

Another thing you can do under Workspace Configuration is enable the content of modified files to be saved in a Storage account.  To get started, click on the “File Content” tab. Enabling this feature will generate a Share Access Signature (SAS) Uri that can be used to access the stored data.  I won’t go into the details on how this works, but it’s very helpful if you would like to provide others access to the change data.

Lastly, let’s see what can be done under “Windows Services”.  The tracking for Windows services is enabled by default, but you can adjust the frequency that changes to them are reported.  The default setting is every 30 minutes, but you can adjust to anywhere between that and 10 seconds.  Here, I set the collection frequency to 10 minutes.

Querying Change Tracking Logs
Now I am going to discuss my favorite feature in Change Tracking – Logs search and query.  Click on “Log Analytics” to view the page to execute queries against the logged changes.

In the left side pane, there’s a schema that is built on the Workspace called “Trackchangesspace” that Log Analytics uses for Change tracking.  In here there are two databases:  ChangeTracking and LogManagement, which holds records of the data collected by the Tracking tool.  with the Change Tracking database expanded to show its tables ConfigurationChange and ConfigurationData.  In the main window is where queries are executed and the results are shown.  When you first open the Log Analytics page, all the changes collected in the last 24 hours are displayed by default in a tabular view.

By clicking on the “>” symbol next to an item, you can get more details.  For example, below are details about one of the software modifications.  It shows the SoftwareName, Computer, ChangeCategory, Previous and Current states.  In this case, an update for Windows Defender Antivirus was added.

What if you want to run a query of all Windows services that are stopped?  In the top pane, you can write a KQL query that searches in the ConfigurationChange table for all services whose SvcState is equal to stopped in the following format:
Configuration | where SvcState == “Stopped”.  I was pleasantly surprised to discover that the query editor has built-in tab completion and intellisense.  This is very helpful for lousy typists like me, or you are used to IDE’s like VScode or Powershell ISE.  Do note this: the query is case sensitive. If you type “Stopped” with a lower case “s”, the query will not yield any results.  If you are a Powersheller, your experience is that comparison values are not case-sensitive.

Here’s another time saver for building queries:  In the below figure, I went back to the results pane and expanded one of the Windows services to get the detailed information.  By placing the cursor next to “SvcState”, two buttons will appear:  a plus and minus.  Clicking on the plus button will add the search logic for all Windows services where the SvcState is stopped.  Then just click “Run” to get the results.  Nice!

Change Alerting
Lastly, Change Tracking provides the ability to configure alert rules if you would like to receive notifications on certain changes.  In the Logs window, click on “New alert rule” to set this up.  In the screen blow, set the resource as the Workspace used by Change tracking, add the appropriate logic for the condition and define how to receive notifications under Action groups.  You can have the alerts emailed or sent via text messages.

Supplemental resources
The following links are to articles that discuss topics that are related to material covered in this article:
1. – an overview of Kusto Query Language (KQL)
2. – an overview of Chang tracking
3. –  how to view contents of tracked files

This concludes Part 2 of 3 on Configuration management using Azure automation.  In summary, this article explained how to utilize Change tracking to monitor particular resources on an AWS virtual machine, how to perform queries against logged changes and setup alerts for those changes. The final article, part 3, will cover Azure State configuration (DSC).












Configuration Management of Non-Azure VM’s with Azure Automation – Part 1

One of the first questions asked whenever a system or application goes down is “what changed recently”.  Ill-planned or unplanned changes are often the underlying cause of failures.  And if you live on the Operations side of the IT fence like me, a large portion of your existence is dedicated to mitigating the negative impact of these age accelerating events.

To help in this ongoing battle, you can leverage Azure Configuration Management.  This feature is comprised of three tools:  Inventory, Change tracking and State configuration (DSC).  Inventory gives you a view of installed software, Windows services, registry keys, files and Linux Daemons that are present on your machine.  Change Tracking monitors and reports changes to these items.  And DSC allows you to compile configuration files to maintain a desired state for these monitored sources.  Configuration Management can be used for machines that are in Azure of course, but also an On-Premises data center and other cloud services.  For non-Azure machines, the data is gathered by an installed agent and reported back to Azure, where it is stored in a Log Analytics Workspace.

This article will be the first in a three part series that covers the components of Azure Configuration Management and how they can be used with a virtual machine running in AWS EC2. In part 1, I will show you how to get started using Inventory and provide an overview of it’s capabilities.  Part 2 will cover Change Tracking and Part 3 will outline State configuration (DSC).

Here’s what is needed:

  • An Automation account in Azure
  • A Log Analytics Workspace
  • A non-Azure VM (AWS)

I’m not going to outline the steps of creating the above resources.  Here, I will just focus on the process of enabling and using Inventory.

How to Enable Inventory
In the Azure portal, go to your Automation account and select “Inventory” under Configuration Management.  On the popup screen that appears, select your subscription and the Log Analytics Workspace in the drop-down menus, and click on the enable button. This step will also enable Change Tracking.

The next step is to configure the virtual machine to communicate with Log Analytics.

Configure Non-Azure Machine
This Microsoft doc explains the following in more detail. For this task, I will be using an AWS t2.micro instance of Windows 2016.  The first thing that needs to be done is to add registry subkeys for TLS 1.2 and .Net Framework to use strong cryptography.  The Microsoft Monitoring Agent (MMA) that will be installed, uses TLS 1.2 to report changes to the Log Analytics service in Azure.  Here are the Registry Keys for TLS 1.2 and .NET 4.6 that need to be added:

  1. HKLM\System\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client – you will need to create the TLS 1.2 and Client subkeys.  And in the Client subkey, create the following DWORD value:
    • Enabled [Value = 1]
    • DisabledByDefault [Value = 0]
  2. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 – no subkeys need to be created; create the following DWORD value: SchUseStrongCrypto with a value of 1.
  3. HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319 – no subkeys need to be created; created the DWORD value SchUseStrongCrypto with a value of 1.

Now that the Registry edits have been completed, the MMA can be installed.  There are three options for installing the agent: the installation wizard, command line or Powershell DSC.  In this case, I will use the command line.  Also, there are two agent options:  a 64-bit and 32-bit version.  Once the appropriate agent has been downloaded, open an elevated command prompt, navigate to the location of the file and run the following command to extract the MSI file to a folder path.  Finally, install the agent with the extracted MSI file.

Once the agent is installed you can verify the installation a couple of ways.  In Control Panel, go to MMA Agent to check that it’s intstalled as seen here.

Also, in the Azure Portal, you can run a search query to check that the agent has a heartbeat and is communicating with Log Analytics.  Go to the Logs tab on your Workspace to run the query. The following figure shows there’s a heartbeat detected from the AWS EC2 instance.

Now that the VM is communicating with Azure, we can complete the onboarding of the AWS VM.  Go back to Inventory and you will see a new screen with various options:  A button to Add Azure VMs; a doc on how to Add non-Azure machine; another button to Manage Machines.  Click on Manage Machines.

Now you can see the AWS EC2 instance as an available machine.  You have three options to choose on how to enable Change Tracking and Inventory.  I select the default to “Enable on all available machines and click enable.

Now that the Inventory has been enabled, the agent has been installed and the machine is reporting to Azure, let’s see what we get out of the box.  First, if you go to the Inventory tab of your Automation account, it will show you the machine(s) reporting, along with details about the machine (O/S, version, platform, etc).  Additionally, there are tabs to view the software installed on the node, along with Windows services, Windows Registry keys and Linux daemons if applicable.  You will need to configure Files and Registry keys to monitor under Edit settings.  I will demonstrate that in the next article on Change Tracking.

To see the installed software, click on the “Software” tab.  It will show the software name, version, publisher, last refreshed time and number of machines it’s installed on.

Also, go to the “Windows Services” tab to see details such as the service name, status, startup type, last refreshed time and number of machines.

Lastly, the Inventory tools gives you the ability to add machines to a Machine Group.  A default group is created when a machine is added to Inventory.

A Machine group is a group of machines defined by a Log Analytics KQL query.  For instance, if you go to Machine groups and select the “MicrosoftDefaultComputerGroup“, you can see the query that was used to generate the group as well as member machines.

You can create your own Machine group as well.  Click on “Create a Machine Group” on the Inventory screen and fill in the fields on the popup window below.  Here, I’m creating a group of machines that contain “EC2” in the name.  Once you enter a query in the definition box, validate the query and then create it.  A more practical use case would be a group of machines that are missing updates.

This completes part 1 of Azure Configuration Management.  Here, I covered how to enable Inventory, install the monitoring agent on an AWS VM and use Inventory to view a machines configuration.  In Part 2 of this series I will cover Change Tracking.