Automating Incident & Problem management – Part I: Creating a SCOM monitor and generating fake alerts

This is the first part of a series of posts which will demonstrate how automation can be implemented in Incident and Problem Management. For this, fake SCOM alerts will auto-generate incidents in SCSM, triggering specific Orchestrator runbooks that will provide a fix for the generating issue.

This first part walks through the steps of configuring a Monitor in SCOM and creating fake alerts to be used in future posts.

Create Monitor in SCOM

Create an “Unit Monitor”. For the type, choose: Windows Event – Simple Event Detection – Windows Event Reset. Put this in a separate Management Pack – since the monitor will generate fake alerts, I called it “FakeMonitorMP”.

201311-Create Unit Monitor in SCOM - Windows Event Reset

Create Unit Monitor in SCOM - Management Pack

Create Unit Monitor in SCOM - Windows Event Reset

Target this monitor on “SQL Server 2012 DB Engine”. Leave the default, Parent Monitor “Availability” and make sure that “Monitor is enabled” is checked.

Create Unit Monitor in SCOM - SQL DB Engine

Create Unit Monitor in SCOM - Create Monitor 2

The events to be monitored will show up in the Application Log.

Create Unit Monitor in SCOM - Application Log

Configure the “Unhealthy event” with  “1111” ID and “SQL Server Windows Service Name (SQL DB Engine)” as source.

Event ID Unhealthy Events

For the healthy event, specify “Application” log again, with “2222” ID and “SQL Server Windows Service Name (SQL DB Engine)” as source.

Application Log

Event ID Healthy

Finally, check “Generate alerts for this monitor”. Give the alert a name, such as “Fake Alert for Demo”.

Alert Settings

Create the Alert view

To view the alerts generated by the the monitor we configured above, you need to create an Alert View. First create a folder for this view. You can put it in the same MP you used before, “FakeMonitorMP”.

New Folder in SCOM

Create an Alert View and target data related to “SQL Server 2012 DB Engine”

Alert View in SCOM

Create a fake event

Let’s invent our fake problem and the fake fix for this problem. To populate the Application Event Log with fake events, use the “Write-EventLog” cmdlet in Powershell.

Problem:

On a monitored SQL server machine, open the PowerShell console and type the following command:

> Write-EventLog -LogName Application -EntryType Error -Source MSSQLSERVER -EventId 1111 -Message "There is a fake problem with your DB"

A new alert should appear in SCOM, in the Alert view created above.

New SCOM Alert

Fix:

Check that the “fix” for this problem is also working and type this command:

> Write-EventLog -LogName Application -EntryType Error -Source MSSQLSERVER -EventId 2222 -Message "Everything is OK now"

In SCOM, the alert should be closed now. Note that, while this event is technically not an error, I left the entry type as “Error” to spot it more easily in the EventLog.

Closed SCOM Alert

The first prerequisite is complete. The next post will demonstrate how to configure the SCSMSCOM connector and create a rule for incident creation when the fake alert is triggered in SCOM.