When using an enterprise monitoring tool such as Oracle Enterprise Manager 12c, one of the most critical components is notification.Once an alert or issue has been identified, how do you tell the right people at the right time?Most enterprises use e-mail or open a trouble ticket. As you can imagine, no two enterprises are the same when it comes to their tools and processes. Many customers use one of the more common and well known trouble ticketing systems but quite a few use non-standard or custom (homegrown) trouble ticketing systems. Some customers have special routing requirements or corporate standards and have custom applications which handle all emailing functions instead of directly emailing using an SMTP server.
Oracle Enterprise Manager 12c can handle all of these situations by utilizing one of the various notification methods provided:E-mail, 3rd party connectors and advanced notification methods.There are three types of advanced notifications: SNMP, OS Command or PL/SQL.This blog will introduce you to the OS Command and PL/SQL notification methods available in EM 12c and provide an example of using a custom OS script for notifications.
Advanced Notification Methods: OS Command and PL/SQL
With the advanced notification methods, you can write a notification directly to a table or an OS log for further processing or push a notification to any trouble ticketing system using their command line tools providing the data and variables from EM 12c as input. This method is used by some customers whose corporate standard requires that all alerts be written to a log file, which the ticketing system can poll on a regular basis for alerts.Additionally, advanced notifications allow you to call a procedure whose interface is PL/SQL or access additional data in the Enterprise Manager repository. For example, for a database alert on % of Processes or Sessions Used, you could use PL/SQL to notify the proper application team (stored in a custom target property) that they have too many sessions. To create and configure advanced notifications, the user must have Super Admin privileges.
When creating an advanced notification the first step is to create the OS or PL/SQL script.It’s highly recommended to include debugging and logging information so you can fully understand what information EM 12c is passing and assist in troubleshooting. When using an OS Command in a multi-OMS environment, the script needs to reside on all OMS servers or preferably in a shared location.If you plan to use PL/SQL, the procedure must be created in the repository database before configuring it as a notification method. Of course, any custom objects created in the repository should be created under a separate schema and privileges granted to the SYSMAN user.
TheOracle Enterprise Manager Cloud Administrator’s Guide chapter on Notifications has detailed examples of OS Scripts and PL/SQL that can be utilized in different situations. This chapter also provides detailed information on using passing information to the OS or PL/SQL script and troubleshooting.
Creating Custom Notification Methods
After defining the OS Command or PL/SQL Script, you need to add it to EM 12c as a notification method. In this example, we will create a simple OS script that logs events to a log file, which can be further processed by a custom ticketing system.
#!/bin/ksh
LOG_FILE=/tmp/event.log
if test -f $LOG_FILE
then
echo $TARGET_NAME $MESSAGE $EVENT_REPORTED_TIME >> $LOG_FILE
else
exit 100fi
To create a notification method login as a user with Super Admin privileges and select Setup / Notifications /Notification Methods. Under the Scripts and SNMP Trapssection, click the Add drop down box and select OS Commandand click Go.
Enter a name and provide the fully qualified script location. Use the Test button to validate and click Save.
Adding Notification Methods to Incident Rules
To receive notifications, you will need to create an Incident Rule set and select relevant targets and events to notify and then select the OS Command advanced notification method we created earlier.In this section we will go through the steps to create a simple incident rule. For full details on how to configure your Incident Rules, see the Oracle Enterprise Manager Cloud Control Administrator’s Guide.
Go to Setup / Incidents / Incident Rules. At this point you can either select a rule set to edit, or create a new rule set. In this example we are going to create a new rule set by clicking Create.
On the first screen, enter a user friendly name and description. Since this example is planning to use an advanced notification choose Enterprise.
Since we only want notifications on our Production targets, we’re going to narrow down the list of targets.On Targetstab select Specific targets, select Groupsin the drop down box and click Add.Search for the desired group and select.
Click on the Rules tab, and select Create.
Select the type of rule to create, in this example we are using Incoming events and updates to events.
Enter filter criteria, in this example we are filtering on all Metric Alerts inSeverity Warning and Critical, click Next.
On the Add Actions screen you will define the action that you wish the notification to perform. Click Add.
In the Create Incident or Update Incident section
you can choose to create an Incident, assign Incidents to an administrator, set priority/status or
escalate (if update is selected). Under the Notifications section,
you can select Basic Notifications to send an e-mail or page to a particular
user or users, on top of any methods you might select in the Advanced
Notifications section. In Advanced Notifications you will see the
method we previously created, select that method and click Continue.
ClickNext.
Provide a
user friendly name for your rule and a description, click Next.
Review the details of your rule and click Continue.
Note the rules are not saved until the Save button is clicked.Click OK.
Review your rule set and click Save.
ClickOK.
Validating Incident Rules
Once you have configured your rule set, you should trigger a critical alert on one of the targets included in the rule set.Verify that the rule set triggers the notification method you created and logs the event in the log file. It's helpful to keep the e-mail option turned on during testing so you can validate when the rule was triggered as well.
Identify a target in the group you configured notifications for in the previous section.For this example, we will use a database instance and trigger a Process Limit Usage (%) event.From OracleDatabase / Monitoring / All Metrics, I’ve identified a metric that I can lower thresholds on to trigger an event.Click on Process Limit Usage (%) to drill down to this metric.Notice our Real Time Value of Processes is 16.4.
Click on Modify Thresholdsto set thresholds.
Set the Warning and Critical thresholds lower than the Real Time Value we made note of earlier.Ours was 16.4, so setting the Warning to 5, and Critical to 10.In the Occurrences before Alert field, you may wish to change the value to 1.Since the collection frequency is 10 minutes, if occurrences are set to 3, you will have to wait 30 minutes for it to trigger and again to clear.Click Save Thresholds.
Close the confirmation window.
Navigate to Oracle Database / Monitoring / Incident Manager.Since we have the out-of-box rule set enabled we get an Incident for every critical metric alert and we can see the incident for Process Limit in the Unacknowledged incidentsview.If you have disabled the out-of-box rule set or don’t see the incident, check the Events without incidentsview.Select the appropriate line and in the lower pane click on the Eventstab.
On the Eventstab click on the Message link to drill into the event details.
Here you’ll see the notification method was called under the Last Commentfield.From here, click on Updatestab to see more details.
In the Updates tab, you will see the details of the alert and the notification method that was called to run our /tmp/event_log.sh script.
Finally,
we can check the /tmp/event.log file and see the information that was reported. Be sure to go back and set your threshold to its regular value to clear the false alert you triggered!
Summary
The options for notifications in Oracle Enterprise Manager 12c are very flexible.You can choose e-mail, one of the available connectors to integrate with a 3rd party ticketing system, or use one of the advanced notification methods (SNMP, OS Commands or PL/SQL).In this blog, we’ve shown you how to create an advanced OS Command notification method, how to configure an incident rule set to call that method and how to validate by triggering an alert.Once you are familiar with how to create advanced notification methods and rule sets, you can customize your notifications to suit your needs.
For additional information on configuring your environment for enterprise monitoring, see the whitepaper Strategies for Scalable, Smarter Monitoring using Oracle Enterprise Manager Cloud Control 12c.
Stay Connected with Oracle Enterprise Manager:
Twitter
| 
Facebook
| 
YouTube
| 
Linkedin
| 
Newsletter