Title: Options for Alerting
Course: CIS 208: Linux/UNIX III: Intermediate System Administration, Networking, and Security ~ For IGNITE Review > Section: Learning Modules

Options for Alerting

A good service monitoring program will have the ability to send the minimal number of alerts to the minimal number of people to keep a service available as needed.  Several considerations are made which can prevent overworked sysadmins from receiving unwanted pages or text messages.

Some services will not need to send alerts, such as experimental or test servers.  Others will only need to send alerts during business hours, such as development and staging servers.  Some services may have regularly scheduled down-time for backups or routine maintenance.  Good monitoring software will consider the time of day and pre-scheduled maintenance before sending alerts.

Also, alerts may need to go to different groups depending on the type of service.  For example, problems with network equipment many need to go to one team, while most go to another support team.  Some database issues may need to be routed to a DBA team and not to sysadmins.  In addition, depending on the time of day, alerts may need to go to an entire team or only to the "oncall" personnel.

Alerts can also be sent in an escalated manner.  For example, if the primary oncall contact doesn't respond within a certain time, the secondary can be contacted.  If there is still no response, an alert can go to a manager or to a group.

An important consideration is the thresholds used to determine when an alert will be sent.  For example, it may be normal for a web server to respond in less than 8 seconds, but it occasionally takes up to 12 seconds.  If one test shows a 15 second response time, ideally the monitoring software will wait a short period and retest before issuing an alert.  By default Nagios will wait 60 seconds between tests and will need to see three failures in a row before alerting.  This prevents a lot of "false positives" when sysadmins receive notification of a problem that doesn't exist.