Monitoring Sentries From the Console

From Documentation

This section covers how to open a console window; the layout of the console; and how to diagnose and respond to an event.

Starting a Console

To start a console window from a shell, run this command:

cos sentinel

To start a console window from COSmanager, click the Sentinel button from the COSmanager button bar:

Sentinel3G displays the Desktop view, which is a top-level view of the sentries and folders that you have access to. The Desktop view always contains the Host view folder, and typically also contains several other sentries and folders of particular interest.

Console Toolbar

The Toolbar provides shortcuts to the most common operations.

Figure 9 — Console Toolbar

Context Menu

The context menu lists common menu options for the currently selected object.

Figure 10 — A sample context menu

Changing the View

The standard view shows all the sentries that are visible to the user in the current folder, whatever their current state. The console starts up in Desktop view. You can double-click a folder to open it, or change the view using the view buttons on the Toolbar or the options under the Go menu.

The context menu is activated by clicking with the secondary mouse button (usually the right button for right-handed users) on a sentry or folder, or on the background of the console window.

The context menu provides shortcuts to the most commonly used menu options for the selected object.

Figure 11 — Changing the console view

Go > Desktop shows any top-level custom folders that have been added, plus the system folder to view sentries for a selected host (Host View).

Go > Hosts lists all hosts currently being monitored. You can then open a host folder to show the folders and sentries on that host.

Go > Abnormal sentries combines in a single window all the abnormal sentries in the current folder and all of its sub-folders. Abnormal sentries are those that are in a state other than normal or down. Use the Back button ( ) to return from Abnormal view.

Creating a New Folder

You can create new folders containing selected sentries and folders. This is useful for grouping related sentries from different folders on various hosts. For example, a File Servers folder could contain sentries for filesystems only on hosts that are used as file servers.

1. In Desktop view, select Edit > New folder, or select New folder from the context menu.
2. Enter a name for the folder.
3. Enter a description. This will appear in the status line on the console when this folder is selected.
4. Choose an icon. If none of the existing icons is suitable you can add a new one. See Add Icons on page 154.
5. Enter the file name only (without the path) of a notes file in the Sentinel3G doc directory. These notes will be available from the console to operators when monitoring or responding to alerts relating to this folder.
6. Click Accept to save the new folder. The next step is to copy selected sentries into the new folder.
7. Start another console window containing some sentries you wish to copy: select the folder, then select Open in new window from the context menu.
8. Select and copy the sentries: Ctrl-click or middle-click on a sentry to add it to the selection; Shift-click to add a range of sentries to the selection.
9. Click in the target folder and paste the sentries: select Paste from the context menu.

To specify which roles can access a folder/sentry class

You can restrict access to this folder to users who have specified roles.

1. Click the folder icon to highlight it., then select Edit > Access.
2. Click and select one or more roles. This folder will only be visible to Sentinel3G users who have been assigned at least one of these roles. Leave this field blank to make the folder visible to all Sentinel3G users.
3. Click Accept to save the access details for the folder.

To change details of a folder/sentry class

1. Select the folder.
2. Select Edit > Change. The Sentry/Class Details form opens.
3. Enter or change any of these fields. Which fields can be changed depends on whether the selected object is a system folder, user-defined folder, or host.

Name

You can change the name that is displayed under the icon. System folders cannot be renamed.

This field is available for the Host View icon and user-defined folders.

Description

This will appear in the status line on the console when this folder is selected.

Icon

Choose a different icon to represent this folder on the console.

Notes

file Enter the file name only (without the path) of a notes file in the Sentinel3G doc directory. These notes will be available from the console to operators when monitoring or responding to alerts relating to this folder.

This field is available for system folders.

Click Accept to save the changes.

To delete a user-defined folder/sentry class

Note Only user-defined folders can be deleted. A folder icon with the lock symbol is a system folder. It is required by Sentinel3G and cannot be deleted.

1. In Desktop view, select the folder.
2. Select Delete from the context menu.

Diagnosing an Event

A event is usually first signified by a change in the sentry’s icon or that of its parent folder (such as a change its color, or the appearance of an overlay icon). When a sentry indicates a possible problem, the console provides several ways to view more information to help diagnose the extent and cause of the problem.

The color of the sentry’s name and the color and type of its indicator icon show the current state or severity.

See Overlays and Indicator Icons for a full list of indicators and other overlay icons and what they mean.

Console text provides useful status information about the selected sentry.

If you notice a folder has gone from normal to a higher severity you can quickly find the extent of the problem by clicking the button to view abnormal sentries.

View the sentry’s property sheet

The Property sheet identifies the sentry and shows its current state and severity.

1. Select the sentry.
2. From the context menu, select Properties.

Figure 12 — Sample property sheet

If a sentry is in Failed state, it indicates a problem with the agent (usually that it failed to start or has never returned any valid data). Undefined state indicates that the sentry does not match any of the defined states (in other words, none of the states’ entry conditions evaluated as true).

View the sentry log

The sentry log shows details such state changes, actions performed, and error or warning messages. You can use this log to check the recent history of the sentry leading up to the present alert.

1. Select the sentry.
2. Select Logs > Sentry log.

Figure 13 — Sample sentry log

The most recent entries in the log are displayed in a scrollable window. New entries appear at the bottom of the data as they are added to the log. You can scroll back through the log, or click Follow to return to the end of the log.

View a graph associated with this sentry

1. Select the sentry.
2. Select Report > Realtime graph.
3. If there is more than one graph, choose one.

View the monitoring notes file

1. Select the sentry.
2. Select Help > Monitoring notes.

To add or edit the notes file, see Add a Monitoring Notes File on page 148.

Generate historical reports

The console provides several options for showing the current state of sentries and variables. Sentinel3G also provides two detailed reports based on historical data.

You can use these to analyse changes in the data leading up to an event, or look for trends in the amount of time sentries are spending out of their normal state. These reports can be saved on the Host Monitor host and recalled either from the console or the command line.

The Service Level report summarizes state changes for selected sentries, based on data from the Event Manager log. It shows the proportion of time sentries have spent in each state.

The Logged Data report extracts historical variable data for selected sentries from the sentry logs. It is like a version of the realtime graph option extended back in time.

Responding to an Event

Running an action

Actions are predefined responses associated with a sentry that may be invoked by an operator. Each action runs a command. Depending on its type, each action may display output in the form of a report, or may simply run the command to try to fix the problem.

1. How you run an action depends on the number and type of the sentries and how the action was configured.
   • select a single sentry (or a single instance of a sentry).
   • select multiple instances of a sentry. Note that selecting multiple sentries will only work with sentries that are in the same state and are of the same type.
   • select the parent class folder, or click on the background of the console when in a class.
2. Select Sentry > Action.
3. If there is more than one action, choose one. Only the actions that are appropriate for the current state of the selected sentries will be shown.
4. You may be asked at this point to enter your password (the password of your user account on this host) to confirm that you have the authority to run this action.

The action is run. If the action is a report the output will be displayed in a browser.

Acknowledging an event

A sentry may request acknowledgement from an operator before changing to another state. This is usually done to confirm that the operator has been made aware of a probable “one-off ” incident before returning the sentry to normal state.

If a sentry is waiting for acknowledgement this overlay icon will appear next to it.

You should check to see whether any monitoring notes have been provided to explain your options at this point, and what will happen next if the alert is acknowledged.

To acknowledge a waiting sentry:

1. Select the sentry.
2. Select Sentry > Acknowledge. The sentry will change to the new state.

Enable/disable a sentry

Disabling a sentry stops the state transitions from being processed. You can use this as a temporary way to suppress alerts and notifications, for example when an application or service is down for maintenance. Note that only the sentry is disabled — the agent is still running and its variables are still being set, so any history variables will still be collected.

1. Select the sentry.
2. Select Sentry > Disable.
3. Enter the reason why the sentry is being disabled, then click Accept. This message will be added to the sentry log for auditing purposes and to help other operators who might check the status of this sentry.

To reenable the sentry:

1. Select the sentry.
2. Select Sentry > Enable.

Enable/disable notification for a sentry

Disabling notification for a sentry stops notification messages from being sent. Only notification is disabled—the agent is still running and its variables are still being set, so any history variables will still be collected and alerts will still be processed.

1. Select the sentry.
2. Select Sentry > Notification off.

The sentry changes to show the 'notification off ’ overlay icon:

To reenable notification for the sentry:

1. Select the sentry.
2. Select Sentry > Notification on.

Starting and Stopping the Host Monitor

Restarting the host monitor

If the host monitor is not currently running on a particular host, you must restart it before you can monitor or configure sentries on that host. You must also restart the host monitor to pick up the new settings after making changes to sentries or Sentinel3G’s configuration tables.

1. From the console, select Monitor > Restart host monitor.
2. If you have not already selected which host to work with, Sentinel3G will ask you to choose one now.
3. Click Accept to restart the host monitor. If the action was successful, the message Restart of Host Monitor on <host name>

successful will appear on the status line. If the action was not successful, see Checking the host monitor status log.

Checking the host monitor status log

The host monitor status log records status and error messages generated by sentries, agent programs and the Host Monitor itself. You can use this log to find problems with the configuration of sentries and agents.

1. From the console, select Logs > Host log.

   (If this option is disabled (greyed out) it mean no host is currently selected; go to Host View, select a host and try again.)

The most recent entries in the log are displayed in a scrollable window. New entries appear at the bottom of the data as they are added to the log. You can scroll back through the log, or click Follow to return to the end of the log.

To view the entire log:

1. From the console, select Monitor > Host configuration.
2. From the ‘All Sentries’ window, select Hostmon > View log.

Stopping the host monitor

1. From the console, select Monitor > Stop host monitor.
2. Choose the host.
3. Click Accept to stop the host monitor. If the action was successful, the message Stop Host Monitor on <host name> successful will

appear on the status line. If the action was not successful, see Checking the host monitor status log.

A Host Monitor can also be stopped from a root shell:

1. Start a root shell.
2. Enter this command on Linux systems:

/etc/rc.d/init.d/hostmon stop

1. Enter this command on Solaris and other UNIX systems:

/etc/init.d/hostmon stop

Viewing the event log

The event log contains error and status messages logged by the Event Manager process.

It includes details such as times when the Event Manager was started or stopped, when host monitors connected or disconnected, and error or warning messages generated by these processes. You can use this log to diagnose problems with Sentinel3G.

1. Select Logs > Event log.

The most recent entries in the log are displayed in a scrollable window. New entries appear at the bottom of the data as they are added to the log. You can scroll back through the log, or click Follow to return to the end of the log.