Configuring Sentries and Agents

From Documentation

Revision as of 06:21, 3 May 2006

This chapter describes how to define sentries to monitor resources and respond to events. The main procedures, in order, are:

• define an agent to collect data about the resource you wish to monitor
• define raw and derived variables to pass key parts of this data to the sentry
• define a sentry to represent the resource on the console, and specify whatvariables are to be logged
• define all the different states that the sentry may be in, including any automatic responses that may be run to solve problems
• define actions, reports, and graphs that operators may choose to run at the time of an alert

The first topic lists some questions you should look into when preparing to configure sentries.

Note You must have the Admin or Manager role to configure sentries.

Planning your Sentry

When preparing to build sentries it is useful to consider these questions:

What object or resource do you wish to monitor?

What is the purpose of monitoring this resource:

• To alert operations staff when a threshold is reached?
• To display useful information?
• To record information for future analysis?

How can the status of the resource be queried:

• By running a command?
• By checking in a log file?
• By querying a database?
• By querying an SNMP MIB?
• By receiving data from an application via an API?

Does running one command return data about several instances of an object or resource? Multiple instances imply a ‘cloning’ sentry, in which case one of the attributes or variables that uniquely identifies each instance must be designated as the key column. Will every instance be monitored, or should selected instances be filtered out?

What part of the output from the agent do you wish to capture?

• The exit status?
• Some part of a message in a log file?
• Some part of the text output from STDOUT or STDERR?

How can a message of interest be identified (e.g., by its position? by searching for a unique string or pattern)? How can portions of the data destined to be saved in variables be separated from the rest of the output? Can the output be simplified by discarding header lines or a message prefix?

Will there be ‘spikes’ or gaps in the data that must be allowed for? In other words, when the sentry tests a variable, is it sufficient to test just the current value, or would it be more realistic to take an average of several recent values?

How often does the agent need to collect data? You may need to consider a tradeoff between running the command too frequently and affecting system performance, and not running it frequently enough, in which case the information provided may not be current.

What agent data (that is, variables) do you want to appear on the console? How should they be formatted?

What are the thresholds or triggers that cause a sentry to move from one state to another? If the sentry remains in a state for a long time, does that indicate that the problem is getting more serious? Or that the problem may have resolved itself ?

Should the sentry be changed to another state after a certain period?

How many different states are of interest? Be careful not to multiply states unnecessarily. For example, if you would not expect Sentinel3G or an operator to respond differently when a sentry is in either of two states, then it’s probably safe to merge one of the states into the other. You can even define an ‘information only’ sentry with no states, which simply logs and displays on the console information supplied ny the agent. On the other hand, you may wish to define separate states for reporting purpose. For example, a printer can be idle or printing and both are considered of normal severity, but if you are interested in the ratio of time spent idle to printing, you would need two states to represent this information.

What additional information would be useful to help an operator understand, diagnose, or fix a problem when the sentry is in each state? Is there a command, report, or graph that could be offered to operators to help diagnose the problem?

Is there a standard procedure that should be followed by operators? If so, consider attaching a monitoring notes file to the sentry, state, or agent.

Adding an Agent

Each sentry tests variables supplied by an agent. If the agent has not already been defined you must add it first, then define its variables, before adding the sentry.

1. From the console, select Configure > Host monitor.
2. If you have not already selected the host to update, Sentinel3G will ask you to choose one now. This is the host that the agent will run on. The ‘All Sentries’ window is displayed, showing details of all the sentries defined on this host.
3. Select Tables > Agents. The ‘Agents’ window opens, showing details of all the agents defined on this host.
4. Select Maintain > Add. The ‘Add Agent’ form opens.
5. Figure 19 — ‘Add Agent’ window
6. Enter these fields:

Knowledge BaseThe KB that this agent belongs to. Click to choose one of the predefined KBs installed on this host, otherwise leave this field blank if the agent is not associated with a particular KB. Agent name Enter a unique name for the agent. The name must not contain spaces. Class Choose the TCL handler. This tells Sentinel3G how to parse the output from the agent command. See Agent Classes and Variables on page 38 for more details about selecting an agent class. API An external application sends data via the Sentinel3G API. DB The agent returns data in Functional Database format, typically as a result of a query on a Functional Database table. ExitStatus The agent returns the exit status of the command. LogFile The agent searches in a log file for messages that match a pattern. In the Agent options form you specify a select pattern to select records of interest, and an extract pattern for each text string in the record that must be assigned to a variable. SNMPPolled The agent polls for the current status of a managed object in an SNMP MIB, such as a device or port. Text The agent returns text data to STDOUT. You can use the Agent options form to filter out extraneous text such as blank lines, header lines, and labels. Note Additional classes may be listed depending on which KBs are installed. See the documentation accompanying the KB for more information. Description Enter a description for the agent. Command If the Class is DB, ExitStatus, or Text, enter the command to be run. Other agent classes don’t require a command as they obtain their data by different means. Poll time (secs) There are two ways to use the poll time setting. • A short-running command runs once per poll and terminates immediately after returning its data. Poll time is the time the agent waits before rerunning 96 Configuring Sentries and Agents the command. Note that this is the time between the end of the previous run and the start of the next, so the command won’t run exactly this often. In other words, if the poll time is set to 60 seconds and command takes about 5 seconds, on average the command will run about every 65 seconds. • A persistent or ‘long-running’ command simply returns to the agent the latest data accumulated at an interval determined by the poll time. The command must accept the polling frequency as an argument, which you pass in an environment variable $PollTime. Example: you wish a command ntping to return data to the agent every 60 seconds. ntping has a flag - p that sets the polling frequency. Set Poll time (secs) to 60 and the Command field to: exec ntping -sentinel -p$PollTime <other_arguments> Note Avoid running the command too frequently if testing shows that it may degrade system performance. Multi-instance agents 6. Some agents can return a matrix of values, with each row of the matrix potentially supplying data for one sentry. One example would be an agent that runs the df command to list details of mounted filesystems and returns a sentry instance for each one. An agent that can return a matrix of values and can support multiple sentry instances is called a multi-instance agent. If this is a multi-instance agent, tick the field Multi-instance? and enter these fields also: Instance variable The instance variable is the key field that uniquely identifies each row. In the df example, the instance variable would be the filesystem name. You cannot add an agent’s variables until the agent itself exists, so type in the variable name now and continue to the next field. You can add the details of the variable later. See Adding Variables Used By the Agent on page 106. Instance type This setting specifies how the list of instances is generated. explicit Each sentry will explicitly list the names of the instances. For example, a sentry that monitors log files would list the names of Configuring Sentries and Agents 97 the files it will monitor. The agent will ‘gather’ all the instance names from its associated sentries and pass them to the Command as $Instances. cloning The agent creates or ‘clones’ instances for each row of data returned by the command. both The final list of instances can include both instances specified explicitly by sentries and instances discovered by the agent. For example, the ProcessInfo agent can be set up always to monitor specific system processes, but also to discover arbitrary application or user processes. If Instance type is set to cloning or both, you can specify patterns for instance names to be included in or excluded from the list returned by the agent. If Instance type is explicit, Include and Exclude are disabled. Include and Exclude work in much the same way: the agent gets some data for a particular instance, then, if Include is set, it tests whether the instance name matches any of the patterns. If not it rejects the instance. If the name did match one of the patterns, or if Include is not set, it then matches the instance against the patterns in the Exclude field. If there is a match, the instance is rejected. Include An optional list of patterns used to select instances to be included. Exclude An optional list of patterns used to select instances to be excluded. To stop a sentry being created for a particular instance even if a row is returned, enter its name here. For example if the purpose of the agent is to monitor available disk space, you can exclude rows that represent read-only filesystems such as a CD-ROM drive. Agent options 7. You can pass flags and other arguments recognized by this agent class. Agent options Click to see the available options. These depend on the Class: • If the class is Logfile, see Agent options for LogFile class on page 98. • If the class is SNMPPolled, see Agent options for SNMPPolled class on page 102. • If the class is Text, see Agent options for Text class on page 103. 98 Configuring Sentries and Agents If no options are available for this class the button is disabled. Discovery pgm An optional command that is run before the agent starts. Its job is to return an exit status of true or false based on the existence or status of a resource. If the discovery program returns false, this agent and its associated sentries will not be started. For more details and examples, see Discovery program on page 35. Notes file Enter the file name only (without the path) of a notes file in the Sentinel3G doc directory. These notes will be available from the console to operators when monitoring or responding to alerts relating to this agent. Typically the notes file would describe the variables returned by the agent. Click Accept to save the agent. Agent options for LogFile class The Logfile agent class checks the contents of an ASCII file (usually a log file) for lines that match a pattern. A line is defined as a string of characters terminated by a newline character. One record in a log file may comprise one or more lines. When such a line is found, portions of the record can be assigned to one or more variables. Figure 20 — Example: options for an agent that monitors the message log Configuring Sentries and Agents 99 Logfile name The name of the file to be monitored. Select pattern A regular expression that is used to select records from the log file. Lines matching this pattern will be returned by the agent. Record length The total number of lines in each record. Record offset How many lines before the matching line is the first line in the record. Strip initial chars Ignore this number of characters at the start of every line. Use this to discard a fixed-length prefix (such as a time-stamp) if it will not be used by the sentry. Clear pattern Remove any string that matches this regular expression and replace it with a tab. Use this to discard any text or fields that will not be passed by the agent as a variable, or to simplify the Select pattern by removing extraneous or variable-length text from the middle of a line. Split data by How should the agent assign parts of each matching line to variables? In this field you specify how to break the data into columns–see Assigning Text and Log File Data to Variables on page 41. Later when adding variables for this agent you will specify which columns to assign to each variable–see Adding Variables Used By the Agent on page 106. column Don’t split the data into fields. Instead each line will be treated as a string of characters, with the first character being column 1, the second character column 2, and so on. You will use the "c <col>- <col>" format in the Column field to define each variable. whitespace Break the line into a series of columns separated by white space. Each column is numbered in turn, starting from column 1. tab Break the line into a series of tokens separated by a single tab character (two tabs in a row define a NULL field between them). Each column is numbered in turn, starting from column 1. Clear pattern should be used to replace an unwanted string with a single tab character before the fields are split. pattern Specify a regular expression containing at least one extract pattern, each of which is contained in parentheses. The first extract pat100 Configuring Sentries and Agents tern is treated as column 1, the second extract pattern column 2, and so on. If you selected pattern, specify one or more regular expressions to match patterns in each line of the record. Pattern (line 1) Extract matching variable(s) from the first line in the record. More patterns Click to extract data from additional lines in the record. For example, specify a regular expression in the Pattern (line 2) field to extract matching variables from the second line in the record. Example: selecting a multi-line record from a log file Here is a fragment from a log file, showing one record of five lines. The unique string that identifies this record is the program name, DACCedit, at the start of the third line. The first line containing a variable we wish to keep (the input file) is two lines earlier. If any transactions were flagged we wish the agent to report the file name and the number of transactions that were flagged. • Set Select pattern to DACCedit • Set Record length to 5 • Set Start record to 2 (the first line in the record is two lines before the line containing the select pattern) • Set Split data by to pattern • Set Pattern (line 1) to: INFILE:(.*) • Leave Pattern (line 2) blank • Leave Pattern (line 3) blank • Set Pattern (line 4) to: ([0-9]*) transactions flagged INFILE:/data/MFLA/2001Q2/DACCin/reg3.dat Validating file...... DACCedit V4.1 © 1991–99 TransDACC Ltd. 17 transactions flagged 1128 transactions passed Configuring Sentries and Agents 101 Figure 21 shows the order in which the agent finds a matching record and sets the corresponding variables. Figure 21 — Example: how agent extracts variables from a log file record When you have finished setting the LogFile agent options, click Accept to return to the main Add Agent form. INFILE:/data/MFLA/2001Q2/DACCin/reg3.dat Validating file...... DACCedit V4.1 © 1991–99 TransDACC Ltd. 17 transactions flagged 1128 transactions passed Start record Select pattern Pattern (line 1) Pattern (line 2) Pattern (line 3) Pattern (line 4) 􀁮Agent finds a line in the log file that matches the select pattern 􀁯Agent calculates the first line in the record by counting back from the selected line 􀁰Text that matches an extract pattern in the first 4 lines is used to assign values to variables 102 Configuring Sentries and Agents Agent options for SNMPPolled class Figure 22 — Example: agent options for an agent in SNMPPolled class MIB file name The name of the MIB file, which Sentinel3G expects to find in the directory lib/tnm2.1.10/mibs under the COSmanager home directory. If the MIB file is not already stored there, copy it there now. Multi-host? Select yes if this agent will query multiple hosts, each specified by a separate instance. If this agent does not support multiple instances (that is, if the Multi-instance? field on the Add Agent form is set to no), this field will be disabled. IP address The IP address or hostname of the host to be queried. Port The UDP port number or service name used for SNMP queries (usually specified in /etc/services). SNMP table The name of a table contained in the MIB, which specifies a set of sequences relating to a device being monitored. The agent will “walk the tree” specified in this table to obtain details of each component of the device. For example, if this table specifies that a Configuring Sentries and Agents 103 switch has multiple ports, the agent queries the switch to get the specified details of each port. SNMP version The version of SNMP that the MIB file conforms to. Community The SNMP community to identify and validate the sender of SNMP messages (SNMPv1 and SNMPv2c only). User The user name to identify the sender of SNMP messages (SNMPv2u only). Password Password corresponding to the user name (SNMPv2u only). Timeout (secs) The maximum time to wait for a response from the node being polled. When you have finished setting the SNMPPolled agent options, click Accept to return to the main Add Agent form. Agent options for Text class Figure 23 — Agent options for agents in the Text class 104 Configuring Sentries and Agents Record length The total number of lines in each record. Skip initial lines How many lines to skip at the start of the record. You can use this field to skip a repeating title or header. Skip initial records When the agent starts up, some spurious alerts may be generated from the first couple of polls. For example if the agent is being started during the system boot procedure, the resource being monitored may be under an unusual load from all the other user processes being started, or a large number of events may have accumulated while the agent was not running. You can choose not to process the data collected by the agent in the first few polls. Examples: enter 2 to skip the first two polls; enter 1 to skip only the first poll. Sentinel3G can extract variables from up to four lines of data. If a record contains more than four lines, you can use the next two fields to discard lines that don’t contain data needed by a sentry, such blank lines and headers. Skip blank lines Select yes to discard blank lines. Skip pattern Skip lines containing a match for this pattern. Use this to discard lines that won’t be used to set variables. Skip initial chars Ignore this number of characters at the start of every line. Use this to discard a fixed-length prefix (such as a time-stamp) if it will not be used by the sentry Clear pattern Remove any string that matches this regular expression and replace it with a tab. Use this to discard any text or fields that will not be passed by the agent as a variable, or to simplify the Select pattern by removing extraneous or variable-length text from the middle of a line. Split data by How should the agent assign parts of each matching line to variables? In this field you specify how to break the data into columns–see Assigning Text and Log File Data to Variables on page 41. Later. when adding variables for this agent you will Configuring Sentries and Agents 105 specify which columns to assign to each variable–see Adding Variables Used By the Agent on page 106 column Don’t split the data into fields. Instead each line will be treated as a string of characters, with the first character being column 1, the second character column 2, and so on. You will use the "c <col>- <col>" format in the Column field to define each variable. whitespace Break the line into a series of columns separated by white space. Each column is numbered in turn, starting from column 1. tab Break the line into a series of tokens separated by a single tab character (two tabs in a row define a NULL field between them). Each column is numbered in turn, starting from column 1. Clear pattern should be used to replace an unwanted string with a single tab character before the fields are split. pattern Specify a regular expression containing at least one extract pattern, each of which is contained in parentheses. The first extract pattern is treated as column 1, the second extract pattern column 2, and so on. If you selected pattern, specify one or more regular expressions to match patterns in each line of the record. Pattern (line 1) Extract matching variable(s) from the first line in the record. More patterns Click to extract data from additional lines in the record. For example, specify a regular expression in the Pattern (line 2) field to extract matching variables from the second line in the record. When you have finished setting the Text agent options, click Accept to return to the main Add Agent form. 106 Configuring Sentries and Agents Adding Variables Used By the Agent 1. From the console, select Configure > Host monitor. 2. If you have not already selected the host to update, Sentinel3G will ask you to choose one now. This is the host that the agent will run on. The ‘All Sentries’ window opens, showing details of all the sentries defined on this host. 3. Select Tables > Agents. The ‘Agents’ window opens, showing details of all the agents defined on this host. 4. Select the agent, then select Maintain > Variables. The ‘Variables from Agent <agent_name>’ window opens. 5. Select Maintain > Add. The ‘Add Variable’ form opens. Figure 24 — ‘Add variable’ window Variable name Enter a name for the variable. The name must not contain spaces. It must not have the same name as another variable belonging to Configuring Sentries and Agents 107 this agent. It doesn’t need to be unique across agents; two agents may have variables with the same name. If the agent is in the API class, the name must match one of the variable names passed by the external application. Class Select one of these options, depending on how the value of the variable will be set: raw The value is set by the agent. derived The value will be computed from other variables (in the Expression field on this form). This is often used to express the value of another variable in a different way, such as a rate, proportion, or percentage. trigger If this variable is included in the list of trigger variables for a state, the Expression field will be evaluated when the sentry changes into that state. This is usually used to save the previous value when a new value is received, so that the old and new values can be compared. Type The internal data type. number An integer or floating point number. string A text string. boolean Mainly used by agents in the ExitStatus class. If the command returns 0, the boolean value is true, otherwise it is false. date A date in Functional Database internal format. The date will be stored in the form YYYYMMDD and output as MM/DD/YY (U.S. display format) or DD/MM/YY (European display format). Mainly used by agents in the DB class. datetime A date and time in Functional Database internal format. The date will be stored in the form YYYYMMDD.hhmmss and output as MM/DD/YY-hh:mm (U.S. display format) or DD/MM/YY-hh:mm (European display format). Mainly used by agents in the DB class. clock A count of the number of seconds since 1 Jan 1970 (GMT). You can subtract from the current value a clock value saved earlier to return a time period (for example how long a process has run). Private Leave unchecked if you wish this variable to be available for logging and graphing. 108 Configuring Sentries and Agents Description A longer description of the source or purpose of this variable. Column Enter the field name or the column number(s) in the output that you want to assign to this variable. The format depends on the agent class: On NULL How should the variable be set if the agent doesn’t return a valid value? The options are: zero set the variable to zero null set the variable to null ignore leave the value of the variable unchanged from the previous poll History Should recent values be stored for use in state conditions and realtime graphing? none don’t keep historical values–the previous value will be overwritten each time the agent polls. time keep all values collected within a time period. count keep this number of recent values, one for each time the agent has run. Keep the last If History is set to time, enter a number of seconds to store all values collected within this period. If History is set to count, enter a number of values to be stored. Expression An expression (using TCL EXPR syntax) that calculates, modifies, or reformats the current value of the variable (see Expressions on Text or LogFile Enter the column number(s)— see Assigning Text and Log File Data to Variables on page 41. DB Enter the column name from the Functional Database dictionary entry. SNMPPolled Enter the object ID from the MIB. API Leave this field blank. The variable name and value will be passed explicitly by the external application. ExitStatus Leave this field blank. Configuring Sentries and Agents 109 page 23). How the expression is used depends on the variable class: derived Reformulate the value of another variable as a rate, proportion, or percentage. Any variables attached to the same sentry (including history variables) may be used in the expression. trigger Trigger variables are used to save a previous value that would otherwise be overwritten when the agent receives new data. If this is a trigger variable, use the Expression field to copy the value of another variable you want to save. Any variables attached to the same sentry (including history variables) may be used in the expression. raw An optional expression to post-process the data received from the agent (contained in $data) E.g change units from KB to MB. Note that for “raw variables” the only variable that can be used in the expression is $data, which is the value returned by the agent. Note To return a floating point number, put .0 at the end of any constant values. This is because TCL will do an integer calculation if both parameters are integers. Example: if $data is an integer, $data / 1024.0 will return a floating-point value; $data / 1024 will return an integer. Initial value An expression (using TCL EXPR syntax) that, when evaluated, returns an initial value for the variable. This can be used to set a starting value before the first time the agent polls, or to initialize a trigger variable before it is set with a real value. This is important if, for example, this variable is used elsewhere in an arithmetic expression, to avoid the calculation generating a data error. The last two fields affect how the variable will appear on the console. Units Choose from the table of descriptive units. Examples: % (percent); MB; per secs. To add a new type of unit, see Maintain List of Numeric Units on page 146. Decimal places The value will be rounded to this number of decimal places. Click Accept to save the variable. 110 Configuring Sentries and Agents Adding a Sentry 1. If the console is not in Host View, select Go > Hosts. 2. Select the host that the agent will run on. 3. Select Configure > Host monitor. The ‘All Sentries’ window opens, showing details of all the sentries defined on this host. 4. Select Maintain > Add. The ‘Add Sentry’ form opens. Figure 25 — Sentry details form Knowledge Base Choose the name of the KB that this sentry belongs to. Click to choose one of the predefined KBs installed on this host, otherwise leave this field blank if the agent is not associated with a particular KB. Configuring Sentries and Agents 111 Class/Folder Choose the name of the folder that the sentry will appear within in the console. Sentry Enter a name for the sentry. The name must not contain spaces. Host The host to which this sentry applies. If you leave the field blank it defaults to being the Host Monitor host. If the agent is running remotely from the Host Monitor, you may want the icon for a resource to appear under a different host. If so, you can enter the name of the remote host here. On/Off Set the initial condition of the sentry. on means the sentry will be operating normally. This means the agent will be running and setting variables to be tested by the sentry. off means the agent is not required to collect data on behalf of this sentry, in which case the agent will not be running (unless it also happens to be collecting data for another sentry). You can switch the sentry off if you wish to test it before running it on a production system— see Running Sentries in Test Mode on page 143. Description Enter a description for the sentry. Primary agent Click to choose the main agent whose variables supply this sentry with data. If other agents also supply variables, list them in the Secondary agents field on the Advanced options form below. Variables from the primary agent can be referenced by their name alone. Note that if a primary agent and a secondary agent both have a variable with the same name, the primary agent’s variable is used. Instance details If the primary agent supports multiple instances, click the Instance details button to specify the instance details. 112 Configuring Sentries and Agents There are three ways to define sentry instances: • by cloning one instance for each key value returned by the primary agent • by explicitly defining each instance • by running a command to ‘discover’ the list of instances Clone Tick this checkbox to clone (create another instance of) a new sentry for each instance return by the multi-instance agent. Clone if If Clone is ticked, you can specify an optional TCL expression. New sentries will only be cloned if this expression evaluates to true. Example: You can create two almost identical sentries, one for small filesystems ( < 1GB) and one for large filesystems ( >= 1GB) with different thresholds. Both would use the same agent, but each would have Clone if set to "$size < 1000" and "$size >= 1000" (where $size is in MB) respectively. Sentinel3G will then clone the appropriate sentry only. The two sentries can even have the same name so that they look indistinguishable on the console. If this field is left blank, new sentries will always be cloned. Discover insts This is an optional command that is run when the Host Monitor starts up. For example, the commend could return a list of object names. A sentry instance will be generated for line of data returned by the command. Configuring Sentries and Agents 113 Instances The Instances window enables you to define sentries explicitly. If the Clone field is ticked, you can predefine some of the attributes of the cloned instances (for example, to specify a different label or to turn the instance off). Instances Click to maintain the list of instance names. Select Maintain > Add to add up to four instances at a time. Instance The name of a specific instance, which the sentry passes to the agent. through the $Instances variable in the agent command. On/Off Set the initial condition of the sentry. on means the sentry will be operating normally. off means the sentry will not process agent data unless it is switched on manually. Label If set, this will be used on the console as the name of the instance. If it is not set, the instance name will be used on the console Agent data Agent-specific data for this instance (used by certain agents only). Example: the ProcessInfo agent can be passed a regular expression in this field to match process names. The Instances of Sentry … window includes options to turn off instances or assign them to instance groups. The Turn on an Turn off methods on the Instance menu simply turn on or off the selected instance. These options work on either cloning or explicit instance sentries. For example, for the cloning sentry Free_Space just add the instance for a particular filesystem and turn it off. Assign to group lets you add selected instances to a previously defined instance group. 114 Configuring Sentries and Agents Maintain > Instance groups brings up the instance groups defined for this sentry. When you have finished filling in the Instances form, click Accept to save the instances. When you have finished adding instances, press F3 in the Instances form to return to the Sentry Instance Details form. Note Changes to instances will not be processed until you exit the Instances window and restart the Host Monitor. Instance label If set, this will be used on the console as the name of the instance. You can use a raw variable to give a more meaningful label (for example: use $printer to label each instance with the printer name). If this field is blank the instance will not have a label. Separate Logs? Tick this checkbox to create a separate log file for each instance. Leave the checkbox blank to write entries for all instances to a combined log file. When you have finished filling in the Sentry Instance Details form, click Return to complete the remaining fields in the main Add Sentry form. Console fields Define how the sentry should appear on the console: Text Enter a text string to provide extra information about the current state of the sentry. The string will be displayed in the status area of the console, and may contain both informative messages and the values of variables returned by the primary agent or a secondary agent. Example: HTTP hits: &HttpHits; HTTP errors: &HttpErrors If a variable name is prefixed with $, the raw or unformatted value will be displayed. If a variable name is prefixed with & (ampersand), the formatted value will be displayed. The formatted value appends the Units field, if specified, and converts some variable types such as dates from internal storage format to display format. Configuring Sentries and Agents 115 Examples: if Units= MB: $MBfree would display an amount like "120.6"; &MBfree would display "120.6 MB" Icon Click to choose an icon to represent this sentry on the console. To see what each icon looks like or to add new icons, see Add Icons on page 154. Indicator Select a type of overlay icon to represent the current state of the sentry or to give a rough indication (to within 10 percent) of the current data value returned by Variable. default Represent the current state of the sentry with the default overlay icon for that state or severity. pie chart Represent the current percentage value returned by Variable as a small pie chart. thermometer Represent the current percentage value returned by Variable as a thermometer. Variable Click to choose a variable to be represented next to the icon for this sentry. You specify in the Indicator field whether to show the value of the variable as a pie chart or a thermometer icon. Note For the indicator to work properly, the variable you choose must always be in the range 0 to 100. Default action Click to specify the action that is to be performed by default when an operator double-clicks this sentry on the console. Variables Display the contents of the variables returned by the agent Graph Draw a realtime graph. Click to choose a predefined graph. Action Run a predefined action command. Click to choose a predefined action. 116 Configuring Sentries and Agents Logged_data Generate a logged data report. Click to choose a predefined report. Click Return to complete the remaining fields in the main Add Sentry form. Notes file Enter the file name only (without the path) of a notes file in the Sentinel3G doc directory. These notes will be available from the console to operators when monitoring or responding to alerts relating to this sentry. Advanced options Click the Advanced button to display some additional options relating to notification and data logging. Figure 26 — Advanced sentry options form Configuring Sentries and Agents 117 Notification type Select none to turn off notification for this sentry. Select default to use the global NotifyList and NotifySeverity settings (see Maintain Notification Settings on page 149). Select specify to use the Whom to notify and On severity fields to override the global settings. Whom to notify Choose the name of one or more users to be notified when this sentry changes state. This overrides the global NotifyList setting. On severity Select a threshold level at which the user(s) listed in the Notify field should be notified. Notification will happen when the sentry changes into a state with this severity or higher. This overrides the global NotifySeverity setting. Show variables Click to choose the variables collected by the agent that are used by this sentry. You can do this to shorten the list of variables that will be displayed when an operator double-clicks on the sentry, and the list of variables available for graphing and reporting, or for running actions. This is useful for an agent that returns a large number of variables, such as sar performance statistics which may include CPU, memory, and network, all from the one agent. When an operator double-clicks on a CPU sentry, you can arrange for them to see only the variables relating to CPU statistics. Leave this field blank to show all variables associated with the primary agent. Data Logging The fields in the Data Logging frame are used to specify what data will be collected for the Logged Data Report. This report can be used to check the events leading up to an alert, and for longer-term trend analysis and capacity planning. 118 Configuring Sentries and Agents If you wish to be able to generate a Logged Data Report for this sentry, you need to specify now which numeric variables must be logged and how often. There are two logging methods: time-based (variables are logged at the specified time interval) and state-based (variables are logged after every poll while the sentry is in a specified state or higher). Note that the period for time-based logging is approximate only. The value is actually taken from the next poll after the interval. It follows that there is no benefit to logging data more often than the polling frequency. Avoid specifying too short a period, otherwise the log files can grow large very quickly. It’s better to log data only occasionally during periods of normal operation. then increase the logging frequency during alerts using the Or on severity field. Enable logging Select this option to enable collection of logging data for this sentry. Log variables Click to choose which numeric variables to be logged. A snapshot of the current values of all these variables will be added to the log file at the frequency specified in the Every field. Default settings?Select this option to use the global DefLogTime and DefLogSeverity settings. Leave this option unselected to specify a period and minimum severity level manually. Every Enter a period in minutes. Examples: enter 10 to log data every 10 minutes; enter 60 to log data every hour. Enter 0 to log data every time the agent polls. This field overrides the global DefLogTime setting. Or on severity You can also log data while the sentry is in a particular state or any state of a higher severity. The variables are logged every time the agent polls. This is a way to selectively log more data during alerts. For example, select severe to log every poll while the sentry is in severe or critical state. To switch off state-based logging, select never. This field overrides the global DefLogSeverity setting. Configuring Sentries and Agents 119 Secondary agents Secondary agents If the sentry needs to use variables collected by an agent other than the Primary agent, click to specify these secondary agents. From the Secondary Agents window, select Maintain > Add then enter the following fields: Agent Click to choose the secondary agent. Instance Enter the name of the instance. Leave this field blank to use the same instance as the sentry. Trigger sentry? Select this option to force the sentry’s state to be reevaluated when the agent returns new data. Click Accept to save the details of this secondary agent. When you have finished specifying secondary agents, press F3 to return to the Advanced Sentry Details form. Note Changes to secondary agents will not be processed until you exit the Secondary Agents window. No-data state Click to choose a default state to be used if the agent doesn’t return data for the sentry. An instance of this sentry will change to this state if the agent stops returning data for that instance. Example: when a filesystem is unmounted and df no longer returns details about that filesystem, then the sentry for that instance only will be put into the No-data state. A Delete state is often used in cases like this, where multiple instances of a 120 Configuring Sentries and Agents sentry are created by “cloning”, and you wish to selectively suppress any instance while it is not returning data. Discovery pgm This is an optional command that is run during Host Monitor startup. Its job is to return an exit status of true or false based on the existence or status of a resource. If the discovery program returns false, this sentry will not be started. When you have finished setting the advanced sentry options, click Accept to return to the main Add Sentry form. If you have finished defining the sentry, click Accept to save it and return to the ‘All Sentries’ window. Configuring Sentries and Agents 121 Adding States The next step is to define the states that the sentry can be in. 1. If the console is not in Host View, select Go > Hosts. 2. Select the host that the agent will run on. 3. Select Configure > Host monitor. The ‘All Sentries’ window opens, showing details of all the sentries defined on this host. 4. Select the sentry, then select Maintain > States. The ‘States for sentry <sentry_name>’ window opens. States are listed in the order in which the entry conditions are evaluated. By default, this is in order of decreasing severity, with the most severe at the top. The first state whose entry condition evaluates to true will cause the sentry to enter that state. It follows that states with a NULL entry condition (usually the ‘normal’ state) should be last, for example: You can the drag the states into a different order using the Order > Reorder menu option. 5. There are three ways in which you can add states: • If the sentry was created by cloning, a separate copy of the original sentry’s states is made. If these states are exactly as required, you don’t need to do anything more. If you need to modify a state in some way, select it now and use Maintain > Change to make the changes (see Maintain State Details on page 122). • If there is another sentry that already has a set of states that are similar or identical to what is required, you can copy that sentry’s states: select Maintain > Copy states, then choose the sentry. Note Copy states is only available if the sentry has no states defined. If the new sentry already has states and you wish use the states of another sentry instead, remove the states from the new sentry first. critical $pct_free == 0 severe $pct_free < 5 alarm $pct_free < 10 warning $pct_free < 15 normal <null> 122 Configuring Sentries and Agents If the states are exactly as required, you don’t need to do anything more. If you need to modify a state in some way, select it now and use Maintain > Change to make the changes (see Maintain State Details on page 122). • You can add each state manually: select Maintain > Add. The ‘Add States’ form opens, as shown in Figure 27. Maintain State Details To define the sentry’s attributes and appearance while in this state, enter these fields: Figure 27 — State details form State Enter a unique name for the state. The name must not contain spaces. Severity Select the severity level for this state. The severity determines how the sentry will look while in this state (that is, its color and the color and type of any associated overlay icon). Note that this will result in a notification message being sent if the new severity is at or above the notification level specified globally or for this sentry. Configuring Sentries and Agents 123 The options are listed in order of increasing severity from normal to critical. disabled is a special severity that indicates the sentry is ‘down’ or otherwise unavailable, but doesn’t require attention (for example, a device that has been taken offline for maintenance). Description Enter a description for the state. Entry condition Enter a conditional expression. This is a TCL expression made up of any combination of agent variables, constants, text strings, numbers, history variables, boolean values, and TCL functions. Examples: $Status == "Off" && $PID != "-1" $pct_free < $LOW [hist_avg @cpu_idle] If the entry condition is left blank it evaluates to true. For the correct syntax to refer to variables, see Expressions on page 23. Console These fields define how the sentry will appear on the console while in this state: Text Enter a text string to provide extra information about the current state of the sentry. The string will be displayed in the status area of the console, and may contain both informative messages and the values of variables. Example: HTTP hits: &HttpHits; HTTP errors: &HttpErrors … where HttpHits and HttpErrors are the names of variables returned by the primary agent or a secondary agent. If you prefix a variable name with & it will be formatted for output on the console. For example, numeric variables will be displayed with the correct number of decimal places and with the units after the value, while date/time values will be formatted as a readable dates or times. If you prefix a variable name with $ the raw value will be displayed on the console. 124 Configuring Sentries and Agents You can change the appearance of the sentry’s icon while it is this state, by specifying a different icon or by overlaying the normal icon with an additional indicator icon to modify its appearance. Note To add new icons, see Add Icons on page 154. Icon Click to choose a different icon to represent this sentry while it is in this state. Example: the sentry normally has a 32x32 icon representing a remote system. When it is in network_down state, you choose to use another icon that has a red X indicating a problem with the network connection. If Icon is left blank the default sentry icon will be used. Indicator Click to choose a 16x16 indicator icon. This is an additional small icon that overlays the main icon in the top right hand corner. You can use this to modify the appearance of the icon while the sentry is in this state. If Indicator is left blank the overlay specified in the sentry (thermometer or pie chart) will be used. If neither overlay is specified in the sentry, the default overlay icon and color for the current severity is used (see Overlays and Indicator Icons on page 16). Notes file Enter the file name only (without the path) of a notes file in the Sentinel3G doc directory. These notes will be available from the console to operators when monitoring or responding to alerts relating to this sentry. Trigger variables Click next to the Trigger vars field to list the trigger variables for this sentry. Choose one or more variables whose values you want to reset when the sentry enters this state (see Trigger Variables on page 43). Configuring Sentries and Agents 125 Responses The Responses button lets you define several responses that can be run automatically while the sentry is in this state. Responses Click to see the options for notification, escalation and running automatic responses. Figure 28 — Defining the responses for a state The Response form contains three blocks of response fields which are run in turn if the sentry remains in this state for longer than the specified period. Each response block specifies a waiting period, then three actions Sentinel3G can take at the end of each period. The actions are: changing the severity level, which in turn changes the appearance of the sentry on the console; sending a notification message; and running a command. These options are not mutually exclusive; you can specify any or all actions in each response block. The final group, the Escalation/acknowledgement block, specifies a period after which the sentry will be forced into a different state. See Escalation/acknowledgement on page 127. A typical response is to run a command to fix the problem, and if it succeeds to return the sentry to a normal state. If the command doesn’t fix the problem you may choose to leave the sentry in that state, and specify a later response to run another command or to notify someone. These responses are all optional. You can specify all three, or one, or even none. If no responses or escalation period are specified here, the sentry will remain in this state until and unless the evaluation condition evaluates to a different severity level. 126 Configuring Sentries and Agents If a sentry changes state while it is waiting to process a response, then all responses for that state are cancelled, and any responses for the new state are started. Wait (secs) The length of time to wait after running the previous response, or if this is the first response, after entering this state. Each period is cumulative. In other words the period for Response #2 is counted from the end of the period for Response #1. New severity After the wait period, change the appearance of the sentry on the console to this new severity level. This would usually be done to trigger a global or sentry-level notification message or to increase the apparent urgency of the event by making the icon flash or change color. Note that changing the severity does not change the state of the sentry. Select unchanged if you wish to leave the severity level as it is. Notification Click to specify who will be notified if the sentry is still in this state at the end of the wait period. In the Type field, select one of the following: default Use the default notification list for this sentry. specify In the Who field, choose the names of one or more users. These are in addition to the default notification level for this sentry (see under Advanced options on page 116). If you don’t want to do any additional notification for this response, select none. Command Enter a command to be run by the Host Monitor. This would usually attempt to fix the problem. Example: a ‘free disk space’ sentry could archive files to an offline storage device or remove files such as core and *.o that are deemed expendable. Fire agent Choose an agent to be run. This agent will be polled immediately at the end of the wait period. To poll a specific instance only, enter the instance name in brackets after the agent name. Example: Filesystem(/tmp). Configuring Sentries and Agents 127 Escalation/acknowledgement Another way to respond to an alert is simply to wait for a while to see if the problem corrects itself or more information is received, then to change to another state at the end of that period. Figure 29 — Defining the escalation condition for a state Change to a more severe state if the problem would normally be expected to resolve itself either spontaneously or by the running of the automatic responses. If the sentry is still in this state at the end of the waiting period it suggests some other action must be taken. Change to a less severe state (typically, normal state) if the problem appears to have been a one-off event. For example, the Bad_SU sentry goes into warning state if a failed su attempt is detected. If no further failed su attempts are detected by the end of the waiting period no action need be taken and the sentry can be returned to normal state. The change of state may depend on manual confirmation from an operator (Acknowledgement) or it may happen automatically (Escalation). Wait (secs) The length of time to wait after running the previous response, or if there are no previous responses, after entering this state. If the waiting period is set to 0 seconds, the response (either escalation or the appearance of the acknowledgement icon on the console) will occur as soon as the sentry enters this state. Go to state Choose the new state to change to. Type Select acknowledge if the change of state depends on manual confirmation from an operator. Select escalation if the change of state should happen automatically at the end of the waiting period. 128 Configuring Sentries and Agents When you have finished defining responses and escalation details, click Accept to return to the main Add State form. If you have finished defining this state, click Accept to save it and return to the ‘State Details’ window. Constants Click next to the Constants field to maintain the list of constants and thresholds for this sentry. You can add a new constant, change the details of an existing constant, or adjust the threshold values at which the sentry changes from one state to another. For details about all these tasks, see Maintaining Constants and Thresholds on page 140. Configuring Sentries and Agents 129 Adding an Action or Report A sentry can have several associated actions, which an operator can choose to run from the console. Actions may either be tied to particular states, or can be made available when the sentry is in any state. There are two types: actions typically are used to try to fix a problem; reports display output on the screen and help the operators to diagnose the problem. You can assist operators by explaining in the monitoring notes for the sentry or state when and how each action should be used. When designing an action for a multi-instance sentry you can set it up to run for selected instances or for every instance in a parent folder. For example, you can set up an action so that the output for all instances is combined into one report. 1. From the ‘All Sentries’ window, select the sentry. 2. Select Maintain > Actions. The ‘Actions for sentry <sentry_name>’ window opens. 3. Select Maintain > Add. The ‘Add actions for sentry <sentry_name>’ form opens.(Tip: If a similar action has already been defined for a sentry that uses the same agent, it may be faster to use Maintain > Copy.) Figure 30 — Action details form 130 Configuring Sentries and Agents Action Enter a name for the action. This is the name that will appear in the list of actions that the operator can select from. Type Select whether or not you want the output from the command to be displayed on the operator’s screen: action Simply runs the command without displaying any output. Example: starting a service when it is stopped. report Displays the command’s output on the screen. Command Enter the command, using UNIX shell syntax. The command can make use of the variables $Sentry, $Host, and $Action, which will be set in the environment when the action is run. For multi-instance sentries you can also refer to $Instance, which contains the instance name. To use agent variables in the command, select Uses agent data? below. This example shows how to define a simple report for a singleinstance sentry: echo -n "Report '$Action' "; date; echo " Sentry: $Sentry"; echo " Host: $Host"; When the report is run it will display the name of this action, the date, and the name of the sentry and the host it runs on. For more details and examples of actions and reports, see Actions on page 24. Display command If Type is report, enter a command to display the output from the Command (examples: scroll, db_scroll, db_graph). The default is Sentinel3G’s own browser widget. If the action is run on several instances, all the output from all the commands will be piped to the same display command. Display command is optional if Command handles the displaying of the data itself. Run as user Choose a user name from the password file on this host. The command will run with the privileges of this user. The default account is root. Example: some RDBMS packages require that certain administrative commands be run from a special DB admin account. Configuring Sentries and Agents 131 In state(s) You can make this action available in only certain states. Example: the Services sentry has Stop and Restart actions that are available when a service is in a state that indicates it is running, and a Start action when it is not running Click to choose one or more states. Leave this field blank to make the action available at all times. Access role If set, only users with the specified role can perform this action. If blank, all Sentinel3G users who have the action capability may perform this action. Authenticate? Tick this field to ask for the operator’s password before running the action. Uses agent data? Tick this checkbox if you wish to use any of the agent variables in the Command or Display command fields. This gives the commands access to the same primary agent variables as the sentry. Reads from STDIN? If Uses agent data? is ticked, use this field to specify where the action can find the data: no Command will expect the variables to be set in the environment and accessed by name (e.g. $pct_free). yes The data will be passed from STDIN in Functional Database format. Use this option if you wish to manipulate the data using the Functional Toolset. Fire agent Tick this checkbox if you wish the sentry’s agent to be polled after the action has been run. Export to parent? Tick this checkbox if you wish the action to be available from the parent folder of this sentry. If the action is exported, operators will be able to choose this action both in relation to a selected sentry and for all sentries in the parent folder. 132 Configuring Sentries and Agents Example: a Free Space report that displays details for a selected filesystem (single sentry) or all user filesystems on a host (parent folder). 4. Click Accept to save the action. To test the action from the console, select the sentry and then select Sentry > Action. Configuring Sentries and Agents 133 Adding a Realtime Graph Realtime graphs plot recent values returned by selected variables for a sentry. 1. From the ‘All Sentries’ window, select the sentry. 2. Select Maintain > Realtime graphs. The ‘Realtime graphs for sentry <sentry_name>’ window opens. 3. Select Maintain > Add. The ‘Add realtime graphs for sentry <sentry_name>’ form opens. (Tip: If a similar graph has already been defined for a sentry that uses the same agent, it may be faster to use Maintain > Copy.) Figure 31 — Example: displaying free disk space as a stack graph 134 Configuring Sentries and Agents Specify attributes of the graph Graph type Select the type of graph you wish to use to display the data: line line graphs are useful for gauging trends. bar bar graphs are useful for comparing variables within one observation or comparing adjacent observations. stack stack graphs are typically used where all values add up 100%, Example: CPU usage = %user + %system + %idle Figure 32 shows the same data presented using each type of graph: Figure 32 — Sample disk space data shown as line, bar, and stack graphs Polls displayed The number of values to display across the X-axis of the graph. For example, if you enter 3, values from the last three polls will be displayed. Line graph… …Stack graph Bar graph… Configuring Sentries and Agents 135 The first two fields control the scale on the graph’s Y-axis. If Min value and Max value are not specified, the Y-axis will be sized to the current minimum and maximum data value. This means the scale may change as new values are graphed. To keep the scale constant, set both Min value and Max value. Use close minimum and maximum values if you want to focus on relatively small differences among data values. For example, if a set of variables is mainly of interest when the values are clustered near 100%, a minimum value of 90 will help to separate them. Min value The minimum value to display next to the Y-axis. If the values will always be positive, set Min value=0. Max value The maximum value to display next to the Y-axis. Scale to max? (For stack charts only) Tick this checkbox if you want the values to be scaled so that their sum equals Max value. This is useful where the total adds up approximately to Max Value. Scaling ensures a flat top to the stack. Specify variables You can now choose the names of up to five agent variables whose values are to be graphed. Click next to the Variable details field to specify the attributes of the chosen variables. For example, Figure 33 shows the details for two variables called MBfree and MBused. 136 Configuring Sentries and Agents Figure 33 — Example of a realtime graph that plots two variables For each variable, enter the following details: Color Select the color to be used to display this variable. Label If you wish you can change the default label displayed for this variable. For example, if you wish to scale down a value for free disk space by a factor of 1000, you could also change the label to read GB (gigabytes) instead of MB (megabytes). Scale by This is an optional scaling factor. The values displayed will be multiplied (scaled up) by this factor. Use this to convert very large or small numbers to more manageable units. Example: specify 0.001 to divide the reported values by 1000. Click Return to save the variable details. Configuring Sentries and Agents 137 Specify threshold markers You can now specify up to four markers to be superimposed over the data values. Each marker is displayed as a colored horizontal line and represents a state threshold or other significant value. You can specify both constants associated with this sentry and enter arbitrary integers or floating-point numbers (such as 20, 40, 60, 80). Click next to the Threshold markers field to specify up to four markers. For each marker, specify these details: At value Enter a floating point number, or click to choose one of the constant values defined for this sentry. See Maintaining Constants and Thresholds on page 140. Color Select the color to be used to display this threshold. Remember to use a different color from those used to graph the variables. Use the Test graph option to see which colors show up best. If the threshold is equivalent to a boundary between states, it may be helpful to use the color of the severity level for the higher state. For example, if the constant LOW is the boundary between normal and warning state, and the sentry goes orange when it is in warning state, use orange as the color of the threshold marker. 138 Configuring Sentries and Agents Figure 34 — Generating a Test graph to check colors and thresholds When you have finished specifying markers, click Return to return to the ‘Add realtime graphs for sentry <sentry_name>’ form. Test the graph Now you can test the appearance of the graph. Click next to the Test graph field to generate a graph based on the settings in the form and the most recent data returned by the agent on this host. Note The host monitor must be running and the agent must be returning valid data. You can display several graphs at once by experimenting with different settings and clicking Test graph again. If this is a multi-instance sentry you can test different instances. When you are finished with each graph press F3 to dismiss it. Save the graph details Graph name Enter a unique name to identify this graph. Threshold markers 0.75 0.5 Configuring Sentries and Agents 139 Description Enter a description that explains what this graph will show or when it should be used. This will help operators to select the correct graph to diagnose problems. Title The title that appears in the heading of the graph. It can contain plain text, a variable such as $Instance (for a multi-instance sentry, the name of this instance), or a combination of the two. Export to parent? Tick this checkbox if you wish the graph to be available from the parent folder of this sentry. If the graph is exported, operators will be able to choose this graph when they select the parent folder. 4. Click Accept to save the graph. To test the graph from the console, restart the host monitor, select the sentry and then select Report > Realtime graph. 140 Configuring Sentries and Agents Maintaining Constants and Thresholds Constants are like variables, but they are associated with a sentry rather than an agent. You can use constants in a state’s Entry condition field to define thresholds between states, and as a visual aid on realtime graphs. Example of use: You create a sentry and its states. Some of the states have an entry condition that compares the current data value from the agent with a constant such as VERY_LOW. You clone the sentry. The same set of states is shared between the old and new sentry, but you set the constant VERY_LOW to different a value in each sentry. To display the constants for a sentry 1. From the ‘All Sentries’ window, select the sentry. 2. Select Maintain > Constants. The ‘Constants for sentry <sentry_name>’ window opens. To add a constant for a sentry 1. Select Maintain > Add. The ‘Add Constants’ form opens. 2. Enter the following fields: Configuring Sentries and Agents 141 Constant Enter a name for the constant. The convention is to use uppercase letters and underscores only (e.g., HALF_FULL). The name must be different from other constants belonging to this sentry, though another sentry can have a constant with the same name. Value Set the value of the constant (examples: 3; 0.5; true). Comment Enter an optional comment. Group override? Can an instance group override the value of this constant? If this option is set to yes, the value of this constant always applies to any instance that uses it. If this option is set to no, the value set in an instance group can override the value set here. 3. Click Accept to save the constant. To adjust the values of a sentry’s constants You can adjust the values of all the constants belonging to a sentry. You can use this to fine tune the thresholds at which a sentry changes from one state to another. 1. From the ‘Constants for sentry <sentry_name>’ window, select Maintain > Change values. 142 Configuring Sentries and Agents 2. Change the values next to any of the constants. 3. Click Accept to save the new values. Configuring Sentries and Agents 143 Running Sentries in Test Mode When you are developing sentries, you leave them “switched off ” until you are ready to move them to production mode. You can use the hostmon -T command to test sentries even if they are off, or are in KBs that are off. Running the sentries in test mode will attempt to start all the agents required by the selected sentries and display status messages including an error messages. You can correct any configuration problems and retest the sentries. When you are satisfied that the sentries will work correctly you can change their condition to on. To test sentries, start a Sentinel3G shell then run hostmon -T <sentries…>. Example: cos sentinel -c bash hostmon -T Clients Swap_Size