Operating System KBs
From Documentation
Contents |
Overview
The primary aim of the operating system knowledge bases in Sentinel3G is to provide a base level of operations monitoring that is consistent across various UNIX/Linux platforms. Due to differences between the various operating systems we monitor, complete consistency is not always achievable. This document describes the general content of the OS knowledge bases, and the discrepancies between them on different platforms.
Standard Knowledge Base
The standard knowledge base is OS independent, and so is packaged with Sentinel3G on all Operating Systems. It can be upgraded, but not uninstalled.
| Sentry | AIX | HPUX | Linux | Solaris | Tru64 | Unixware | Windows |
|---|---|---|---|---|---|---|---|
| Connectivity¹ | √ | √ | √ | √ | √ | √ | √ |
| Event_Manager¹ | √ | √ | √ | √ | √ | √ | √ |
| Host_Monitor | √ | √ | √ | √ | √ | √ | √ |
| Scheduler² | √ | √ | √ | √ | √ | √ | √ |
¹ Connectivity and Event_Manager sentries are only started on the Event Host.
² Scheduler sentry is not started by default. Please read the online documentation for details on how to use the Scheduler sentry.
OS Knowledge Base Versions
| OS | Version | Availability Date | Min Sentinel Version |
|---|---|---|---|
| AIX risc | 2.1 | 17th Mar, 2004 | 4.4 |
| HPUX parisc | 2.1 | 11th May, 2004 | 4.4 |
| HPUX intel | 2.2 | 6th Jul, 2006 | 4.4 |
| Linux intel | 2.1 | 20th Apr, 2004 | 4.4.3 |
| Solaris intel | 2.1 | 14th Jan, 2003 | 4.4 |
| Solaris sparc | 2.2 | 10th Apr, 2006 | 4.4.3 |
| Tru64 alpha | 2.1 | 2nd Jun, 2004 | 4.4 |
| Unixware intel | 1.0 | 10th Mar, 2004 | 4.2 |
| Windows intel | 2.2 | 28th Apr, 2006 | 4.4.3 |
OS Knowledge Bases
CPU Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| CPU_States¹ | √ | √ | √ | √ | √ | √ | √ |
| Context_Switches | √ | √ | √ | √ | √ | √ | |
| Interrupts | √ | √ | √ | √ | √ | ||
| Run_Queue | √ | √ | √ | √ | √ | √ | √ |
| Processors | √ | √ | √ | √ | |||
| System_Calls | √ | √ | √ | √ | √ |
- NOTE
- Certain operating systems do not provide all the CPU statistics by default, and collecting them may require kernel patches or third party collection tools. Solaris requires packages SUNWaccr and SUNWaccu. Tru64 requires …
¹ All operating systems monitor % System, % User and % Idle CPU time, some OSes provide more information:
| OS | More CPU_States Information | Description |
|---|---|---|
| AIX, Solaris, Tru64 | % Wait IO | The amount of time spent waiting for blocked I/0 to complete. |
| Linux | % Nice CPU | The percentage of time that the system is in the user state running processes at low (nice) scheduling priority. |
| HPUX |
Disk Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| Disk | √ | √ | √ | √ | √ | √ | √ |
Error Log Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| Error_Log | √ |
Filesystem Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| Filesystem | √¹ | √ | √ | √ | √ | √ | √ |
¹ AIX provides two sentries for free space monitoring, one sentry specifically for /usr (with less sensitive thresholds) and another for the other filesystems.
Memory Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| Paging_Rate | √ | √ | √ | √ | √ | √ | |
| Physical_Memory | √ | √ | √ | ||||
| Swap_Rate | √ | √ | √ | ||||
| Swap_Space | √ | √ | √ | √ | √ | √ | √ |
- NOTE
- Certain operating systems do not provide all the memory statistics, as it may not be relevant (eg Swap_Rate on Tru64 and AIX).
Network Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| Network | √ | √ | √ | √ | √ | √ | √ |
Services Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| Services | √¹ | √ | √² | √ | √ | √ | √ |
¹ AIX provides a complete service management interface using lssrc, startsrc and stopsrc. This interface has been implemented via actions on the Services sentry on AIX.
² Linux provides a complete service management interface using chkconfig and the startup/shutdown scripts in /etc/init.d (/etc/rc.d/init.d on older systems). This interface has been implemented via actions on the Services sentry on Linux.
System Class
| Sentry | AIX | HPUX | Linux | SCO | Solaris | Tru64 | Windows |
|---|---|---|---|---|---|---|---|
| CPU_Information | √ | √ | √¹ | √ | √ | √ | √ |
| Memory_Information | √ | √ | √ | √ | √ | √ | √ |
| Operating_System | √ | √ | √ | √ | √ | √ | √ |
| System_Uptime | √ | √ | √ | √ | √ | √ | √ |
¹ The Linux OS on the i386 platform provides additional CPU information including the approximate speed and vendor of the processors.
Sentry Details
Overview
| Sentry | Class | Agent | Poll Time | States | Logging |
|---|---|---|---|---|---|
| CPU_States | CPU | Performance | 60s | AIX only | √ |
| Context_Switches | CPU | Performance | 60s | √ | |
| Interrupts | CPU | Performance | 60s | √ | |
| Run_Queue | CPU | Performance | 60s | √ | √ |
| Processors | CPU/Processors | MultiProcessor | 60s | ||
| System_Calls | CPU | Performance | 60s | √ | |
| Disk | Disk | Disk | 120s | √ | √ |
| Error_Log | Error_Log | ErrorLog | 120s | √ | |
| Filesystem | Filesystem | Filesystem | 300s | √ | √ |
| Paging_Rate | Memory | Performance | 60s | AIX only | √ |
| Physical_Memory | Memory | Performance | 60s | Solaris only | √ |
| Swap_Rate | Memory | Performance | 60s | √ | |
| Swap_Space (Linux) | Memory | Performance | 60s | √ | √ |
| Swap_Space (non-Linux) | Memory | Swap | 180s | √ | √ |
| Network | Network | Network | 120s | √ | √ |
| Services | Services | Service | 120s | √ | |
| CPU_Information | System | Information | n/a³ | ||
| Memory_Information | System | Information | n/a³ | ||
| Operating_System | System | Information | n/a³ | ||
| System_Uptime | System | Uptime | 100s |
¹ Packets sent and received are known only as sent and received on Solaris and Linux.
² The BadSU agent is a LogFile agent, and so does not have a poll time. Any new data is interpreted whenever the logfile being monitored changes.
³ The Information agent (Hardware agent on Linux) is essentially run only once.
Sentry State Details
CPU States Sentry
- Availability
- AIX¹, HPUX, Linux, SCO, Solaris, Tru64, Windows
Constants (AIX only)
| Constant | Description | Value |
|---|---|---|
| CPU_BUSY | User + System percentage indicating the CPU is busy | 90 |
| CPU_OVERLOADED | User + System percentage indicating the CPU is overloaded | 95 |
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| OVERLOAD_CPU | warning | $cpu_user + $cpu_system > $CPU_OVERLOADED | severe after 120s |
| BUSY_CPU | normal | $cpu_user + $cpu_system > $CPU_BUSY | warning after 120s |
| NOT_BUSY | normal |
¹ The CPU States sentry only has constants and states defined for AIX.
Run Queue Sentry
- Availability
- AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows
Constants
| Constant | Description | Value |
|---|---|---|
| RUNQ_WARN | Run queue is getting long | 3 |
| RUNQ_PROB | Run queue is too long | 6 |
States (HPUX, Linux, SCO, Solaris, Tru64, Windows)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Very_Busy | warning | $run_queue > $RUNQ_PROB | alarm after 210s |
| Busy | normal | $run_queue > $RUNQ_WARN | warning after 210s |
| OK | normal |
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| OVERLOAD | warning | $run_queue > $RUNQ_PROB | severe after 120s |
| BUSY | normal | $run_queue > $RUNQ_WARN | warning after 120s |
| NORMAL | normal |
System Calls Sentry
- Availability
- AIX, HPUX, SCO, Tru64, Windows
Constants
| Constant | Description | Value |
|---|---|---|
| CPU_SYSCALLS | Too many system calls per second | 10000 |
States
| State | Severity | Condition | Escalation |
|---|---|---|---|
| BUSY | normal | $sys_per_sec > $CPU_SYSCALLS | alarm after 120s |
| NORMAL | normal |
Disk Sentry
- Availability
- AIX¹, HPUX, Linux, SCO, Solaris, Tru64, Windows
Constants (Linux, Solaris, Tru64)
| Constant | Description | Value |
|---|---|---|
| DSK_BUSY_WARN | % busy indicating disk is busy | 5 |
| DSK_BUSY_PROB | % busy indicating disk is very busy | 20 |
| DSK_SVCT_WARN | Indicates a long service time (ms) | 30 |
| DSK_SVCT_PROB | Indicates a very long service time (ms) | 50 |
States (Linux, Solaris, Tru64)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Very_Busy | warning | $percent_busy >= $DSK_BUSY_PROB && $service_time >= $DSK_SVCT_PROB | alarm after 390s |
| Busy | normal | $percent_busy >= $DSK_BUSY_WARN && $service_time >= $DSK_SVCT_WARN | warning after 390s |
| OK | normal | ||
| Delete | built-in | No data state |
Constants (AIX, Windows)
| Constant | Description | Value |
|---|---|---|
| DSK_BUSY_WARN | % busy indicating disk is busy | 40 |
| DSK_BUSY_PROB | % busy indicating disk is very busy | 60 |
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| DSK_VERYBUSY | warning | $percent_busy > $DSK_BUSY_PROB | severe after 120s |
| DSK_BUSY | normal | $percent_busy > $DSK_BUSY_WARN | warning after 120s |
| DSK_NORMAL | normal |
¹ Unfortunately the service time statistic is not available on AIX. The service time is a better indicator of disk IO performance. Even if a disk is 100% busy, there is no real problem unless the service time for the disk is also getting high.
Error Log Sentry
- Availability
- AIX only
- NOTE
- Certain error log entries are ignored by Sentinel 3G. The list of error codes can be found in a file called exclude_errors under the distrib.db folder under the Sentinel installation (/usr/lpp/cosmos/sentinel_4.2/distrib.db by default on AIX)
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| UNKNOWN | severe | $Type == “unknown” | acknowledgement |
| PERMANENT | alarm | $Type == “permanent” | acknowledgement |
| TEMPORARY | warning | $Type == “temporary” | acknowledgement |
| INFORMATION | info | $Type == “informational” | acknowledgement |
| PENDING | info | $Type == “pending” | acknowledgement |
| PERFORMANCE | info | $Type == “performance” | acknowledgement |
Filesystem Sentry
- Availability
- AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows
Constants
| Constant | Description | Value |
|---|---|---|
| LOW | Indicating low free space | 10 |
| VERY_LOW | Indicating very low free space | 5 |
| NEARLY_FULL | Indicating the filesystem is nearly full | 2 |
| FULL | Indicating the filesystem is full | 0 |
States (Linux, Solaris, Tru64)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Full | critical | $pct_free == $FS_FULL | |
| Nearly_Full | alarm | $pct_free < $FS_NEARLY_FULL | severe after 930s |
| Very_Low | warning | $pct_free < $FS_VERY_LOW | alarm after 930s |
| Low | normal | $pct_free < $FS_LOW | warning after 930s |
| OK | normal | ||
| Delete | built-in | No data state |
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| FULL | critical | $pct_free == $FS_FULL | |
| NO_INODES | critical | $pct_free_inodes == $FS_FULL | |
| NEARLY_FULL | severe | $pct_free < $FS_NEARLY_FULL | |
| FEW_INODES | severe | $pct_free_inodes < $FS_NEARLY_FULL | |
| VERY_LOW | alarm | $pct_free < $FS_VERY_LOW | |
| VLOW_INODES | alarm | $pct_free_inodes < $FS_VERY_LOW | |
| LOW | warning | $pct_free < $FS_LOW | |
| LOW_INODES | warning | $pct_free_inodes < $FS_LOW | |
| SUFFICIENT | normal |
Paging Rate Sentry
- Availability
- AIX, HPUX, Linux, SCO, Solaris, Tru64
Constants (AIX only)
| Constant | Description | Value |
|---|---|---|
| OVER_PAGING | Too many page ins or outs per second | 10 |
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| BUSY | normal | $pgins_per_sec >= $OVER_PAGING || $pgouts_per_sec >= $OVER_PAGING | alarm after 62s |
| ACCEPTABLE | normal |
Physical Memory Sentry
- Availability
- Linux, Solaris, Windows
Constants (Solaris only)
| Constant | Description | Value |
|---|---|---|
| RESTIME_LONG | Very long residency time | 600 |
| RESTIME_OK | Acceptable residency time (ms) | 40 |
| RESTIME_PROB | Indicating residency time is too short | 20 |
States (Solaris only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Very_Low | warning | $residency_time <= $RESTIME_PROB | alarm after 210s |
| Low | normal | $residency_time <= $RESTIME_OK | warning after 210s |
| OK | normal |
Swap Space Sentry
- Availability
- AIX, HPUX, Linux, Solaris, Tru64, Windows
Constants
| Constant | Description | Value |
|---|---|---|
| SWAP_LOW | Low percent free swap space | 15 |
| SWAP_VERY_LOW | Very low percent free swap space | 10 |
States
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Very_Low | warning | $swap_pct_free <= $SWAP_VERY_LOW | alarm after 570s |
| Low | normal | $swap_pct_free <= $SWAP_LOW | warning after 570s |
| OK | normal |
Network Sentry
- Availability
- AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows
Constants
| Constant | Description | Value |
|---|---|---|
| NET_WORKING | Less than this many transfers and the network is under-utilised | 50 |
| NET_COLL_PROB | Indicating excessive collisions | 30 |
| NET_COLL_WARN | Indicating many collisions | 15 |
| NET_ERROR_OK | Indicating hardware is OK | 0 |
| NET_ERROR_PROB | Indicating possible hardware error | 0.05 |
States
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Many_Errors | warning | $errors_total >= $NET_ERROR_PROB | alarm after 390s |
| Very_Busy | warning | $collisions >= $NET_COLL_PROB && $pckts_transmit > $NET_WORKING | alarm after 390s |
| Some_Errors | normal | $errors_total > $NET_ERROR_OK | warning after 390s |
| Busy | normal | $collisions >= $NET_COLL_WARN && $pckts_transmit > $NET_WORKING | warning after 390s |
| OK | normal |
Services Sentry
- Availability
- AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows
States (Linux only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Confused | info | $Status == “Off” && $PID != “-1” | |
| Off | normal | $Status == “Off” | |
| Not_Running | warning | $PID == -1 | |
| Running | normal | $PID != -1 | |
| Delete | built-in | No data state |
States (Solaris, Tru64)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| Not_Running | warning | $count == 0 | |
| Runing | normal | ||
| Delete | built-in | No data state |
States (AIX only)
| State | Severity | Condition | Escalation |
|---|---|---|---|
| INACTIVE | disabled | $status == “inoperative” | |
| ACTIVE | normal |