Operating System KBs

From Documentation

Contents 1 Overview 1.1 Standard Knowledge Base 1.2 OS Knowledge Base Versions 2 OS Knowledge Bases 2.1 CPU Class 2.2 Disk Class 2.3 Error Log Class 2.4 Event Log Class 2.5 Files Class 2.6 Filesystem Class 2.7 Memory Class 2.8 Network Class 2.9 Services Class 2.10 System Class 3 Sentry Details 3.1 Overview 3.2 Sentry State Details 3.2.1 CPU States Sentry 3.2.2 Run Queue Sentry 3.2.3 System Calls Sentry 3.2.4 Disk Sentry 3.2.5 Error Log Sentry 3.2.6 EventLog Sentry 3.2.7 FileInfo Sentry 3.2.8 Filesystem Sentry 3.2.9 Paging File Space Sentry 3.2.10 Paging Rate Sentry 3.2.11 Physical Memory Sentry 3.2.12 Swap Space Sentry 3.2.13 Network Sentry 3.2.14 Services Sentry

Overview

The primary aim of the operating system knowledge bases in Sentinel3G is to provide a base level of operations monitoring that is consistent across various UNIX/Linux platforms. Due to differences between the various operating systems we monitor, complete consistency is not always achievable. This document describes the general content of the OS knowledge bases, and the discrepancies between them on different platforms.

Standard Knowledge Base

The standard knowledge base is OS independent, and so is packaged with Sentinel3G on all Operating Systems. It can be upgraded, but not uninstalled.

Sentry	AIX	HPUX	Linux	Solaris	Tru64	Unixware	Windows
Connectivity¹	√	√	√	√	√	√	√
Event_Manager¹	√	√	√	√	√	√	√
Host_Monitor	√	√	√	√	√	√	√
Scheduler²	√	√	√	√	√	√	√

¹ Connectivity and Event_Manager sentries are only started on the Event Host.
² Scheduler sentry is not started by default. Please read the online documentation for details on how to use the Scheduler sentry.

OS Knowledge Base Versions

OS	Version	Availability Date	Min Sentinel Version
AIX risc	2.1	17th Mar, 2004	4.4
HPUX parisc	2.1	11th May, 2004	4.4
HPUX intel	2.2	6th Jul, 2006	4.4
Linux intel	2.1	20th Apr, 2004	4.4.3
Solaris intel	2.1	14th Jan, 2003	4.4
Solaris sparc	2.2	10th Apr, 2006	4.4.3
Tru64 alpha	2.1	2nd Jun, 2004	4.4
Unixware intel	1.0	10th Mar, 2004	4.2
Windows intel	2.2	28th Apr, 2006	4.4.3

OS Knowledge Bases

CPU Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
CPU_States¹	√	√	√	√	√	√	√
Context_Switches	√	√	√	√		√	√
Interrupts	√	√	√			√	√
Run_Queue	√	√	√	√	√	√	√
Processors	√	√	√		√
System_Calls	√	√		√		√	√

NOTE

Certain operating systems do not provide all the CPU statistics by default, and collecting them may require kernel patches or third party collection tools. Solaris requires packages SUNWaccr and SUNWaccu. Tru64 requires …

¹ All operating systems monitor % System, % User and % Idle CPU time, some OSes provide more information:

OS	More CPU_States Information	Description
AIX, Solaris, Tru64	% Wait IO	The amount of time spent waiting for blocked I/0 to complete.
Linux	% Nice CPU	The percentage of time that the system is in the user state running processes at low (nice) scheduling priority.
HPUX

Disk Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
Disk	√	√	√	√	√	√	√

Error Log Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
Error_Log	√

Event Log Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
EventLog							√

Files Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
File_Info							√

Filesystem Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
Filesystem	√¹	√	√	√	√	√	√

¹ AIX provides two sentries for free space monitoring, one sentry specifically for /usr (with less sensitive thresholds) and another for the other filesystems.

Memory Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
Paging_Rate	√	√	√	√	√	√
Physical_Memory			√		√		√
Paging_File_Space							√
Swap_Rate		√	√		√
Swap_Space	√	√	√	√	√	√

NOTE

Certain operating systems do not provide all the memory statistics, as it may not be relevant (eg Swap_Rate on Tru64 and AIX).

Network Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
Network	√	√	√	√	√	√	√

Services Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
Services	√¹	√	√²	√	√	√	√

¹ AIX provides a complete service management interface using lssrc, startsrc and stopsrc. This interface has been implemented via actions on the Services sentry on AIX.

² Linux provides a complete service management interface using chkconfig and the startup/shutdown scripts in /etc/init.d (/etc/rc.d/init.d on older systems). This interface has been implemented via actions on the Services sentry on Linux.

System Class

Sentry	AIX	HPUX	Linux	SCO	Solaris	Tru64	Windows
CPU_Information	√	√	√¹	√	√	√	√
Memory_Information	√	√	√	√	√	√	√
Operating_System	√	√	√	√	√	√	√
System_Uptime	√	√	√	√	√	√	√

¹ The Linux OS on the i386 platform provides additional CPU information including the approximate speed and vendor of the processors.

Sentry Details

Overview

Sentry	Class	Agent	Poll Time	States	Logging
CPU_States	CPU	Performance	60s	AIX only	√
Context_Switches	CPU	Performance	60s		√
Interrupts	CPU	Performance	60s		√
Run_Queue	CPU	Performance	60s	√	√
Processors	CPU/Processors	MultiProcessor	60s
System_Calls	CPU	Performance	60s	√
Disk	Disk	Disk	120s	√	√
Error_Log	Error_Log	ErrorLog	120s	√
EventLog	EventLog	EventLog	90s	√
File_Info	Files	FileInfo	60s	√
Filesystem	Filesystem	Filesystem	300s	√	√
Paging_Rate	Memory	Performance	60s	AIX only	√
Physical_Memory	Memory	Performance	60s	Solaris only	√
Paging_File_Space	Memory	PageSpace	180s	√	√
Swap_Rate	Memory	Performance	60s		√
Swap_Space (Linux)	Memory	Performance	60s	√	√
Swap_Space (Unix)	Memory	Swap	180s	√	√
Network	Network	Network	120s	√	√
Services	Services	Service	120s	√
CPU_Information	System	Information	n/a³
Memory_Information	System	Information	n/a³
Operating_System	System	Information	n/a³
System_Uptime	System	Uptime	100s

¹ Packets sent and received are known only as sent and received on Solaris and Linux.
² The BadSU agent is a LogFile agent, and so does not have a poll time. Any new data is interpreted whenever the logfile being monitored changes.
³ The Information agent (Hardware agent on Linux) is essentially run only once.

Sentry State Details

CPU States Sentry

Availability

AIX¹, HPUX, Linux, SCO, Solaris, Tru64, Windows

Constants (AIX only)

Constant	Description	Value
CPU_BUSY	User + System percentage indicating the CPU is busy	90
CPU_OVERLOADED	User + System percentage indicating the CPU is overloaded	95

States (AIX only)

State	Severity	Condition	Escalation
OVERLOAD_CPU	warning	$cpu_user + $cpu_system > $CPU_OVERLOADED	severe after 120s
BUSY_CPU	normal	$cpu_user + $cpu_system > $CPU_BUSY	warning after 120s
NOT_BUSY	normal

¹ The CPU States sentry only has constants and states defined for AIX.

Run Queue Sentry

Availability

AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows

Constants

Constant	Description	Value
RUNQ_WARN	Run queue is getting long	3
RUNQ_PROB	Run queue is too long	6

States (HPUX, Linux, SCO, Solaris, Tru64, Windows)

State	Severity	Condition	Escalation
Very_Busy	warning	$run_queue > $RUNQ_PROB	alarm after 210s
Busy	normal	$run_queue > $RUNQ_WARN	warning after 210s
OK	normal

States (AIX only)

State	Severity	Condition	Escalation
OVERLOAD	warning	$run_queue > $RUNQ_PROB	severe after 120s
BUSY	normal	$run_queue > $RUNQ_WARN	warning after 120s
NORMAL	normal

System Calls Sentry

Availability

AIX, HPUX, SCO, Tru64, Windows

Constants (AIX, HPUX, SCO, Tru64)

Constant	Description	Value
CPU_SYSCALLS	Too many system calls per second	10000

States (AIX, HPUX, SCO, Tru64)

State	Severity	Condition	Escalation
BUSY	normal	$sys_per_sec > $CPU_SYSCALLS	alarm after 120s
NORMAL	normal

Disk Sentry

Availability

AIX¹, HPUX, Linux, SCO, Solaris, Tru64, Windows

Constants (Linux, Solaris, Tru64)

Constant	Description	Value
DSK_BUSY_WARN	% busy indicating disk is busy	5
DSK_BUSY_PROB	% busy indicating disk is very busy	20
DSK_SVCT_WARN	Indicates a long service time (ms)	30
DSK_SVCT_PROB	Indicates a very long service time (ms)	50

Constants (AIX, Windows)

Constant	Description	Value
DSK_BUSY_WARN	% busy indicating disk is busy	40
DSK_BUSY_PROB	% busy indicating disk is very busy	60

States (Linux, Solaris, Tru64)

State	Severity	Condition	Escalation
Very_Busy	warning	$percent_busy >= $DSK_BUSY_PROB && $service_time >= $DSK_SVCT_PROB	alarm after 390s
Busy	normal	$percent_busy >= $DSK_BUSY_WARN && $service_time >= $DSK_SVCT_WARN	warning after 390s
OK	normal
Delete	built-in	No data state

States (AIX only)

State	Severity	Condition	Escalation
DSK_VERYBUSY	warning	$percent_busy > $DSK_BUSY_PROB	severe after 120s
DSK_BUSY	normal	$percent_busy > $DSK_BUSY_WARN	warning after 120s
DSK_NORMAL	normal

¹ Unfortunately the service time statistic is not available on AIX. The service time is a better indicator of disk IO performance. Even if a disk is 100% busy, there is no real problem unless the service time for the disk is also getting high.

States (Windows only)

State	Severity	Condition	Escalation
Very_Busy	warning	$percent_busy >= $DSK_BUSY_PROB	alarm after 390s
Busy	normal	$percent_busy >= $DSK_BUSY_WARN	warning after 390s
OK	normal

Error Log Sentry

Availability

AIX only

NOTE

Certain error log entries are ignored by Sentinel 3G. The list of error codes can be found in a file called exclude_errors under the distrib.db folder under the Sentinel installation (/usr/lpp/cosmos/sentinel_4.2/distrib.db by default on AIX)

States (AIX only)

State	Severity	Condition	Escalation
UNKNOWN	severe	$Type == “unknown”	acknowledgement
PERMANENT	alarm	$Type == “permanent”	acknowledgement
TEMPORARY	warning	$Type == “temporary”	acknowledgement
INFORMATION	info	$Type == “informational”	acknowledgement
PENDING	info	$Type == “pending”	acknowledgement
PERFORMANCE	info	$Type == “performance”	acknowledgement

EventLog Sentry

Availability

Windows only

States (Windows only)

State	Severity	Condition	Escalation
Error	severe	$type == “error” || $type == “audit failure”	delete after acknowledgement
Warning	warning	$type == “temporary”	delete after acknowledgement
Information	info	$type == “information” || $type == “audit success”	delete after acknowledgement
Unknown	alarm		delete after acknowledgement

FileInfo Sentry

Availability

Windows only

States (Windows only)

State	Severity	Condition	Escalation
Nonexistent	alarm	$exists == 0
No_Access	warning	$owner == “CAN'T ACCESS FILE”
Dir_Exists	normal	$type == “directory”
File_Exists	normal

Filesystem Sentry

Availability

AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows

Constants

Constant	Description	Value
LOW	Indicating low free space	10
VERY_LOW	Indicating very low free space	5
NEARLY_FULL	Indicating the filesystem is nearly full	2
FULL	Indicating the filesystem is full	0

States (Linux, Solaris, Tru64)

State	Severity	Condition	Escalation
Full	critical	$pct_free == $FS_FULL
Nearly_Full	alarm	$pct_free < $FS_NEARLY_FULL	severe after 930s
Very_Low	warning	$pct_free < $FS_VERY_LOW	alarm after 930s
Low	normal	$pct_free < $FS_LOW	warning after 930s
OK	normal
Delete	built-in	No data state

States (AIX only)

State	Severity	Condition	Escalation
FULL	critical	$pct_free == $FS_FULL
NO_INODES	critical	$pct_free_inodes == $FS_FULL
NEARLY_FULL	severe	$pct_free < $FS_NEARLY_FULL
FEW_INODES	severe	$pct_free_inodes < $FS_NEARLY_FULL
VERY_LOW	alarm	$pct_free < $FS_VERY_LOW
VLOW_INODES	alarm	$pct_free_inodes < $FS_VERY_LOW
LOW	warning	$pct_free < $FS_LOW
LOW_INODES	warning	$pct_free_inodes < $FS_LOW
SUFFICIENT	normal

Paging File Space Sentry

Availability

Windows only

Constants

Constant	Description	Value
SWAP_LOW	Low percent free swap space	15
SWAP_VERY_LOW	Very low percent free swap space	8

States

State	Severity	Condition	Escalation
Very_Low	warning	$pct_avail_page <= $SWAP_VERY_LOW	alarm after 570s
Low	normal	$pct_avail_page <= $SWAP_LOW	warning after 570s
OK	normal

Paging Rate Sentry

Availability

AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows

Constants (AIX only)

Constant	Description	Value
OVER_PAGING	Too many page ins or outs per second	10

States (AIX only)

State	Severity	Condition	Escalation
BUSY	normal	$pgins_per_sec >= $OVER_PAGING || $pgouts_per_sec >= $OVER_PAGING	alarm after 62s
ACCEPTABLE	normal

Physical Memory Sentry

Availability

Linux, Solaris, Windows

Constants (Solaris only)

Constant	Description	Value
RESTIME_LONG	Very long residency time	600
RESTIME_OK	Acceptable residency time (ms)	40
RESTIME_PROB	Indicating residency time is too short	20

States (Solaris only)

State	Severity	Condition	Escalation
Very_Low	warning	$residency_time <= $RESTIME_PROB	alarm after 210s
Low	normal	$residency_time <= $RESTIME_OK	warning after 210s
OK	normal

Swap Space Sentry

Availability

AIX, HPUX, Linux, Solaris, Tru64

Constants

Constant	Description	Value
SWAP_LOW	Low percent free swap space	15
SWAP_VERY_LOW	Very low percent free swap space	10

States

State	Severity	Condition	Escalation
Very_Low	warning	$swap_pct_free <= $SWAP_VERY_LOW	alarm after 570s
Low	normal	$swap_pct_free <= $SWAP_LOW	warning after 570s
OK	normal

Network Sentry

Availability

AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows

Constants

Constant	Description	Value
NET_WORKING	Less than this many transfers and the network is under-utilised	50
NET_COLL_PROB	Indicating excessive collisions	30
NET_COLL_WARN	Indicating many collisions	15
NET_ERROR_OK	Indicating hardware is OK	0
NET_ERROR_PROB	Indicating possible hardware error	0.05

States

State	Severity	Condition	Escalation
Many_Errors	warning	$errors_total >= $NET_ERROR_PROB	alarm after 390s
Very_Busy	warning	$collisions >= $NET_COLL_PROB && $pckts_transmit > $NET_WORKING	alarm after 390s
Some_Errors	normal	$errors_total > $NET_ERROR_OK	warning after 390s
Busy	normal	$collisions >= $NET_COLL_WARN && $pckts_transmit > $NET_WORKING	warning after 390s
OK	normal

Services Sentry

Availability

AIX, HPUX, Linux, SCO, Solaris, Tru64, Windows

States (Linux only)

State	Severity	Condition	Escalation
Confused	info	$Status == “Off” && $PID != “-1”
Off	normal	$Status == “Off”
Not_Running	warning	$PID == -1
Running	normal	$PID != -1
Delete	built-in	No data state

States (Solaris, Tru64)

State	Severity	Condition	Escalation
Not_Running	warning	$count == 0
Runing	normal
Delete	built-in	No data state

States (AIX only)

State	Severity	Condition	Escalation
INACTIVE	disabled	$status == “inoperative”
ACTIVE	normal