FS
Documentation

Task3G/User Guide/Setting Up Jobs

This page was last modified 04:38, 30 May 2013.

From Documentation

< Task3G | User Guide(Difference between revisions)
Jump to: navigation, search
Revision as of 01:14, 6 March 2008
Moff (Talk | contribs)
(Checking Dependencies)
← Previous diff
Current revision
Mike (Talk | contribs)
(Before you run your first job)
Line 9: Line 9:
=== Before you run your first job === === Before you run your first job ===
-Task3G forms the skeleton of a complete task control and batch job submission system. Your next task is to flesh out this skeleton by adding job components for the main jobs and procedures at your site. The Overview on page 15, explains what job+Task3G forms the skeleton of a complete task control and batch job submission system. Your next task is to flesh out this skeleton by adding job components for the main jobs and procedures at your site. The [[Task3G/User Guide/Overview|Task3G Overview]] explains what job components are and how they are related.
-components are and how they are related.+
These details are stored centrally in task3G's database, rather than being duplicated in shell-scripts and applications throughout the network. This means that generalized components need only to be entered once and can be maintained in one place. These details are stored centrally in task3G's database, rather than being duplicated in shell-scripts and applications throughout the network. This means that generalized components need only to be entered once and can be maintained in one place.
Line 21: Line 20:
<br> <br>
 +
=== Planning your job control strategy === === Planning your job control strategy ===
Line 62: Line 62:
;Subjobs: are jobs within a job. These jobs cannot be scheduled. They are to be defined in another job as a task. When it is time for this task (subjob) to run, then the tasks of this subjob will execute. Subjobs help in the sense that larger, more complex jobs can be broken into smaller, simpler steps. ;Subjobs: are jobs within a job. These jobs cannot be scheduled. They are to be defined in another job as a task. When it is time for this task (subjob) to run, then the tasks of this subjob will execute. Subjobs help in the sense that larger, more complex jobs can be broken into smaller, simpler steps.
{{note| Any job can be used as a subjob. However, jobs defined as subjobs cannot be run by themselves.}} {{note| Any job can be used as a subjob. However, jobs defined as subjobs cannot be run by themselves.}}
-{{note| A step by step guide to defining a job can be found in [[Tutorial 2: Creating a Job]].}} 
-{{note| A step by step guide to defining a subjob can be found in [[Tutorial 3: Creating a Subjob]].}} 
<br> <br>
Line 140: Line 138:
;Job parameters: Job parameters cause SHELL ENVIRONMENT variables to be set which are available to all tasks in the job. Commonly they would be included as $ variables which are passed as parameters to the commands in a task. ;Job parameters: Job parameters cause SHELL ENVIRONMENT variables to be set which are available to all tasks in the job. Commonly they would be included as $ variables which are passed as parameters to the commands in a task.
-;Fixed Parameters: Job parameters can be set to fixed values in the job configuration+;Fixed Parameters: Job parameters can be set to fixed values in the job configuration. They should be a space separated list in the form VARIABLE=VALUE. The given VALUE will be assigned to the VARIABLE at the start of the job, and is available as an ENVIRONMENT VARIABLE to all tasks in the job.
-;Prompt form: Job parameters can be set at job submission via a PROMPT form. The full path must be specified, unless the form is in <tt>$APPL_HOME/local/prompt</tt>+;Prompt form: Job parameters can also be set at job submission via a PROMPT form. The full path must be specified, unless the form is in <tt>$APPL_HOME/local/prompt</tt>. As with the 'Fixed parameters' the value(s) specified in the form will be available as ENVIRONMENT VARIABLES to all tasks in the job.
;Pre- and Post-processing: This consists of commands that are executed before and/or after the job has run. The pre-processing consists of the at-submit command, the wait command and the pre-condition. These are executed in the order they appear. The post-processing consists of the at-end command. ;Pre- and Post-processing: This consists of commands that are executed before and/or after the job has run. The pre-processing consists of the at-submit command, the wait command and the pre-condition. These are executed in the order they appear. The post-processing consists of the at-end command.
Line 148: Line 146:
;At-submit command: An operating system command that executes when the job is submitted. This may be interactive if the job is manually submitted. ;At-submit command: An operating system command that executes when the job is submitted. This may be interactive if the job is manually submitted.
-;Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a job begins execution. The job will wait till the expression is satisfied. Form more information, see Creating Wait Conditions on page 49.+;Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a job begins execution. The job will wait till the expression is satisfied. Form more information, see [[#Creating Wait Conditions]].
;Timeout (mins): This is the number of minutes before the wait condition times out. If the wait condition times out, the condition fails and the job is not started. ;Timeout (mins): This is the number of minutes before the wait condition times out. If the wait condition times out, the condition fails and the job is not started.
-;Pre-condition: A UNIX command, script or boolean expression which is executed before the job is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If the precondition fails, the job is not run. For more information, see Creating Wait Conditions on page 49.+;Pre-condition: A UNIX command, script or boolean expression which is executed before the job is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If the precondition fails, the job is not run. For more information, see [[#Creating Wait Conditions]].
;Check Syntax: Checks the syntax of the wait condition and pre-condition and notifies if the conditions are OK or have errors. ;Check Syntax: Checks the syntax of the wait condition and pre-condition and notifies if the conditions are OK or have errors.
Line 168: Line 166:
;Notes file: A file that contains extra information about a particular job which the operator can view. This field must contain a full path to the file, unless it is under <tt>$APPL_HOME/local/notes</tt>. ;Notes file: A file that contains extra information about a particular job which the operator can view. This field must contain a full path to the file, unless it is under <tt>$APPL_HOME/local/notes</tt>.
 +<br>
 +
 +==== Notification Options ====
 +Task3G can notify certain people either when this job requests operator assistance, or when it ends. For each of these, you can specify if and whom to notify, using the following fields:
 +;Notify: When and if to notify if the job fails or requires assistance. 'never' => don't perform this type of notification for this job; 'default' => notify the default people defined in the Task3G settings; 'specify' => notify the people defined in the 'Who' field below.
 +;Who: Space separated list of people to notify. These can be COSmanager users, email addresses or mobile phone numbers (for SMS).
 +
 +
 +{{note| If you specify a COSmanager user their notification method and address must be configured in the COSmanager 'Users' table}}
 +{{note| The mechanics of the EMAIL and SMS notification should be configured in the COSmanager 'Notification Method' table}}
<br> <br>
Line 178: Line 186:
Tasks defined in the 'Task list' console are placed in a pool of tasks which can be chosen when adding tasks to a job. Tasks defined in the 'Tasks in job' console are added to the job but are also placed in this pool, and so can later be added to other jobs. Tasks defined in the 'Task list' console are placed in a pool of tasks which can be chosen when adding tasks to a job. Tasks defined in the 'Tasks in job' console are added to the job but are also placed in this pool, and so can later be added to other jobs.
- +<br>
-'''Configuration Console'''+==== Configuration Console ====
[[Image:Task img 8.jpg|frame|Figure 9 — Configuration console, or ‘Maintain Tasks’ window]] [[Image:Task img 8.jpg|frame|Figure 9 — Configuration console, or ‘Maintain Tasks’ window]]
Line 186: Line 194:
To access the 'Tasks in job' console from the task3G Configuration console select {{cnav|Maintain > Tasks in Job}}. To access the 'Tasks in job' console from the task3G Configuration console select {{cnav|Maintain > Tasks in Job}}.
-To access the 'task3G Jobs' console see COStask Configuration on page 27.+To access the 'task3G Jobs' console see [[#Task3G Configuration]].
-{{note| A step by step guide to defining a queue can be found in Tutorial 4: Creating a Task on page 88.}}+
The {{cnav|Maintain}} pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects {{cnav|Change}}. The {{cnav|Maintain}} pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects {{cnav|Change}}.
Line 198: Line 205:
<br> <br>
 +
=== To Define Information About a Task === === To Define Information About a Task ===
Line 215: Line 223:
;Comments: A description or comment about the task. ;Comments: A description or comment about the task.
- +<br>
-'''Advanced options'''+==== Advanced options ====
You can configure advanced options such as pre- and post-conditions, time-outs, recover actions and operator privileges. Press {{cnav|Advanced}} to enter one or more of these command or options. You can configure advanced options such as pre- and post-conditions, time-outs, recover actions and operator privileges. Press {{cnav|Advanced}} to enter one or more of these command or options.
Line 223: Line 231:
;Pre- and Post-processing: This consists of commands that are executed before and/or after the main command. These can be operating system commands or script. ;Pre- and Post-processing: This consists of commands that are executed before and/or after the main command. These can be operating system commands or script.
-;Pre-condition: A UNIX command, script or boolean expression which is executed before the main command is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If it succeeds, the main command is run, if it fails, neither the main command nor any post-processing is done and the result is dependent on what job recovery action is specified for 'On pre-cond fail'. For more information, see Creating Wait Conditions on page 49.+;Pre-condition: A UNIX command, script or boolean expression which is executed before the main command is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If it succeeds, the main command is run, if it fails, neither the main command nor any post-processing is done and the result is dependent on what job recovery action is specified for 'On pre-cond fail'. For more information, see [[#Creating Wait Conditions]].
;Check Syntax: Checks the syntax of the wait condition and notifies if the condition is OK or has an error. ;Check Syntax: Checks the syntax of the wait condition and notifies if the condition is OK or has an error.
Line 231: Line 239:
;On-error cmd: The on-error command is executed after the main command fails, it will not execute if it succeeds. If the ‘On-error’ command fails, the task fails. If it succeeds, the result is dependent on what job recovery action is specified for 'On-error'. ;On-error cmd: The on-error command is executed after the main command fails, it will not execute if it succeeds. If the ‘On-error’ command fails, the task fails. If it succeeds, the result is dependent on what job recovery action is specified for 'On-error'.
-;Override: status This field only applies if an ‘On-error command’ is specified. If this field is ‘yes’ and the On-error command succeeds, the task completes successfully. This will set the completed tasks status to SUCCESSFUL even though the main command FAILED. If ‘no’ the status of the task will be set to FAIL even if On-error completes successfully.+;Override status: This field only applies if an ‘On-error command’ is specified. If this field is ‘yes’ and the On-error command succeeds, the task completes successfully. This will set the completed tasks status to SUCCESSFUL even though the main command FAILED. If ‘no’ the status of the task will be set to FAIL even if On-error completes successfully.
;Timeout: This is the number of minutes before the task times out. If the task executes for longer than this time, the task is killed and the result is dependent on what job recovery action is specified for 'On timeout'. ;Timeout: This is the number of minutes before the task times out. If the task executes for longer than this time, the task is killed and the result is dependent on what job recovery action is specified for 'On timeout'.
 +<br>
-'''Job recovery actions'''+==== Job recovery actions ====
These recovery actions are triggered on command and timeout failures. There are one of four possible actions: assist, stop, continue and ignore. These recovery actions are triggered on command and timeout failures. There are one of four possible actions: assist, stop, continue and ignore.
Line 253: Line 262:
;On timeout: This recovery action is triggered when the task times out. ;On timeout: This recovery action is triggered when the task times out.
- +<br>
-'''Operator Privileges'''+==== Operator Privileges ====
The following options allow or disallow the operator from performing certain functions on this task: The following options allow or disallow the operator from performing certain functions on this task:
Line 274: Line 283:
From the 'task3G Jobs' console, select Maintain > Tasks in job. From the 'task3G Jobs' console, select Maintain > Tasks in job.
-To access the 'task3G Jobs' console, see COStask Configuration on page 27.+To access the 'task3G Jobs' console, see [[#Task3G Configuration]].
[[Image:Task img 11.jpg|frame|Figure 12 — 'Tasks in Job' console]] [[Image:Task img 11.jpg|frame|Figure 12 — 'Tasks in Job' console]]
Line 409: Line 418:
=== Creating Wait Conditions === === Creating Wait Conditions ===
-'Wait For' conditions are defined against a task in a job. To create a wait for condition, select a task then select Dependencies > Wait for.+'Wait For' conditions are defined against a task in a job. To create a wait for condition, select a task then select {{cnav|Dependencies > Wait for}}.
-{{note| A step by step guide to defining a wait condition can be found in Tutorial 6: Creating Wait Conditions on page 94.}}+
- Figure 13 — Wait for condition+[[Image:Task img 16.jpg|frame|Figure 13 — Wait for condition]]
 +;Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a task begins execution. The task will wait till the expression is satisfied.
-Wait condition A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a task begins execution. The task will wait till the expression is satisfied. 
The wait condition uses the following symbols: The wait condition uses the following symbols:
-Normal precedence for boolean expressions applies and brackets can be used. Primary+ 
-expressions can be:+:{| border="0" cellpadding="3" cellspacing="0"
-• A job or task name of the form+! align="left" width="75" style="border-bottom:1px solid grey;border-right:1px solid grey;" | Symbol
-Symbol Meaning+! align="left" width="250" style="border-bottom:1px solid grey;" | Meaning
-|| OR+|-
-&& AND+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | &brvbar;&brvbar;
-== EQUAL TO+| style="border-bottom:1px solid grey;" | OR
-!= NOT EQUAL TO+|-
-> GREATER THAN+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | &&
-< LESS THAN+| style="border-bottom:1px solid grey;" | AND
->= GREATER THAN OR EQUAL TO+|-
-<= LESS THAN OR EQUAL TO+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | ==
-! NOT+| style="border-bottom:1px solid grey;" | EQUAL TO
-Setting Up Jobs 51+|-
-$name where 'name' is a task in the current job.+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | !=
-$job where 'job' is another task3G job.+| style="border-bottom:1px solid grey;" | NOT EQUAL TO
-$job:name where 'name' is a task in job 'job'.+|-
-You can extend the above form by testing for a particular exit status, using+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | >
-the numeric operators listed above.+| style="border-bottom:1px solid grey;" | GREATER THAN
-Example: $step3 == 5+|-
-This expression will wait for the task or job to finish, then either return+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | <
-TRUE or FALSE.+| style="border-bottom:1px solid grey;" | LESS THAN
-An command (including a shell test [ ] command). This will be run periodically+|-
-until it returns TRUE (zero exit status).+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | >=
-Note that if a shell command returns a non-zero status, the condition will+| style="border-bottom:1px solid grey;" | GREATER THAN OR EQUAL TO
-WAIT for 60 seconds and try again. It will not return FALSE.+|-
-task3G uses the following convention for both task and job statuses:+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | <=
 +| style="border-bottom:1px solid grey;" | LESS THAN OR EQUAL TO
 +|-
 +| valign="top" style="border-right:1px solid grey;" | !
 +| NOT
 +|}
 + 
 + 
 +Normal precedence for boolean expressions applies and brackets can be used. Primary expressions can be:
 +*A job or task name of the form
 +*:'''''<tt>$name'''''&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt> where 'name' is a task in the current job.
 +*:'''''<tt>$job'''''&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt> where 'job' is another task3G job.
 +*:'''''<tt>$job:name'''''&nbsp;</tt> where 'name' is a task in job 'job'.
 + 
 +*You can extend the above form by testing for a particular exit status, using the numeric operators listed above.
 +*:Example: <code>$step3 == 5</code>
 +*:This expression will wait for the task or job to finish, then either return TRUE or FALSE.
 + 
 +*An command (including a shell test <tt>[ ]</tt> command). This will be run periodically until it returns TRUE (zero exit status).
 +{{note| that if a shell command returns a non-zero status, the condition will WAIT for 60 seconds and try again. It will not return FALSE.}}
 + 
 + 
 +Task3G uses the following convention for both task and job statuses:
 + 
 +:{| border="0" cellpadding="3" cellspacing="0"
 +! align="left" width="150" style="border-bottom:1px solid grey;border-right:1px solid grey;" | Numeric Exit Status
 +! align="left" width="150" style="border-bottom:1px solid grey;" | Symbol
 +|-
 +| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 0
 +| style="border-bottom:1px solid grey;" | OK
 +|-
 +| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 1
 +| style="border-bottom:1px solid grey;" | ERR[OR]
 +|-
 +| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 2
 +| style="border-bottom:1px solid grey;" | WARN[ING]
 +|-
 +| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 10
 +| style="border-bottom:1px solid grey;" | KILL[ED]
 +|}
 + 
 + 
The following are some examples of wait conditions: The following are some examples of wait conditions:
-Example. $step3 == 0 
-Will wait for task 'step3' in the current job to finish, and when it does it will 
-return TRUE or FALSE depending whether its exit status is 0. 
-Example. $step3 == KILLED 
-Will wait for task 'step3' in the current job to finish, and when it does it will 
-return TRUE or FALSE depending whether its exit status is 10 (killed). 
-Example. $step3 == ERR 
-Will wait for task 'step3' in the current job to finish, and when it does it will 
-Numeric Exit Status Symbol 
-0 OK 
-1 ERR[OR] 
-2 WARN[ING] 
-10 KILL[ED] 
-52 Setting Up Jobs 
-return TRUE or FALSE depending whether its exit status is 1 (error). 
-Example. $job2:step3 < 2 
-Will wait for task 'step3' in job 'job2' to finish, then check its exit status is 
-less than 2. 
-Example. $job2:step3 && [ -f /tmp/in_data ] 
-Will wait for task 'step3' in job 'job2' to finish and for a file <tt>/tmp/in_data</tt> to exist. 
-A task_name (or job_name) by itself will simply wait for the task or job to terminate, then return TRUE.+:Example: <code>$step3 == 0</code>
 +::Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 0.
-Check Syntax Checks the syntax of the wait condition and notifies if the condition is OK or has an error.+:Example: <code>$step3 == KILLED</code>
 +::Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 10 (killed).
- Figure 14 — Check syntax report+:Example: <code>$step3 == ERR</code>
 +::Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 1 (error).
-On failure If the wait condition fails, i.e. evaluates to FALSE, the task will either go into assist mode.+:Example: <code>$job2:step3 < 2</code>
 +::Will wait for task 'step3' in job 'job2' to finish, then check its exit status is less than 2.
-;assist: enter assist mode on failure. The operator then decides the action.+:Example: <code>$job2:step3 && [ -f /tmp/in_data ]</code>
-;stop: stops the task on failure and doesn't run any dependencies. An+::Will wait for task 'step3' in job 'job2' to finish and for a file <tt>/tmp/in_data</tt> to exist.
-;error: status is raised in the job.+ 
 + 
 +A task_name (or job_name) by itself will simply wait for the task or job to terminate, then return TRUE.
 + 
 +[[Image:Task img 17.jpg|frame|Figure 14 — Check syntax report]]
 +;Check Syntax: Checks the syntax of the wait condition and notifies if the condition is OK or has an error.
 + 
 +;On failure: If the wait condition fails, i.e. evaluates to FALSE, the task will either go into assist mode.
 +:;'''''assist''''': enter assist mode on failure. The operator then decides the action.
 +:;'''''stop''''': stops the task on failure and doesn't run any dependencies. An
 +:;'''''error''''': status is raised in the job.
<br> <br>
 +
=== Viewing other Jobs and Subjobs === === Viewing other Jobs and Subjobs ===
Line 492: Line 536:
== How to Define a Job Queue == == How to Define a Job Queue ==
-Task3G jobs can be submitted to the 'at' queue, 'batch' queue, 'task' queue or to a+Task3G jobs can be submitted to the 'at' queue, 'batch' queue, 'task' queue or to a user defined job queue. The 'cron' queue is reserved for cron jobs, it is not available for job submission.
-user defined job queue. The 'cron' queue is reserved for cron jobs, it is not available+
-for job submission.+
-Job queues are used to stream jobs with similar characteristics. For each queue, you+Job queues are used to stream jobs with similar characteristics. For each queue, you can define a default priority and the maximum number of concurrent jobs. The queue will only release up to this number of jobs to run at the same time. Once one of these jobs finishes, the next job in that queue is started. The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.
-can define a default priority and the maximum number of concurrent jobs. The+
-queue will only release up to this number of jobs to run at the same time. Once one+
-of these jobs finishes, the next job in that queue is started. The priority determines+
-the job's relative share of the CPU; it equates to the UNIX nice value.+
-Each queue is identified by a letter, for example 'a' for the AT_queue, 'b' for the+Each queue is identified by a letter, for example 'a' for the AT_queue, 'b' for the BATCH_queue, 't' for the TASK queue. d - z are available for user-defined queues. 'c' is reserved for cron jobs.
-BATCH_queue, 't' for the TASK queue. d - z are available for user-defined queues.+{{note| Some operating systems do not support all queues d-z. Also some operating systems do not support specifying the maximum number of concurrent jobs.}}
-'c' is reserved for cron jobs.+
- +
-Note Some operating systems do not support all queues d-z. Also some+
-operating systems do not support specifying the maximum number of+
-concurrent jobs.+
- +
-Note A step by step guide to defining a queue can be found in Tutorial 1:+
-Creating a Queue on page 83.+
<br> <br>
=== Configuration Console === === Configuration Console ===
-The task3G Job Queue Details console is where all configuration concerning queues+The task3G Job Queue Details console is where all configuration concerning queues is accomplished. This is access via the task3G Jobs console. From the task3G Jobs console, select {{cnav|Tables > Queues}}.
-is accomplished. This is access via the task3G Jobs console. From the task3G Jobs+
-console, select Tables > Queues+
-To access the task3G Jobs console, see COStask Configuration on page 27.+[[Image:Task img 18.jpg|frame|Figure 15 — The task3G Job Queue Details]]
 +To access the task3G Jobs console, see [[#Task3G Configuration]].
- Figure 15 — The task3G Job Queue Details+Figure 15 shows all defined queues.
-The above screen shows all defined queues.+The {{cnav|Maintain}} pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects {{cnav|Change}}.
- +
-The Maintain pulldown menu contains the standard Add, Change, Clone and+
-Remove items which work the same way as maintaining other COSMmanager+
-tables. Generally, Double-Clicking an item automatically selects Change.+
The following is a list of actions for each menu item: The following is a list of actions for each menu item:
-Creating a new queue is done by selecting Maintain > Add.+*Creating a new queue is done by selecting {{cnav|Maintain > Add}}.
-Changing the details of a queue is done by selecting Maintain >+*Changing the details of a queue is done by selecting {{cnav|Maintain > Change}}. Double-Clicking on the task is the equivalent to {{cnav|Maintain > Change}}.
-Change. Double-Clicking on the task is the equivalent to Maintain >+*Copying a queue is done by selecting {{cnav|Maintain > Clone}}.
-Change.+*Deleting a queue is done by selecting {{cnav|Maintain > Remove}}.
-Copying a queue is done by selecting Maintain > Clone.+
-Deleting a queue is done by selecting Maintain > Remove.+
-'''To define a new queue:'''+==== To define a new queue ====
- Figure 16 — Defining a task3G queue+[[Image:Task img 19.jpg|frame|Figure 16 — Defining a task3G queue]]
- +;Queue: This is the unique name of the queue which is used as the identifier.
-Queue This is the unique name of the queue which is used as the+;Description: A description or comment about the queue.
-identifier.+;UNIX queue: The UNIX at queue name to be used with this task3G queue. You can normally use up to 25 job queues. Jobs can be submitted to the standard system queues provided, such as the at queue (a), batch queue (b) and the task queue (t). Another 22 queues may be defined with the letters, d through to z, though some operating systems may restrict this range.
-Description A description or comment about the queue.+{{note| Jobs can't be submitted to the c queue as it is reserved for use by cron.}}
-UNIX queue The UNIX at queue name to be used with this task3G queue. You+{{note| The number of queues that can be defined may be restricted in some versions of UNIX.}}
-can normally use up to 25 job queues. Jobs can be submitted to the+;Concurrent jobs: The maximum number of jobs that can run concurrently in this queue.
-standard system queues provided, such as the at queue (a), batch+;Re-scheduled time: The number of seconds between checks to determine if new jobs can be launched.
-queue (b) and the task queue (t). Another 22 queues may be+;Priority: The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.
-defined with the letters, d through to z, though some operating+{{note| Remember that the lower the number, the higher the priority. -19 is the highest priority, +19 being the lowest.}}
-systems may restrict this range.+
-Note Jobs can't be submitted to the c queue as it is reserved for use by cron.+
-Note The number of queues that can be defined may be restricted in some+
-versions of UNIX.+
-Concurrent jobs The maximum number of jobs that can run concurrently in this+
-queue.+
-Re-scheduled time+
-The number of seconds between checks to determine if new jobs+
-can be launched.+
-Priority The priority determines the job's relative share of the CPU; it+
-equates to the UNIX nice value.+
-Note Remember that the lower the number, the higher the priority. -19 is+
-the highest priority, +19 being the lowest.+
<br> <br>
 +
== Adding a Schedule == == Adding a Schedule ==
Line 575: Line 586:
The base schedule is simply a list of days and months. The prompt form for adding schedules has three fields, Day(s) of Month, Month(s) of Year, and Day(s) of Week. If all the conditions in these fields are met, then the duty is scheduled for today. For The base schedule is simply a list of days and months. The prompt form for adding schedules has three fields, Day(s) of Month, Month(s) of Year, and Day(s) of Week. If all the conditions in these fields are met, then the duty is scheduled for today. For
example, to schedule duties for the first Monday and Friday of every month, you would fill in the fields like this: example, to schedule duties for the first Monday and Friday of every month, you would fill in the fields like this:
-Day(s) of Month:1-7+ 
-Month(s) of Year:1-12+:Day(s) of Month:1-7
-Day(s) of Week:1,5+:Month(s) of Year:1-12
 +:Day(s) of Week:1,5
 + 
Leaving a field’s default value means ‘always true’, so in this example every month is included. Leaving a field’s default value means ‘always true’, so in this example every month is included.
-This base schedule can be further refined by defining ‘include’ and ‘exclude’ datelists.+ 
-Finally, you can also enter a UNIX command which must return TRUE (status 0) if the current date is in the schedule.+This base schedule can be further refined by defining ‘include’ and ‘exclude’ datelists. Finally, you can also enter a UNIX command which must return TRUE (status 0) if the current date is in the schedule.
<br> <br>
 +
=== Adding a schedule === === Adding a schedule ===
Select {{cnav|Maintain > Schedules}} from the ‘task3G jobs’ console. Select {{cnav|Maintain > Schedules}} from the ‘task3G jobs’ console.
 +
Select {{cnav|Maintain > Add}}. Select {{cnav|Maintain > Add}}.
-Order task3G allocates an order number, by default ten more than the previous highest. The order number determines the order in which the list of outstanding scheduled jobs is presented to the user. If you want to keep the new time with a group of similar schedule times, change the order to an available number close to that of the group.+[[Image:Task img 20.jpg|frame|Figure 17 — The task3G Job Queue Details]]
-Schedule name Enter a descriptive term, consistent with the names displayed in the list of schedules.+;Order: task3G allocates an order number, by default ten more than the previous highest. The order number determines the order in which the list of outstanding scheduled jobs is presented to the user. If you want to keep the new time with a group of similar schedule times, change the order to an available number close to that of the group.
 + 
 +;Schedule name: Enter a descriptive term, consistent with the names displayed in the list of schedules.
The next three fields control which days a duty will be scheduled. By default the fields return ‘true’ for every day of the month, every month of the year and every day of the week. You restrict the scheduled days by limiting one or more of the fields to certain values or ranges. The next three fields control which days a duty will be scheduled. By default the fields return ‘true’ for every day of the month, every month of the year and every day of the week. You restrict the scheduled days by limiting one or more of the fields to certain values or ranges.
-Day(s) of month Enter a list of days of then month. Example: first week of the+ 
-month: 1-7+;Day(s) of month: Enter a list of days of then month. Example: first week of the month: 1-7
-Month(s) of Year Enter a list of months. Example: first month of every quarter:+ 
-1,4,7,10+;Month(s) of year: Enter a list of months. Example: first month of every quarter: 1,4,7,10
-Day(s) of Week Enter the days of the week (0=Sunday, 6=Saturday). Example:+ 
-Monday to Wednesday and Friday: 1-3,5+;Day(s) of week: Enter the days of the week (0=Sunday, 6=Saturday). Example: Monday to Wednesday and Friday: 1-3,5
-Start time, Period Length Do not change. They are for use with other COSmanager applications.+ 
-And in list Only schedule the job for dates that are defined above AND are in the datelist selected.+;Start time, Period Length: Do not change. They are for use with other COSmanager applications.
-And not in list Only schedule the job for dates that are defined above and NOT included in the datelist selected.+ 
-Schedule command Enter a command which will return ‘true’ only if today’s date is when the job should be run.+;And in list: Only schedule the job for dates that are defined above AND are in the datelist selected.
-Use with cron Must be ‘yes’ for task3G.+ 
-Press Accept to complete the definition of the scheduled time.+;And not in list: Only schedule the job for dates that are defined above and NOT included in the datelist selected.
 + 
 +;Schedule command: Enter a command which will return ‘true’ only if today’s date is when the job should be run.
 + 
 +;Use with cron: Must be ‘yes’ for task3G.
 + 
 +Press {{cnav|Accept}} to complete the definition of the scheduled time.
<br> <br>
 +
=== Synchronising crontab === === Synchronising crontab ===

Current revision

This section describes the process of defining jobs, including:


Contents

Before you run your first job

Task3G forms the skeleton of a complete task control and batch job submission system. Your next task is to flesh out this skeleton by adding job components for the main jobs and procedures at your site. The Task3G Overview explains what job components are and how they are related.

These details are stored centrally in task3G's database, rather than being duplicated in shell-scripts and applications throughout the network. This means that generalized components need only to be entered once and can be maintained in one place.

You may not need to do all of the following steps, as some of the components and site-specific details will have been defined when task3G and COSmanager were installed.

  1. If necessary define new job queues.
  2. Define tasks for any commands that will be used in a job.
  3. Define jobs to carry out all your regular batch processing.
  4. Create dependencies within a job.


Planning your job control strategy

There is no universal scheme that is best for all sites, and task3G does not try to impose a standard strategy. However, task3G does make it simpler to automate your job submission procedures, as it lets you build jobs from reusable components. task3G also provides a convenient means of documenting your procedures at any point, as information about all your jobs is stored centrally in task3G tables.

There are two main points to consider. First, given the current system environment, how best to handle the current processing workload. Second, planning to maintain service in the face of changes in processing requirements, both expected (increase in workload) and unexpected (system failure).

Following is a list of factors that may influence your strategy:


Task3G Configuration

The configuration screen is accessed via the COSmanager menu

Figure 5 — The COSmanager buttonbar (GUI)
Figure 5 — The COSmanager buttonbar (GUI)


By clicking the Config button, a sub menu appears containing all product configuration options. Access task3G Configuration by selecting the task3G configuration option. Select Product Configuration > task3G configuration .

The task3G Job configuration console should appear. This screen shows all defined jobs as well as providing access for configuring all other aspects.


How to Define a Job

There are three types task3G jobs that can be defined. They are automatic jobs, manual jobs and subjobs.

Automatic jobs
are jobs that are run by cron. An operator is not required to submit jobs of this type. Jobs in this mode need a schedule and time(s) at which the job will run. This is defined by the When and At Time(s) field on the job prompt form.
Manual jobs
are jobs that must be submitted by an operator. These jobs can also be scheduled but are never run automatically. The manual job uses the At Time(s) and On (day) fields for scheduling. However, this is optional, manual jobs need not have scheduling details as these can be provided while submitting a job.
Subjobs
are jobs within a job. These jobs cannot be scheduled. They are to be defined in another job as a task. When it is time for this task (subjob) to run, then the tasks of this subjob will execute. Subjobs help in the sense that larger, more complex jobs can be broken into smaller, simpler steps.
Note
Note
Any job can be used as a subjob. However, jobs defined as subjobs cannot be run by themselves.


Configuration Console

The task3G Job console is where all configuration concerning jobs is accomplished. Select Config > task3G Configuration .

Display > Job report menu option displays a full report of the last run.

The Maintain pulldown menu contains the standard Add, Change, Clone and Remove options which work the same way as maintaining other COSmanager tables.

The following is a list of actions for each menu item:


Figure 6 — Configuration console, or ‘Maintain Jobs’ window
Figure 6 — Configuration console, or ‘Maintain Jobs’ window
Maintain > Tasks in job
selects the task3G Tasks in job console allowing you to view and maintain the tasks in a chosen job.
Maintain > Create Duty
for use with duty3G.
Maintain > Sync crontab
to ensure that the automatic jobs are correctly replicated in COSmanager crontab.
Tables > Tasks
selects the task3G Task details console allowing you to maintain tasks
Tables > Roles
selects the task3G Roles and access capabilities console allowing you to view an maintain task3G roles and capabilities.
Tables > Queues
selects the task3G Job queue details console allowing you to maintain the job queue.
Tables > Schedules
selects the task3G schedules console allowing you to view and maintain task3G schedules.
Tables > Datelists
selects the task3G datelists console allowing you to maintain datelists.


To Define Information About a Job

Defining a job is split into two entry forms. The first comprises of the basic options needed for each task, the latter comprises of advanced options.

Figure 7 — Prompt form to add a job
Figure 7 — Prompt form to add a job
Job
The job name identifies each individual job. This must be unique.
Description
Appears in the job schedule, so make it meaningful to help users to choose the right job.
Mode
The mode in which the job will run in:
automatic
if the job will run to a regular schedule and doesn't need to be manually submitted
manual
if the job will be submitted by the operator
subjob
if the job will only run as a task in another job.


When to initiate the job (automatic job)

The When and At time(s) fields determine the scheduling details associated with the job.

When
The frequency at which the job runs, whether its daily, weekly, monthly, every second week etc. This is used for automatic jobs only.
At time(s)
Time at which the job will start. This is used for automatic jobs and is also optional for manual jobs.
On (day)
The day in which the job will start. This is mandatory for manual jobs if a start time was specified above, i.e. the job is 'scheduled'. ‘Next 24 hrs’ (default) means that the job will run later on the day of submission, or if submitted after the ‘At time(s)’ on the following day.


The next three fields specify the access role; the host this job will run on; and the job queue it will be scheduled to.

Access role
Who can run and control this job.
Run on host
On which host will the job run.
Queue
The job queue the job will be submitted to.


Advanced options

You can configure advanced options such as job parameters, pre- and post-conditions, concurrent tasks, priority, clear status and notes. Press Advanced to enter one or more of these command or options.

Figure 8 — Prompt form to add a job (advanced features)
Figure 8 — Prompt form to add a job (advanced features)
Job parameters
Job parameters cause SHELL ENVIRONMENT variables to be set which are available to all tasks in the job. Commonly they would be included as $ variables which are passed as parameters to the commands in a task.
Fixed Parameters
Job parameters can be set to fixed values in the job configuration. They should be a space separated list in the form VARIABLE=VALUE. The given VALUE will be assigned to the VARIABLE at the start of the job, and is available as an ENVIRONMENT VARIABLE to all tasks in the job.
Prompt form
Job parameters can also be set at job submission via a PROMPT form. The full path must be specified, unless the form is in $APPL_HOME/local/prompt. As with the 'Fixed parameters' the value(s) specified in the form will be available as ENVIRONMENT VARIABLES to all tasks in the job.
Pre- and Post-processing
This consists of commands that are executed before and/or after the job has run. The pre-processing consists of the at-submit command, the wait command and the pre-condition. These are executed in the order they appear. The post-processing consists of the at-end command.
At-submit command
An operating system command that executes when the job is submitted. This may be interactive if the job is manually submitted.
Wait condition
A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a job begins execution. The job will wait till the expression is satisfied. Form more information, see #Creating Wait Conditions.
Timeout (mins)
This is the number of minutes before the wait condition times out. If the wait condition times out, the condition fails and the job is not started.
Pre-condition
A UNIX command, script or boolean expression which is executed before the job is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If the precondition fails, the job is not run. For more information, see #Creating Wait Conditions.
Check Syntax
Checks the syntax of the wait condition and pre-condition and notifies if the conditions are OK or have errors.
On failure
A job log file is created on the commencement of each job. This option specifies whether the log is kept or deleted if the wait condition or pre-condition fails.
At-end command
An operating system command that executes when the job has ended. The ‘Status’ environment variable is set to the job’s exit status prior to calling this command. If the variable is 0, the job was successful.
Clear status
When to clear the job status. Submit - every time the job is submitted. End - when the job finishes. Success - when a job has ended successfully. Never - the job status cannot be cleared other than manually from the Task monitor console, or programmatically via the FSclrstat program.
Concurrent tasks
The maximum number of tasks that can be run concurrently. If more than this number of tasks are ready to run, some will be queued. Leaving the field blank signifies that an unlimited number of tasks can be run concurrently.
Priority
At what priority level the job will run at. This takes precedence over the queue priority.
Notes file
A file that contains extra information about a particular job which the operator can view. This field must contain a full path to the file, unless it is under $APPL_HOME/local/notes.


Notification Options

Task3G can notify certain people either when this job requests operator assistance, or when it ends. For each of these, you can specify if and whom to notify, using the following fields:

Notify
When and if to notify if the job fails or requires assistance. 'never' => don't perform this type of notification for this job; 'default' => notify the default people defined in the Task3G settings; 'specify' => notify the people defined in the 'Who' field below.
Who
Space separated list of people to notify. These can be COSmanager users, email addresses or mobile phone numbers (for SMS).


Note
Note
If you specify a COSmanager user their notification method and address must be configured in the COSmanager 'Users' table
Note
Note
The mechanics of the EMAIL and SMS notification should be configured in the COSmanager 'Notification Method' table


How to Define a Task

The building block of a job is the task. In effect, a task describes the 'what' and 'whether' of one component in a job. The 'What' part identifies an operating system command or script, and the name of the host on which it will run. The 'whether' part consists of pre- and post-conditions. These are UNIX commands or scripts that are run before and after the main UNIX command or script.

Defining tasks can be accomplished from 2 areas, they are from the 'Task list' console and the 'Tasks in job' console.

Tasks defined in the 'Task list' console are placed in a pool of tasks which can be chosen when adding tasks to a job. Tasks defined in the 'Tasks in job' console are added to the job but are also placed in this pool, and so can later be added to other jobs.


Configuration Console

Figure 9 — Configuration console, or ‘Maintain Tasks’ window
Figure 9 — Configuration console, or ‘Maintain Tasks’ window

To access the 'Task list' console from the task3G configuration console select Tables > Tasks .

To access the 'Tasks in job' console from the task3G Configuration console select Maintain > Tasks in Job .

To access the 'task3G Jobs' console see #Task3G Configuration.

The Maintain pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects Change .

The following is a list of actions for each menu item:


To Define Information About a Task

Defining a task is split into two entry forms. The first comprises of the basic options needed for each task, the latter comprises of advanced options.

Figure 10 — Prompt form to add a task
Figure 10 — Prompt form to add a task
Task name
The task name identifies each individual task. This must be a unique identifier.
Command
This is the command the task executes. This can be a UNIX command or a script. Also referred to as the main command. Leave this field blank if the task is to be a subjob.
Run on host
This is the host on which the command will run. If left blank it will run on the default host of the job.
As user
Under which UNIX user the command or script will execute. This is currently ignored on Windows NT hosts.
Subjob
This is where you specify a pre-defined subjob to run within the task. The task will run the subjob, thus giving the ‘job within a job’ effect.
Comments
A description or comment about the task.


Advanced options

You can configure advanced options such as pre- and post-conditions, time-outs, recover actions and operator privileges. Press Advanced to enter one or more of these command or options.

Figure 11 — Prompt form to add a task (advanced options)
Figure 11 — Prompt form to add a task (advanced options)
Pre- and Post-processing
This consists of commands that are executed before and/or after the main command. These can be operating system commands or script.
Pre-condition
A UNIX command, script or boolean expression which is executed before the main command is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If it succeeds, the main command is run, if it fails, neither the main command nor any post-processing is done and the result is dependent on what job recovery action is specified for 'On pre-cond fail'. For more information, see #Creating Wait Conditions.
Check Syntax
Checks the syntax of the wait condition and notifies if the condition is OK or has an error.
On-success cmd
The on-success command is executed after the main command succeeds, it will not execute if it fails. If the ‘On-success’ command succeeds, the task completes successfully. If it fails, the result is dependent on what job recovery action is specified for 'On error'.
On-error cmd
The on-error command is executed after the main command fails, it will not execute if it succeeds. If the ‘On-error’ command fails, the task fails. If it succeeds, the result is dependent on what job recovery action is specified for 'On-error'.
Override status
This field only applies if an ‘On-error command’ is specified. If this field is ‘yes’ and the On-error command succeeds, the task completes successfully. This will set the completed tasks status to SUCCESSFUL even though the main command FAILED. If ‘no’ the status of the task will be set to FAIL even if On-error completes successfully.
Timeout
This is the number of minutes before the task times out. If the task executes for longer than this time, the task is killed and the result is dependent on what job recovery action is specified for 'On timeout'.


Job recovery actions

These recovery actions are triggered on command and timeout failures. There are one of four possible actions: assist, stop, continue and ignore.

assist
enter assist mode on error. The operator then decides the action.
stop
ends the task on error and don't run any of its dependencies. An error status is raised in the job.
continue
ends the task and runs the dependencies. An error status is raised in the job.
ignore
ends the task and runs the dependencies. No error status is raised in the job.
On error
This recovery action is triggered when post-processing fails. If no post-processing is specified, then the 'on error' recovery action will handle main command failures.
On pre-cond fail
This recovery action is triggered when the pre-condition fails.
On kill
This recovery action is triggered when the task is killed via the task monitor console.
On timeout
This recovery action is triggered when the task times out.


Operator Privileges

The following options allow or disallow the operator from performing certain functions on this task:

Can re-run
If the task goes into 'assist mode', this determines whether the operator can re-run the task.
Can continue
If the task goes into 'assist mode', this determines whether the operator can allow the task's dependencies to run.
Can kill
Whether the operator can kill (cancel) this task after it has started.
Notes file
A file that contains extra information about a particular task which the operator can view. This field must contain a full path to the file, unless it is under $APPL_HOME/local/notes.


Inserting Tasks into a Job

Populating a job with tasks and subjobs is done from the 'Tasks in job' console. Also from this console, dependencies and wait conditions can be defined. The console displays the number of tasks and subjobs contained within the job and its dependencies.

From the 'task3G Jobs' console, select Maintain > Tasks in job.

To access the 'task3G Jobs' console, see #Task3G Configuration.

Figure 12 — 'Tasks in Job' console
Figure 12 — 'Tasks in Job' console

The following is a list of the menus and an explanation of each:

Maintain
This menu deals with creating and adding tasks to a job as well as maintaining tasks.
Dependencies
This menu deals with creating and checking dependencies as well as creating wait conditions.
View
This menu deals with viewing jobs.


Adding Tasks to a Job and Task Maintenance

Adding tasks to a job is done via the maintain menu. Selecting Maintain > Add , opens a window containing all pre-defined tasks. From this window, selecting multiple tasks will add them to the job. If you want the new tasks to be dependent on an existing task, highlight that task before selecting Add .

The 'Tasks in job' console also allows the user to add new tasks to a job without having to create them via the 'Tasks' console. This is done by selecting Maintain > New . If you want the new tasks to be dependent on an existing task, highlight that task before selecting New .

Tasks can be changed or deleted by selecting Maintain > Change and Maintain > Remove respectively. Double-Clicking on tasks is the equivalent to Maintain > Change .

The 'Task' console can be accessed by selecting Maintain > Tasks .


Creating Dependencies

With no dependencies defined, a task3G job would start as many of its tasks in parallel as is possible. The maximum number of concurrent tasks can be defined in the job definition. Usually you want to define dependencies so that a task is not started until certain other 'parent' tasks have finished.

Dependent tasks are displayed indented, under their parent with a '+' sign aligned with their parent.

Once all tasks have been added to the job, creating dependencies is the next step. This is done using the following items found under the Dependencies menu:

Cut
By selecting one or more tasks and then selecting Dependencies > Cut , the task(s) are removed from the job and placed into a temporary buffer.
Link
By selecting one or more tasks and then selecting Dependencies > Link , the task(s) are placed into a temporary buffer. They are not removed from the job.
Paste
By selecting Dependencies > Paste , the task(s) in the buffer are added to the job. If a task was highlighted prior to pasting, the buffered task(s) are pasted under the highlighted task, making them dependent on the highlighted task. The buffer is then cleared.
Sequence
By highlighting multiple tasks and selecting Dependencies > Sequence , a sequence of dependent tasks is created. A window appears containing the selected tasks allowing you to put them in the desired order. This is achieved by 'dragging' the tasks up and down in this window. When the correct order has been achieved, push the Accept button and a sequence of dependent tasks is created.


Example 1: one task dependent on another

Task img 12.jpg
  1. Add tasks Update_db (task1) and Reports (task2) to a job. The window contents of the 'Tasks in job' console should appear as the following:
    Reports
    Update_db
  2. Next, create a dependency making task Reports dependent on task Update_db. Select Reports.
  3. Select Dependencies > Cut
  4. Select Update_db
  5. Select Dependencies > Paste . The window contents of the 'Tasks in job' console should appear as the following:
    Update_db
    + Reports
When run, task3G will start Reports once Update_db has completed.


Example 2: two tasks dependent on one

Task img 13.jpg
  1. Add tasks Update_db, Reports and Cleanup to a job. The window contents of the 'Tasks in job' console should appear as the following:
    Cleanup
    Reports
    Update_db
  2. Next, create a dependency making task Reports and Cleanup both dependent on task Update_db. Select both Reports and Cleanup.
  3. Select Dependencies > Cut
  4. Select Update_db
  5. Select Dependencies > Paste . The window contents of the 'Tasks in job' console should appear as the following:
    Update_db
    + Reports
    + Cleanup
When run, task3G will start both Reports and Cleanup once Update_db has completed.


Example 3: one task dependent on two

Task img 14.jpg
  1. Add tasks Update_db, Reports and Cleanup to a job. The window contents of the 'Tasks in job' console should appear as the following:
    Cleanup
    Reports
    Update_db
  2. Next, create a dependency making task Cleanup dependent on both Update_db and Reports tasks. Select Cleanup.
  3. Select Dependencies > Cut
  4. Select Update_db
  5. Select Dependencies > Paste
  6. Select Cleanup again.
  7. Select Dependencies > Link
  8. Select Reports
  9. Select Dependencies > Paste . The window contents of the 'Tasks in job' console should appear as the following:
    Update_db
    + Cleanup
    Reports
    + Cleanup
With the second Cleanup flagged as a link. When run, task3G will start Cleanup once both Update_db and Reports have completed.


Example 4: a sequence of dependent tasks

Task img 15.jpg
  1. Add tasks Update_db, Reports and Cleanup to a job. The window contents of the 'Tasks in job' console should appear as the following:
    Cleanup
    Reports
    Update_db
  2. Next, create a dependency making task Reports dependent on Update_db and Cleanup dependent on Reports tasks. Select all tasks.
  3. Select Dependencies > Sequence
  4. Drag Update_db above Cleanup (top of the display)
  5. Either drag Reports above Cleanup or drag Cleanup below Reports
  6. Press Accept to save this ordering of tasks. The window contents of the 'Tasks in job' console should appear as the following:
    Update_db
    + Reports
    + Cleanup
When run, task3G will run Update_db, then Reports, then Cleanup.


Checking Dependencies

When constructing complex dependencies that involve a large number of tasks, it is possible to unknowingly create deadlocks and/or unnecessary dependencies.

Deadlocks are basically cyclic errors. Deadlocks occur when two or more tasks are dependent on each other, eg. task3 is dependent on task2, which is dependent on task1, which is dependent on task3. It is impossible for task1, task2 or task3 to start.

Unnecessary dependencies are basically redundant dependencies, they are dependencies that are not needed.

These conditions will cause the offending task to appear in the monitor console colored either red or orange.

To fix these conditions, select Dependencies > Optimise .


Creating Wait Conditions

'Wait For' conditions are defined against a task in a job. To create a wait for condition, select a task then select Dependencies > Wait for .

Figure 13 — Wait for condition
Figure 13 — Wait for condition
Wait condition
A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a task begins execution. The task will wait till the expression is satisfied.


The wait condition uses the following symbols:

Symbol Meaning
¦¦ OR
&& AND
== EQUAL TO
 != NOT EQUAL TO
> GREATER THAN
< LESS THAN
>= GREATER THAN OR EQUAL TO
<= LESS THAN OR EQUAL TO
 ! NOT


Normal precedence for boolean expressions applies and brackets can be used. Primary expressions can be:

Note
Note
that if a shell command returns a non-zero status, the condition will WAIT for 60 seconds and try again. It will not return FALSE.


Task3G uses the following convention for both task and job statuses:

Numeric Exit Status Symbol
0 OK
1 ERR[OR]
2 WARN[ING]
10 KILL[ED]


The following are some examples of wait conditions:

Example: $step3 == 0
Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 0.
Example: $step3 == KILLED
Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 10 (killed).
Example: $step3 == ERR
Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 1 (error).
Example: $job2:step3 < 2
Will wait for task 'step3' in job 'job2' to finish, then check its exit status is less than 2.
Example: $job2:step3 && [ -f /tmp/in_data ]
Will wait for task 'step3' in job 'job2' to finish and for a file /tmp/in_data to exist.


A task_name (or job_name) by itself will simply wait for the task or job to terminate, then return TRUE.

Figure 14 — Check syntax report
Figure 14 — Check syntax report
Check Syntax
Checks the syntax of the wait condition and notifies if the condition is OK or has an error.
On failure
If the wait condition fails, i.e. evaluates to FALSE, the task will either go into assist mode.
assist
enter assist mode on failure. The operator then decides the action.
stop
stops the task on failure and doesn't run any dependencies. An
error
status is raised in the job.


Viewing other Jobs and Subjobs

Using the View menu items, other jobs and subjobs can be selected for viewing via the 'Tasks in job' console. The following is an explanation of each item:

Parent job
when viewing a subjob, selecting View > Parent job will view the tasks within the parent job.
Subjob
selecting a subjob and selecting View > Subjob , will view the tasks within the subjob.
Another job
selecting View > Another job will view the tasks within the next job in alphabetical order.
Show notes
selecting View > Show notes will view the task notes.


How to Define a Job Queue

Task3G jobs can be submitted to the 'at' queue, 'batch' queue, 'task' queue or to a user defined job queue. The 'cron' queue is reserved for cron jobs, it is not available for job submission.

Job queues are used to stream jobs with similar characteristics. For each queue, you can define a default priority and the maximum number of concurrent jobs. The queue will only release up to this number of jobs to run at the same time. Once one of these jobs finishes, the next job in that queue is started. The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.

Each queue is identified by a letter, for example 'a' for the AT_queue, 'b' for the BATCH_queue, 't' for the TASK queue. d - z are available for user-defined queues. 'c' is reserved for cron jobs.

Note
Note
Some operating systems do not support all queues d-z. Also some operating systems do not support specifying the maximum number of concurrent jobs.


Configuration Console

The task3G Job Queue Details console is where all configuration concerning queues is accomplished. This is access via the task3G Jobs console. From the task3G Jobs console, select Tables > Queues .

Figure 15 — The task3G Job Queue Details
Figure 15 — The task3G Job Queue Details

To access the task3G Jobs console, see #Task3G Configuration.

Figure 15 shows all defined queues.

The Maintain pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects Change .

The following is a list of actions for each menu item:


To define a new queue

Figure 16 — Defining a task3G queue
Figure 16 — Defining a task3G queue
Queue
This is the unique name of the queue which is used as the identifier.
Description
A description or comment about the queue.
UNIX queue
The UNIX at queue name to be used with this task3G queue. You can normally use up to 25 job queues. Jobs can be submitted to the standard system queues provided, such as the at queue (a), batch queue (b) and the task queue (t). Another 22 queues may be defined with the letters, d through to z, though some operating systems may restrict this range.
Note
Note
Jobs can't be submitted to the c queue as it is reserved for use by cron.
Note
Note
The number of queues that can be defined may be restricted in some versions of UNIX.
Concurrent jobs
The maximum number of jobs that can run concurrently in this queue.
Re-scheduled time
The number of seconds between checks to determine if new jobs can be launched.
Priority
The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.
Note
Note
Remember that the lower the number, the higher the priority. -19 is the highest priority, +19 being the lowest.


Adding a Schedule

Schedules define processing cycles for running jobs. There are also used by other COSmanager applications. A schedule specifies the day which a job is to be run. You choose from the list when defining an automatic job. You can add new schedules based on your own requirements to the schedules provided with task3G.


Defining schedule dates

The base schedule is simply a list of days and months. The prompt form for adding schedules has three fields, Day(s) of Month, Month(s) of Year, and Day(s) of Week. If all the conditions in these fields are met, then the duty is scheduled for today. For example, to schedule duties for the first Monday and Friday of every month, you would fill in the fields like this:

Day(s) of Month:1-7
Month(s) of Year:1-12
Day(s) of Week:1,5

Leaving a field’s default value means ‘always true’, so in this example every month is included.

This base schedule can be further refined by defining ‘include’ and ‘exclude’ datelists. Finally, you can also enter a UNIX command which must return TRUE (status 0) if the current date is in the schedule.


Adding a schedule

Select Maintain > Schedules from the ‘task3G jobs’ console.

Select Maintain > Add .

Figure 17 — The task3G Job Queue Details
Figure 17 — The task3G Job Queue Details
Order
task3G allocates an order number, by default ten more than the previous highest. The order number determines the order in which the list of outstanding scheduled jobs is presented to the user. If you want to keep the new time with a group of similar schedule times, change the order to an available number close to that of the group.
Schedule name
Enter a descriptive term, consistent with the names displayed in the list of schedules.

The next three fields control which days a duty will be scheduled. By default the fields return ‘true’ for every day of the month, every month of the year and every day of the week. You restrict the scheduled days by limiting one or more of the fields to certain values or ranges.

Day(s) of month
Enter a list of days of then month. Example: first week of the month: 1-7
Month(s) of year
Enter a list of months. Example: first month of every quarter: 1,4,7,10
Day(s) of week
Enter the days of the week (0=Sunday, 6=Saturday). Example: Monday to Wednesday and Friday: 1-3,5
Start time, Period Length
Do not change. They are for use with other COSmanager applications.
And in list
Only schedule the job for dates that are defined above AND are in the datelist selected.
And not in list
Only schedule the job for dates that are defined above and NOT included in the datelist selected.
Schedule command
Enter a command which will return ‘true’ only if today’s date is when the job should be run.
Use with cron
Must be ‘yes’ for task3G.

Press Accept to complete the definition of the scheduled time.


Synchronising crontab

To ensure that the automatic jobs are correctly replicated in COSmanager crontab, use Maintain > Sync crontab on the ‘task3G jobs’ console. Carry out this task if problems are experienced with automatic jobs in task3G which may be caused by inconsistencies between the tables.

On selecting the Sync crontab option, task3G displays a dialogue box, which indicates what is about to happen and provides the option to exit or to proceed.

Press Accept to synchronise the tables; task3G will return a message indicating the outcome of the task.

Note
Note
This function does not alter the root crontab in any way; the only table amended is the COSmanager (ie cosmos) crontab.