Task3G/User Guide/Setting Up Jobs

This page was last modified 04:38, 30 May 2013.

From Documentation

< Task3G | User Guide(Difference between revisions)

Jump to: navigation, search

Current revision

This section describes the process of defining jobs, including:

Jobs
Tasks
Queues
Dependencies
Roles and Access Capabilities

1 Before you run your first job
2 Planning your job control strategy
3 Task3G Configuration
4 How to Define a Job
- 4.1 Configuration Console
- 4.2 To Define Information About a Job
5 How to Define a Task
- 5.1 Configuration Console
- 5.2 To Define Information About a Task
6 Inserting Tasks into a Job
7 How to Define a Job Queue
- 7.1 Configuration Console
  - 7.1.1 To define a new queue
8 Adding a Schedule

[edit]

Before you run your first job

Task3G forms the skeleton of a complete task control and batch job submission system. Your next task is to flesh out this skeleton by adding job components for the main jobs and procedures at your site. The Task3G Overview explains what job components are and how they are related.

These details are stored centrally in task3G's database, rather than being duplicated in shell-scripts and applications throughout the network. This means that generalized components need only to be entered once and can be maintained in one place.

You may not need to do all of the following steps, as some of the components and site-specific details will have been defined when task3G and COSmanager were installed.

If necessary define new job queues.
Define tasks for any commands that will be used in a job.
Define jobs to carry out all your regular batch processing.
Create dependencies within a job.

[edit]

Planning your job control strategy

There is no universal scheme that is best for all sites, and task3G does not try to impose a standard strategy. However, task3G does make it simpler to automate your job submission procedures, as it lets you build jobs from reusable components. task3G also provides a convenient means of documenting your procedures at any point, as information about all your jobs is stored centrally in task3G tables.

There are two main points to consider. First, given the current system environment, how best to handle the current processing workload. Second, planning to maintain service in the face of changes in processing requirements, both expected (increase in workload) and unexpected (system failure).

Following is a list of factors that may influence your strategy:

What is the relative priority of interactive users and background processing?
How heavily is the system loaded? Are there cyclic peaks in demand, and if so when? What is the average load and maximum load? How is system load projected to change in future?
Does your site have performance measures or service guarantees that set targets in terms of system availability, terminal response time, or job turnaround?
What is the cost to the organisation if some jobs fail - e.g. database update transactions lost or delayed, delays in processing accounts or payroll, financial reports not distributed.
Are there procedures that should be followed to recover from a major system failure, and are staff aware of them? For example, how and when should interrupted jobs rerun?
Your current and future network configuration - e.g. which jobs will be run centrally and which on distributed hosts?
What resources are available? Are there limitations in terms of disk capacity, I/O and network throughput, tape drives, etc.?
Business requirements - e.g. need for round-the-clock availability, daily/weekly/monthly/yearly processing cycles. What happens to scheduled jobs on public holidays?
Operations environment - e.g. shift coverage. How much time is available each day to process background jobs?

[edit]

Task3G Configuration

The configuration screen is accessed via the COSmanager menu

Figure 5 — The COSmanager buttonbar (GUI)

By clicking the Config button, a sub menu appears containing all product configuration options. Access task3G Configuration by selecting the task3G configuration option. Select Product Configuration > task3G configuration .

The task3G Job configuration console should appear. This screen shows all defined jobs as well as providing access for configuring all other aspects.

[edit]

How to Define a Job

There are three types task3G jobs that can be defined. They are automatic jobs, manual jobs and subjobs.

Automatic jobs: are jobs that are run by cron. An operator is not required to submit jobs of this type. Jobs in this mode need a schedule and time(s) at which the job will run. This is defined by the When and At Time(s) field on the job prompt form.
Manual jobs: are jobs that must be submitted by an operator. These jobs can also be scheduled but are never run automatically. The manual job uses the At Time(s) and On (day) fields for scheduling. However, this is optional, manual jobs need not have scheduling details as these can be provided while submitting a job.
Subjobs: are jobs within a job. These jobs cannot be scheduled. They are to be defined in another job as a task. When it is time for this task (subjob) to run, then the tasks of this subjob will execute. Subjobs help in the sense that larger, more complex jobs can be broken into smaller, simpler steps.

	Note
	Any job can be used as a subjob. However, jobs defined as subjobs cannot be run by themselves.

[edit]

Configuration Console

The task3G Job console is where all configuration concerning jobs is accomplished. Select Config > task3G Configuration .

Display > Job report menu option displays a full report of the last run.

The Maintain pulldown menu contains the standard Add, Change, Clone and Remove options which work the same way as maintaining other COSmanager tables.

The following is a list of actions for each menu item:

Creating a new job is done by selecting Maintain > Add .
Changing the details of a job is done by selecting Maintain > Change .
Copying a job is done by selecting Maintain > Clone .
Deleting a job is done by selecting Maintain > Remove .

Figure 6 — Configuration console, or ‘Maintain Jobs’ window

Maintain > Tasks in job: selects the task3G Tasks in job console allowing you to view and maintain the tasks in a chosen job.

Maintain > Create Duty: for use with duty3G.

Maintain > Sync crontab: to ensure that the automatic jobs are correctly replicated in COSmanager crontab.

Tables > Tasks: selects the task3G Task details console allowing you to maintain tasks

Tables > Roles: selects the task3G Roles and access capabilities console allowing you to view an maintain task3G roles and capabilities.

Tables > Queues: selects the task3G Job queue details console allowing you to maintain the job queue.

Tables > Schedules: selects the task3G schedules console allowing you to view and maintain task3G schedules.

Tables > Datelists: selects the task3G datelists console allowing you to maintain datelists.

[edit]

To Define Information About a Job

Defining a job is split into two entry forms. The first comprises of the basic options needed for each task, the latter comprises of advanced options.

Figure 7 — Prompt form to add a job

Job: The job name identifies each individual job. This must be unique.

Description: Appears in the job schedule, so make it meaningful to help users to choose the right job.

Mode

The mode in which the job will run in:

automatic: if the job will run to a regular schedule and doesn't need to be manually submitted
manual: if the job will be submitted by the operator
subjob: if the job will only run as a task in another job.

[edit]

When to initiate the job (automatic job)

The When and At time(s) fields determine the scheduling details associated with the job.

When: The frequency at which the job runs, whether its daily, weekly, monthly, every second week etc. This is used for automatic jobs only.

At time(s): Time at which the job will start. This is used for automatic jobs and is also optional for manual jobs.

On (day): The day in which the job will start. This is mandatory for manual jobs if a start time was specified above, i.e. the job is 'scheduled'. ‘Next 24 hrs’ (default) means that the job will run later on the day of submission, or if submitted after the ‘At time(s)’ on the following day.

The next three fields specify the access role; the host this job will run on; and the job queue it will be scheduled to.

Access role: Who can run and control this job.
Run on host: On which host will the job run.
Queue: The job queue the job will be submitted to.

[edit]

Advanced options

You can configure advanced options such as job parameters, pre- and post-conditions, concurrent tasks, priority, clear status and notes. Press Advanced to enter one or more of these command or options.

Figure 8 — Prompt form to add a job (advanced features)

Job parameters: Job parameters cause SHELL ENVIRONMENT variables to be set which are available to all tasks in the job. Commonly they would be included as $ variables which are passed as parameters to the commands in a task.

Fixed Parameters: Job parameters can be set to fixed values in the job configuration. They should be a space separated list in the form VARIABLE=VALUE. The given VALUE will be assigned to the VARIABLE at the start of the job, and is available as an ENVIRONMENT VARIABLE to all tasks in the job.

Prompt form: Job parameters can also be set at job submission via a PROMPT form. The full path must be specified, unless the form is in $APPL_HOME/local/prompt. As with the 'Fixed parameters' the value(s) specified in the form will be available as ENVIRONMENT VARIABLES to all tasks in the job.

Pre- and Post-processing: This consists of commands that are executed before and/or after the job has run. The pre-processing consists of the at-submit command, the wait command and the pre-condition. These are executed in the order they appear. The post-processing consists of the at-end command.

At-submit command: An operating system command that executes when the job is submitted. This may be interactive if the job is manually submitted.

Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a job begins execution. The job will wait till the expression is satisfied. Form more information, see #Creating Wait Conditions.

Timeout (mins): This is the number of minutes before the wait condition times out. If the wait condition times out, the condition fails and the job is not started.

Pre-condition: A UNIX command, script or boolean expression which is executed before the job is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If the precondition fails, the job is not run. For more information, see #Creating Wait Conditions.

Check Syntax: Checks the syntax of the wait condition and pre-condition and notifies if the conditions are OK or have errors.

On failure: A job log file is created on the commencement of each job. This option specifies whether the log is kept or deleted if the wait condition or pre-condition fails.

At-end command: An operating system command that executes when the job has ended. The ‘Status’ environment variable is set to the job’s exit status prior to calling this command. If the variable is 0, the job was successful.

Clear status: When to clear the job status. Submit - every time the job is submitted. End - when the job finishes. Success - when a job has ended successfully. Never - the job status cannot be cleared other than manually from the Task monitor console, or programmatically via the FSclrstat program.

Concurrent tasks: The maximum number of tasks that can be run concurrently. If more than this number of tasks are ready to run, some will be queued. Leaving the field blank signifies that an unlimited number of tasks can be run concurrently.

Priority: At what priority level the job will run at. This takes precedence over the queue priority.

Notes file: A file that contains extra information about a particular job which the operator can view. This field must contain a full path to the file, unless it is under $APPL_HOME/local/notes.

[edit]

Notification Options

Task3G can notify certain people either when this job requests operator assistance, or when it ends. For each of these, you can specify if and whom to notify, using the following fields:

Notify: When and if to notify if the job fails or requires assistance. 'never' => don't perform this type of notification for this job; 'default' => notify the default people defined in the Task3G settings; 'specify' => notify the people defined in the 'Who' field below.
Who: Space separated list of people to notify. These can be COSmanager users, email addresses or mobile phone numbers (for SMS).

	Note
	If you specify a COSmanager user their notification method and address must be configured in the COSmanager 'Users' table

	Note
	The mechanics of the EMAIL and SMS notification should be configured in the COSmanager 'Notification Method' table

[edit]

How to Define a Task

The building block of a job is the task. In effect, a task describes the 'what' and 'whether' of one component in a job. The 'What' part identifies an operating system command or script, and the name of the host on which it will run. The 'whether' part consists of pre- and post-conditions. These are UNIX commands or scripts that are run before and after the main UNIX command or script.

Defining tasks can be accomplished from 2 areas, they are from the 'Task list' console and the 'Tasks in job' console.

Tasks defined in the 'Task list' console are placed in a pool of tasks which can be chosen when adding tasks to a job. Tasks defined in the 'Tasks in job' console are added to the job but are also placed in this pool, and so can later be added to other jobs.

[edit]

Configuration Console

Figure 9 — Configuration console, or ‘Maintain Tasks’ window

To access the 'Task list' console from the task3G configuration console select Tables > Tasks .

To access the 'Tasks in job' console from the task3G Configuration console select Maintain > Tasks in Job .

To access the 'task3G Jobs' console see #Task3G Configuration.

The Maintain pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects Change .

The following is a list of actions for each menu item:

Creating a new task is done by selecting Maintain > Add .
Changing the details of a task is done by selecting Maintain > Change . Double-Clicking on the task is the equivalent to Maintain > Change .
Copying a task is done by selecting Maintain > Clone .
Deleting a task is done by selecting Maintain > Remove .

[edit]

To Define Information About a Task

Defining a task is split into two entry forms. The first comprises of the basic options needed for each task, the latter comprises of advanced options.

Figure 10 — Prompt form to add a task

Task name: The task name identifies each individual task. This must be a unique identifier.

Command: This is the command the task executes. This can be a UNIX command or a script. Also referred to as the main command. Leave this field blank if the task is to be a subjob.

Run on host: This is the host on which the command will run. If left blank it will run on the default host of the job.

As user: Under which UNIX user the command or script will execute. This is currently ignored on Windows NT hosts.

Subjob: This is where you specify a pre-defined subjob to run within the task. The task will run the subjob, thus giving the ‘job within a job’ effect.

Comments: A description or comment about the task.

[edit]

Advanced options

You can configure advanced options such as pre- and post-conditions, time-outs, recover actions and operator privileges. Press Advanced to enter one or more of these command or options.

Figure 11 — Prompt form to add a task (advanced options)

Pre- and Post-processing: This consists of commands that are executed before and/or after the main command. These can be operating system commands or script.

Pre-condition: A UNIX command, script or boolean expression which is executed before the main command is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If it succeeds, the main command is run, if it fails, neither the main command nor any post-processing is done and the result is dependent on what job recovery action is specified for 'On pre-cond fail'. For more information, see #Creating Wait Conditions.

Check Syntax: Checks the syntax of the wait condition and notifies if the condition is OK or has an error.

On-success cmd: The on-success command is executed after the main command succeeds, it will not execute if it fails. If the ‘On-success’ command succeeds, the task completes successfully. If it fails, the result is dependent on what job recovery action is specified for 'On error'.

On-error cmd: The on-error command is executed after the main command fails, it will not execute if it succeeds. If the ‘On-error’ command fails, the task fails. If it succeeds, the result is dependent on what job recovery action is specified for 'On-error'.

Override status: This field only applies if an ‘On-error command’ is specified. If this field is ‘yes’ and the On-error command succeeds, the task completes successfully. This will set the completed tasks status to SUCCESSFUL even though the main command FAILED. If ‘no’ the status of the task will be set to FAIL even if On-error completes successfully.

Timeout: This is the number of minutes before the task times out. If the task executes for longer than this time, the task is killed and the result is dependent on what job recovery action is specified for 'On timeout'.

[edit]

Job recovery actions

These recovery actions are triggered on command and timeout failures. There are one of four possible actions: assist, stop, continue and ignore.

assist: enter assist mode on error. The operator then decides the action.
stop: ends the task on error and don't run any of its dependencies. An error status is raised in the job.
continue: ends the task and runs the dependencies. An error status is raised in the job.
ignore: ends the task and runs the dependencies. No error status is raised in the job.

On error: This recovery action is triggered when post-processing fails. If no post-processing is specified, then the 'on error' recovery action will handle main command failures.

On pre-cond fail: This recovery action is triggered when the pre-condition fails.

On kill: This recovery action is triggered when the task is killed via the task monitor console.

On timeout: This recovery action is triggered when the task times out.

[edit]

Operator Privileges

The following options allow or disallow the operator from performing certain functions on this task:

Can re-run: If the task goes into 'assist mode', this determines whether the operator can re-run the task.

Can continue: If the task goes into 'assist mode', this determines whether the operator can allow the task's dependencies to run.

Can kill: Whether the operator can kill (cancel) this task after it has started.

Notes file: A file that contains extra information about a particular task which the operator can view. This field must contain a full path to the file, unless it is under $APPL_HOME/local/notes.

[edit]

Inserting Tasks into a Job

Populating a job with tasks and subjobs is done from the 'Tasks in job' console. Also from this console, dependencies and wait conditions can be defined. The console displays the number of tasks and subjobs contained within the job and its dependencies.

From the 'task3G Jobs' console, select Maintain > Tasks in job.

To access the 'task3G Jobs' console, see #Task3G Configuration.

Figure 12 — 'Tasks in Job' console

The following is a list of the menus and an explanation of each:

Maintain: This menu deals with creating and adding tasks to a job as well as maintaining tasks.

Dependencies: This menu deals with creating and checking dependencies as well as creating wait conditions.

View: This menu deals with viewing jobs.

[edit]

Adding Tasks to a Job and Task Maintenance

Adding tasks to a job is done via the maintain menu. Selecting Maintain > Add , opens a window containing all pre-defined tasks. From this window, selecting multiple tasks will add them to the job. If you want the new tasks to be dependent on an existing task, highlight that task before selecting Add .

The 'Tasks in job' console also allows the user to add new tasks to a job without having to create them via the 'Tasks' console. This is done by selecting Maintain > New . If you want the new tasks to be dependent on an existing task, highlight that task before selecting New .

Tasks can be changed or deleted by selecting Maintain > Change and Maintain > Remove respectively. Double-Clicking on tasks is the equivalent to Maintain > Change .

The 'Task' console can be accessed by selecting Maintain > Tasks .

[edit]

Creating Dependencies

With no dependencies defined, a task3G job would start as many of its tasks in parallel as is possible. The maximum number of concurrent tasks can be defined in the job definition. Usually you want to define dependencies so that a task is not started until certain other 'parent' tasks have finished.

Dependent tasks are displayed indented, under their parent with a '+' sign aligned with their parent.

Once all tasks have been added to the job, creating dependencies is the next step. This is done using the following items found under the Dependencies menu:

Cut: By selecting one or more tasks and then selecting Dependencies > Cut , the task(s) are removed from the job and placed into a temporary buffer.

Link: By selecting one or more tasks and then selecting Dependencies > Link , the task(s) are placed into a temporary buffer. They are not removed from the job.

Paste: By selecting Dependencies > Paste , the task(s) in the buffer are added to the job. If a task was highlighted prior to pasting, the buffered task(s) are pasted under the highlighted task, making them dependent on the highlighted task. The buffer is then cleared.

Sequence: By highlighting multiple tasks and selecting Dependencies > Sequence , a sequence of dependent tasks is created. A window appears containing the selected tasks allowing you to put them in the desired order. This is achieved by 'dragging' the tasks up and down in this window. When the correct order has been achieved, push the Accept button and a sequence of dependent tasks is created.

Example 1: one task dependent on another

Add tasks Update_db (task1) and Reports (task2) to a job. The window contents of the 'Tasks in job' console should appear as the following:
Reports
Update_db
Next, create a dependency making task Reports dependent on task Update_db. Select Reports.
Select Dependencies > Cut
Select Update_db
Select Dependencies > Paste . The window contents of the 'Tasks in job' console should appear as the following:
Update_db
+ Reports

When run, task3G will start Reports once Update_db has completed.

Example 2: two tasks dependent on one

Add tasks Update_db, Reports and Cleanup to a job. The window contents of the 'Tasks in job' console should appear as the following:
Cleanup
Reports
Update_db
Next, create a dependency making task Reports and Cleanup both dependent on task Update_db. Select both Reports and Cleanup.
Select Dependencies > Cut
Select Update_db
Select Dependencies > Paste . The window contents of the 'Tasks in job' console should appear as the following:
Update_db
+ Reports
+ Cleanup

When run, task3G will start both Reports and Cleanup once Update_db has completed.

Example 3: one task dependent on two

Add tasks Update_db, Reports and Cleanup to a job. The window contents of the 'Tasks in job' console should appear as the following:
Cleanup
Reports
Update_db
Next, create a dependency making task Cleanup dependent on both Update_db and Reports tasks. Select Cleanup.
Select Dependencies > Cut
Select Update_db
Select Dependencies > Paste
Select Cleanup again.
Select Dependencies > Link
Select Reports
Select Dependencies > Paste . The window contents of the 'Tasks in job' console should appear as the following:
Update_db
+ Cleanup
Reports
+ Cleanup

With the second Cleanup flagged as a link. When run, task3G will start Cleanup once both Update_db and Reports have completed.

Example 4: a sequence of dependent tasks

Add tasks Update_db, Reports and Cleanup to a job. The window contents of the 'Tasks in job' console should appear as the following:
Cleanup
Reports
Update_db
Next, create a dependency making task Reports dependent on Update_db and Cleanup dependent on Reports tasks. Select all tasks.
Select Dependencies > Sequence
Drag Update_db above Cleanup (top of the display)
Either drag Reports above Cleanup or drag Cleanup below Reports
Press Accept to save this ordering of tasks. The window contents of the 'Tasks in job' console should appear as the following:
Update_db
+ Reports
+ Cleanup

When run, task3G will run Update_db, then Reports, then Cleanup.

[edit]

Checking Dependencies

When constructing complex dependencies that involve a large number of tasks, it is possible to unknowingly create deadlocks and/or unnecessary dependencies.

Deadlocks are basically cyclic errors. Deadlocks occur when two or more tasks are dependent on each other, eg. task3 is dependent on task2, which is dependent on task1, which is dependent on task3. It is impossible for task1, task2 or task3 to start.

Unnecessary dependencies are basically redundant dependencies, they are dependencies that are not needed.

These conditions will cause the offending task to appear in the monitor console colored either red or orange.

To fix these conditions, select Dependencies > Optimise .

[edit]

Creating Wait Conditions

'Wait For' conditions are defined against a task in a job. To create a wait for condition, select a task then select Dependencies > Wait for .

Figure 13 — Wait for condition

Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a task begins execution. The task will wait till the expression is satisfied.

The wait condition uses the following symbols:

Symbol	Meaning
¦¦	OR
&&	AND
==	EQUAL TO
!=	NOT EQUAL TO
>	GREATER THAN
<	LESS THAN
>=	GREATER THAN OR EQUAL TO
<=	LESS THAN OR EQUAL TO
!	NOT

Normal precedence for boolean expressions applies and brackets can be used. Primary expressions can be:

A job or task name of the form
$name where 'name' is a task in the current job.
$job where 'job' is another task3G job.
$job:name where 'name' is a task in job 'job'.

You can extend the above form by testing for a particular exit status, using the numeric operators listed above.
Example: $step3 == 5
This expression will wait for the task or job to finish, then either return TRUE or FALSE.

An command (including a shell test [ ] command). This will be run periodically until it returns TRUE (zero exit status).

	Note
	that if a shell command returns a non-zero status, the condition will WAIT for 60 seconds and try again. It will not return FALSE.

Task3G uses the following convention for both task and job statuses:

Numeric Exit Status	Symbol
0	OK
1	ERR[OR]
2	WARN[ING]
10	KILL[ED]

The following are some examples of wait conditions:

Example: $step3 == 0

Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 0.

Example: $step3 == KILLED

Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 10 (killed).

Example: $step3 == ERR

Will wait for task 'step3' in the current job to finish, and when it does it will return TRUE or FALSE depending whether its exit status is 1 (error).

Example: $job2:step3 < 2

Will wait for task 'step3' in job 'job2' to finish, then check its exit status is less than 2.

Example: $job2:step3 && [ -f /tmp/in_data ]

Will wait for task 'step3' in job 'job2' to finish and for a file /tmp/in_data to exist.

A task_name (or job_name) by itself will simply wait for the task or job to terminate, then return TRUE.

Figure 14 — Check syntax report

Check Syntax: Checks the syntax of the wait condition and notifies if the condition is OK or has an error.

On failure

If the wait condition fails, i.e. evaluates to FALSE, the task will either go into assist mode.

assist: enter assist mode on failure. The operator then decides the action.
stop: stops the task on failure and doesn't run any dependencies. An
error: status is raised in the job.

[edit]

Viewing other Jobs and Subjobs

Using the View menu items, other jobs and subjobs can be selected for viewing via the 'Tasks in job' console. The following is an explanation of each item:

Parent job: when viewing a subjob, selecting View > Parent job will view the tasks within the parent job.
Subjob: selecting a subjob and selecting View > Subjob , will view the tasks within the subjob.
Another job: selecting View > Another job will view the tasks within the next job in alphabetical order.
Show notes: selecting View > Show notes will view the task notes.

[edit]

How to Define a Job Queue

Task3G jobs can be submitted to the 'at' queue, 'batch' queue, 'task' queue or to a user defined job queue. The 'cron' queue is reserved for cron jobs, it is not available for job submission.

Job queues are used to stream jobs with similar characteristics. For each queue, you can define a default priority and the maximum number of concurrent jobs. The queue will only release up to this number of jobs to run at the same time. Once one of these jobs finishes, the next job in that queue is started. The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.

Each queue is identified by a letter, for example 'a' for the AT_queue, 'b' for the BATCH_queue, 't' for the TASK queue. d - z are available for user-defined queues. 'c' is reserved for cron jobs.

	Note
	Some operating systems do not support all queues d-z. Also some operating systems do not support specifying the maximum number of concurrent jobs.

[edit]

Configuration Console

The task3G Job Queue Details console is where all configuration concerning queues is accomplished. This is access via the task3G Jobs console. From the task3G Jobs console, select Tables > Queues .

Figure 15 — The task3G Job Queue Details

To access the task3G Jobs console, see #Task3G Configuration.

Figure 15 shows all defined queues.

The following is a list of actions for each menu item:

Creating a new queue is done by selecting Maintain > Add .
Changing the details of a queue is done by selecting Maintain > Change . Double-Clicking on the task is the equivalent to Maintain > Change .
Copying a queue is done by selecting Maintain > Clone .
Deleting a queue is done by selecting Maintain > Remove .

[edit]

To define a new queue

Figure 16 — Defining a task3G queue

Queue: This is the unique name of the queue which is used as the identifier.
Description: A description or comment about the queue.
UNIX queue: The UNIX at queue name to be used with this task3G queue. You can normally use up to 25 job queues. Jobs can be submitted to the standard system queues provided, such as the at queue (a), batch queue (b) and the task queue (t). Another 22 queues may be defined with the letters, d through to z, though some operating systems may restrict this range.

	Note
	Jobs can't be submitted to the c queue as it is reserved for use by cron.

	Note
	The number of queues that can be defined may be restricted in some versions of UNIX.

Concurrent jobs: The maximum number of jobs that can run concurrently in this queue.
Re-scheduled time: The number of seconds between checks to determine if new jobs can be launched.
Priority: The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.

	Note
	Remember that the lower the number, the higher the priority. -19 is the highest priority, +19 being the lowest.

[edit]

Adding a Schedule

Schedules define processing cycles for running jobs. There are also used by other COSmanager applications. A schedule specifies the day which a job is to be run. You choose from the list when defining an automatic job. You can add new schedules based on your own requirements to the schedules provided with task3G.

[edit]

Defining schedule dates

The base schedule is simply a list of days and months. The prompt form for adding schedules has three fields, Day(s) of Month, Month(s) of Year, and Day(s) of Week. If all the conditions in these fields are met, then the duty is scheduled for today. For example, to schedule duties for the first Monday and Friday of every month, you would fill in the fields like this:

Day(s) of Month:1-7

Month(s) of Year:1-12

Day(s) of Week:1,5

Leaving a field’s default value means ‘always true’, so in this example every month is included.

This base schedule can be further refined by defining ‘include’ and ‘exclude’ datelists. Finally, you can also enter a UNIX command which must return TRUE (status 0) if the current date is in the schedule.

[edit]

Adding a schedule

Select Maintain > Schedules from the ‘task3G jobs’ console.

Select Maintain > Add .

Figure 17 — The task3G Job Queue Details

Order: task3G allocates an order number, by default ten more than the previous highest. The order number determines the order in which the list of outstanding scheduled jobs is presented to the user. If you want to keep the new time with a group of similar schedule times, change the order to an available number close to that of the group.

Schedule name: Enter a descriptive term, consistent with the names displayed in the list of schedules.

The next three fields control which days a duty will be scheduled. By default the fields return ‘true’ for every day of the month, every month of the year and every day of the week. You restrict the scheduled days by limiting one or more of the fields to certain values or ranges.

Day(s) of month: Enter a list of days of then month. Example: first week of the month: 1-7

Month(s) of year: Enter a list of months. Example: first month of every quarter: 1,4,7,10

Day(s) of week: Enter the days of the week (0=Sunday, 6=Saturday). Example: Monday to Wednesday and Friday: 1-3,5

Start time, Period Length: Do not change. They are for use with other COSmanager applications.

And in list: Only schedule the job for dates that are defined above AND are in the datelist selected.

And not in list: Only schedule the job for dates that are defined above and NOT included in the datelist selected.

Schedule command: Enter a command which will return ‘true’ only if today’s date is when the job should be run.

Use with cron: Must be ‘yes’ for task3G.

Press Accept to complete the definition of the scheduled time.

[edit]

Synchronising crontab

To ensure that the automatic jobs are correctly replicated in COSmanager crontab, use Maintain > Sync crontab on the ‘task3G jobs’ console. Carry out this task if problems are experienced with automatic jobs in task3G which may be caused by inconsistencies between the tables.

On selecting the Sync crontab option, task3G displays a dialogue box, which indicates what is about to happen and provides the option to exit or to proceed.

Press Accept to synchronise the tables; task3G will return a message indicating the outcome of the task.

	Note
	This function does not alter the root crontab in any way; the only table amended is the COSmanager (ie cosmos) crontab.

Retrieved from "https://documentation.fs.com.au/index.php/Task3G/User_Guide/Setting_Up_Jobs"

Search

Task3G/User Guide/Setting Up Jobs

From Documentation

Current revision

Contents

Before you run your first job

Planning your job control strategy

Task3G Configuration

How to Define a Job

Configuration Console

To Define Information About a Job

When to initiate the job (automatic job)

Advanced options

Notification Options

How to Define a Task

Configuration Console

To Define Information About a Task

Advanced options

Job recovery actions

Operator Privileges

Inserting Tasks into a Job

Adding Tasks to a Job and Task Maintenance

Creating Dependencies

Checking Dependencies

Creating Wait Conditions

Viewing other Jobs and Subjobs

How to Define a Job Queue

Configuration Console

To define a new queue

Adding a Schedule

Defining schedule dates

Adding a schedule

Synchronising crontab

 === Before you run your first job ===
-Task3G forms the skeleton of a complete task control and batch job submission system. Your next task is to flesh out this skeleton by adding job components for the main jobs and procedures at your site. The Overview on page 15, explains what job
+Task3G forms the skeleton of a complete task control and batch job submission system. Your next task is to flesh out this skeleton by adding job components for the main jobs and procedures at your site. The [[Task3G/User Guide/Overview|Task3G Overview]] explains what job components are and how they are related.
-components are and how they are related.
 These details are stored centrally in task3G's database, rather than being duplicated in shell-scripts and applications throughout the network. This means that generalized components need only to be entered once and can be maintained in one place.
 <br>
 === Planning your job control strategy ===
 ;Subjobs: are jobs within a job. These jobs cannot be scheduled. They are to be defined in another job as a task. When it is time for this task (subjob) to run, then the tasks of this subjob will execute. Subjobs help in the sense that larger, more complex jobs can be broken into smaller, simpler steps.
 {{note| Any job can be used as a subjob. However, jobs defined as subjobs cannot be run by themselves.}}
-{{note| A step by step guide to defining a job can be found in [[Tutorial 2: Creating a Job]].}}
-{{note| A step by step guide to defining a subjob can be found in [[Tutorial 3: Creating a Subjob]].}}
 <br>
 ;Job parameters: Job parameters cause SHELL ENVIRONMENT variables to be set which are available to all tasks in the job. Commonly they would be included as $ variables which are passed as parameters to the commands in a task.
-;Fixed Parameters: Job parameters can be set to fixed values in the job configuration
+;Fixed Parameters: Job parameters can be set to fixed values in the job configuration. They should be a space separated list in the form VARIABLE=VALUE. The given VALUE will be assigned to the VARIABLE at the start of the job, and is available as an ENVIRONMENT VARIABLE to all tasks in the job.
-;Prompt form: Job parameters can be set at job submission via a PROMPT form. The full path must be specified, unless the form is in <tt>$APPL_HOME/local/prompt</tt>
+;Prompt form: Job parameters can also be set at job submission via a PROMPT form. The full path must be specified, unless the form is in <tt>$APPL_HOME/local/prompt</tt>. As with the 'Fixed parameters' the value(s) specified in the form will be available as ENVIRONMENT VARIABLES to all tasks in the job.
 ;Pre- and Post-processing: This consists of commands that are executed before and/or after the job has run. The pre-processing consists of the at-submit command, the wait command and the pre-condition. These are executed in the order they appear. The post-processing consists of the at-end command.
 ;At-submit command: An operating system command that executes when the job is submitted. This may be interactive if the job is manually submitted.
-;Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a job begins execution. The job will wait till the expression is satisfied. Form more information, see Creating Wait Conditions on page 49.
+;Wait condition: A boolean expression which can test the exit status of a job, exit status of a task or the return code from an arbitrary UNIX command or script. If defined, this expression must be satisfied (TRUE) before a job begins execution. The job will wait till the expression is satisfied. Form more information, see [[#Creating Wait Conditions]].
 ;Timeout (mins): This is the number of minutes before the wait condition times out. If the wait condition times out, the condition fails and the job is not started.
-;Pre-condition: A UNIX command, script or boolean expression which is executed before the job is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If the precondition fails, the job is not run. For more information, see Creating Wait Conditions on page 49.
+;Pre-condition: A UNIX command, script or boolean expression which is executed before the job is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If the precondition fails, the job is not run. For more information, see [[#Creating Wait Conditions]].
 ;Check Syntax: Checks the syntax of the wait condition and pre-condition and notifies if the conditions are OK or have errors.
 ;Notes file: A file that contains extra information about a particular job which the operator can view. This field must contain a full path to the file, unless it is under <tt>$APPL_HOME/local/notes</tt>.
+<br>
+==== Notification Options ====
+Task3G can notify certain people either when this job requests operator assistance, or when it ends. For each of these, you can specify if and whom to notify, using the following fields:
+;Notify: When and if to notify if the job fails or requires assistance. 'never' => don't perform this type of notification for this job; 'default' => notify the default people defined in the Task3G settings; 'specify' => notify the people defined in the 'Who' field below.
+;Who: Space separated list of people to notify. These can be COSmanager users, email addresses or mobile phone numbers (for SMS).
+{{note| If you specify a COSmanager user their notification method and address must be configured in the COSmanager 'Users' table}}
+{{note| The mechanics of the EMAIL and SMS notification should be configured in the COSmanager 'Notification Method' table}}
 <br>
 Tasks defined in the 'Task list' console are placed in a pool of tasks which can be chosen when adding tasks to a job. Tasks defined in the 'Tasks in job' console are added to the job but are also placed in this pool, and so can later be added to other jobs.
+<br>
-'''Configuration Console'''
+==== Configuration Console ====
 [[Image:Task img 8.jpg|frame|Figure 9 — Configuration console, or ‘Maintain Tasks’ window]]
 To access the 'Tasks in job' console from the task3G Configuration console select {{cnav|Maintain > Tasks in Job}}.
-To access the 'task3G Jobs' console see COStask Configuration on page 27.
+To access the 'task3G Jobs' console see [[#Task3G Configuration]].
-{{note| A step by step guide to defining a queue can be found in Tutorial 4: Creating a Task on page 88.}}
 The {{cnav|Maintain}} pulldown menu contains the standard Add, Change, Clone and Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects {{cnav|Change}}.
 <br>
 === To Define Information About a Task ===
 ;Comments: A description or comment about the task.
+<br>
-'''Advanced options'''
+==== Advanced options ====
 You can configure advanced options such as pre- and post-conditions, time-outs, recover actions and operator privileges. Press {{cnav|Advanced}} to enter one or more of these command or options.
 ;Pre- and Post-processing: This consists of commands that are executed before and/or after the main command. These can be operating system commands or script.
-;Pre-condition: A UNIX command, script or boolean expression which is executed before the main command is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If it succeeds, the main command is run, if it fails, neither the main command nor any post-processing is done and the result is dependent on what job recovery action is specified for 'On pre-cond fail'. For more information, see Creating Wait Conditions on page 49.
+;Pre-condition: A UNIX command, script or boolean expression which is executed before the main command is run. The boolean expression is identical in syntax as the wait condition. However, a pre-condition returns simply SUCCESS or FAIL immediately, it does not wait. If it succeeds, the main command is run, if it fails, neither the main command nor any post-processing is done and the result is dependent on what job recovery action is specified for 'On pre-cond fail'. For more information, see [[#Creating Wait Conditions]].
 ;Check Syntax: Checks the syntax of the wait condition and notifies if the condition is OK or has an error.
 ;On-error cmd: The on-error command is executed after the main command fails, it will not execute if it succeeds. If the ‘On-error’ command fails, the task fails. If it succeeds, the result is dependent on what job recovery action is specified for 'On-error'.
-;Override: status This field only applies if an ‘On-error command’ is specified. If this field is ‘yes’ and the On-error command succeeds, the task completes successfully. This will set the completed tasks status to SUCCESSFUL even though the main command FAILED. If ‘no’ the status of the task will be set to FAIL even if On-error completes successfully.
+;Override status: This field only applies if an ‘On-error command’ is specified. If this field is ‘yes’ and the On-error command succeeds, the task completes successfully. This will set the completed tasks status to SUCCESSFUL even though the main command FAILED. If ‘no’ the status of the task will be set to FAIL even if On-error completes successfully.
 ;Timeout: This is the number of minutes before the task times out. If the task executes for longer than this time, the task is killed and the result is dependent on what job recovery action is specified for 'On timeout'.
+<br>
-'''Job recovery actions'''
+==== Job recovery actions ====
 These recovery actions are triggered on command and timeout failures. There are one of four possible actions: assist, stop, continue and ignore.
 ;On timeout: This recovery action is triggered when the task times out.
+<br>
-'''Operator Privileges'''
+==== Operator Privileges ====
 The following options allow or disallow the operator from performing certain functions on this task:
 From the 'task3G Jobs' console, select Maintain > Tasks in job.
-To access the 'task3G Jobs' console, see COStask Configuration on page 27.
+To access the 'task3G Jobs' console, see [[#Task3G Configuration]].
 [[Image:Task img 11.jpg|frame|Figure 12 — 'Tasks in Job' console]]
 'Wait For' conditions are defined against a task in a job. To create a wait for condition, select a task then select {{cnav|Dependencies > Wait for}}.
-{{note| A step by step guide to defining a wait condition can be found in Tutorial 6: Creating Wait Conditions on page 94.}}
 [[Image:Task img 16.jpg|frame|Figure 13 — Wait for condition]]
-task3G uses the following convention for both task and job statuses:
+Task3G uses the following convention for both task and job statuses:
-Numeric Exit Status Symbol
+:{| border="0" cellpadding="3" cellspacing="0"
-OK
+! align="left" width="150" style="border-bottom:1px solid grey;border-right:1px solid grey;" | Numeric Exit Status
-ERR[OR]
+! align="left" width="150" style="border-bottom:1px solid grey;" | Symbol
-WARN[ING]
+|-
-KILL[ED]
+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 0
+| style="border-bottom:1px solid grey;" | OK
+|-
+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 1
+| style="border-bottom:1px solid grey;" | ERR[OR]
+|-
+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 2
+| style="border-bottom:1px solid grey;" | WARN[ING]
+|-
+| valign="top" style="border-bottom:1px solid grey;border-right:1px solid grey;" | 10
+| style="border-bottom:1px solid grey;" | KILL[ED]
+|}
 == How to Define a Job Queue ==
-Task3G jobs can be submitted to the 'at' queue, 'batch' queue, 'task' queue or to a
+Task3G jobs can be submitted to the 'at' queue, 'batch' queue, 'task' queue or to a user defined job queue. The 'cron' queue is reserved for cron jobs, it is not available for job submission.
-user defined job queue. The 'cron' queue is reserved for cron jobs, it is not available
-for job submission.
-Job queues are used to stream jobs with similar characteristics. For each queue, you
+Job queues are used to stream jobs with similar characteristics. For each queue, you can define a default priority and the maximum number of concurrent jobs. The queue will only release up to this number of jobs to run at the same time. Once one of these jobs finishes, the next job in that queue is started. The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.
-can define a default priority and the maximum number of concurrent jobs. The
-queue will only release up to this number of jobs to run at the same time. Once one
-of these jobs finishes, the next job in that queue is started. The priority determines
-the job's relative share of the CPU; it equates to the UNIX nice value.
-Each queue is identified by a letter, for example 'a' for the AT_queue, 'b' for the
+Each queue is identified by a letter, for example 'a' for the AT_queue, 'b' for the BATCH_queue, 't' for the TASK queue. d - z are available for user-defined queues. 'c' is reserved for cron jobs.
-BATCH_queue, 't' for the TASK queue. d - z are available for user-defined queues.
+{{note| Some operating systems do not support all queues d-z. Also some operating systems do not support specifying the maximum number of concurrent jobs.}}
-'c' is reserved for cron jobs.
-Note Some operating systems do not support all queues d-z. Also some
-operating systems do not support specifying the maximum number of
-concurrent jobs.
-Note A step by step guide to defining a queue can be found in Tutorial 1:
-Creating a Queue on page 83.
 <br>
 === Configuration Console ===
-The task3G Job Queue Details console is where all configuration concerning queues
+The task3G Job Queue Details console is where all configuration concerning queues is accomplished. This is access via the task3G Jobs console. From the task3G Jobs console, select {{cnav|Tables > Queues}}.
-is accomplished. This is access via the task3G Jobs console. From the task3G Jobs
-console, select Tables > Queues
-To access the task3G Jobs console, see COStask Configuration on page 27.
+[[Image:Task img 18.jpg|frame|Figure 15 — The task3G Job Queue Details]]
+To access the task3G Jobs console, see [[#Task3G Configuration]].
- Figure 15 — The task3G Job Queue Details
+Figure 15 shows all defined queues.
-The above screen shows all defined queues.
+The {{cnav|Maintain}} pulldown menu contains the standard Add, Change, Clone and  Remove items which work the same way as maintaining other COSmanager tables. Generally, Double-Clicking an item automatically selects {{cnav|Change}}.
-The Maintain pulldown menu contains the standard Add, Change, Clone and
-Remove items which work the same way as maintaining other COSMmanager
-tables. Generally, Double-Clicking an item automatically selects Change.
 The following is a list of actions for each menu item:
-• Creating a new queue is done by selecting Maintain > Add.
+*Creating a new queue is done by selecting {{cnav|Maintain > Add}}.
-• Changing the details of a queue is done by selecting Maintain >
+*Changing the details of a queue is done by selecting {{cnav|Maintain > Change}}. Double-Clicking on the task is the equivalent to {{cnav|Maintain > Change}}.
-Change. Double-Clicking on the task is the equivalent to Maintain >
+*Copying a queue is done by selecting {{cnav|Maintain > Clone}}.
-Change.
+*Deleting a queue is done by selecting {{cnav|Maintain > Remove}}.
-• Copying a queue is done by selecting Maintain > Clone.
-• Deleting a queue is done by selecting Maintain > Remove.
-'''To define a new queue:'''
+==== To define a new queue ====
- Figure 16 — Defining a task3G queue
+[[Image:Task img 19.jpg|frame|Figure 16 — Defining a task3G queue]]
+;Queue: This is the unique name of the queue which is used as the identifier.
-Queue This is the unique name of the queue which is used as the
+;Description: A description or comment about the queue.
-identifier.
+;UNIX queue: The UNIX at queue name to be used with this task3G queue. You can normally use up to 25 job queues. Jobs can be submitted to the standard system queues provided, such as the at queue (a), batch queue (b) and the task queue (t). Another 22 queues may be defined with the letters, d through to z, though some operating systems may restrict this range.
-Description A description or comment about the queue.
+{{note| Jobs can't be submitted to the c queue as it is reserved for use by cron.}}
-UNIX queue The UNIX at queue name to be used with this task3G queue. You
+{{note| The number of queues that can be defined may be restricted in some versions of UNIX.}}
-can normally use up to 25 job queues. Jobs can be submitted to the
+;Concurrent jobs: The maximum number of jobs that can run concurrently in this queue.
-standard system queues provided, such as the at queue (a), batch
+;Re-scheduled time: The number of seconds between checks to determine if new jobs can be launched.
-queue (b) and the task queue (t). Another 22 queues may be
+;Priority: The priority determines the job's relative share of the CPU; it equates to the UNIX nice value.
-defined with the letters, d through to z, though some operating
+{{note| Remember that the lower the number, the higher the priority. -19 is the highest priority, +19 being the lowest.}}
-systems may restrict this range.
-Note Jobs can't be submitted to the c queue as it is reserved for use by cron.
-Note The number of queues that can be defined may be restricted in some
-versions of UNIX.
-Concurrent jobs The maximum number of jobs that can run concurrently in this
-queue.
-Re-scheduled time
-The number of seconds between checks to determine if new jobs
-can be launched.
-Priority The priority determines the job's relative share of the CPU; it
-equates to the UNIX nice value.
-Note Remember that the lower the number, the higher the priority. -19 is
-the highest priority, +19 being the lowest.
 <br>
 == Adding a Schedule ==
 The base schedule is simply a list of days and months. The prompt form for adding schedules has three fields, Day(s) of Month, Month(s) of Year, and Day(s) of Week. If all the conditions in these fields are met, then the duty is scheduled for today. For
 example, to schedule duties for the first Monday and Friday of every month, you would fill in the fields like this:
-Day(s) of Month:1-7
-Month(s) of Year:1-12
+:Day(s) of Month:1-7
-Day(s) of Week:1,5
+:Month(s) of Year:1-12
+:Day(s) of Week:1,5
 Leaving a field’s default value means ‘always true’, so in this example every month is included.
-This base schedule can be further refined by defining ‘include’ and ‘exclude’ datelists.
-Finally, you can also enter a UNIX command which must return TRUE (status 0) if the current date is in the schedule.
+This base schedule can be further refined by defining ‘include’ and ‘exclude’ datelists. Finally, you can also enter a UNIX command which must return TRUE (status 0) if the current date is in the schedule.
 <br>
 === Adding a schedule ===
 Select {{cnav|Maintain > Schedules}} from the ‘task3G jobs’ console.
 Select {{cnav|Maintain > Add}}.
-Order task3G allocates an order number, by default ten more than the previous highest. The order number determines the order in which the list of outstanding scheduled jobs is presented to the user. If you want to keep the new time with a group of similar schedule times, change the order to an available number close to that of the group.
+[[Image:Task img 20.jpg|frame|Figure 17 — The task3G Job Queue Details]]
-Schedule name Enter a descriptive term, consistent with the names displayed in the list of schedules.
+;Order: task3G allocates an order number, by default ten more than the previous highest. The order number determines the order in which the list of outstanding scheduled jobs is presented to the user. If you want to keep the new time with a group of similar schedule times, change the order to an available number close to that of the group.
+;Schedule name: Enter a descriptive term, consistent with the names displayed in the list of schedules.
 The next three fields control which days a duty will be scheduled. By default the fields return ‘true’ for every day of the month, every month of the year and every day of the week. You restrict the scheduled days by limiting one or more of the fields to certain values or ranges.
-Day(s) of month Enter a list of days of then month. Example: first week of the
-month: 1-7
+;Day(s) of month: Enter a list of days of then month. Example: first week of the month: 1-7
-Month(s) of Year Enter a list of months. Example: first month of every quarter:
-,4,7,10
+;Month(s) of year: Enter a list of months. Example: first month of every quarter: 1,4,7,10
-Day(s) of Week Enter the days of the week (0=Sunday, 6=Saturday). Example:
-Monday to Wednesday and Friday: 1-3,5
+;Day(s) of week: Enter the days of the week (0=Sunday, 6=Saturday). Example: Monday to Wednesday and Friday: 1-3,5
-Start time, Period Length Do not change. They are for use with other COSmanager applications.
-And in list Only schedule the job for dates that are defined above AND are in the datelist selected.
+;Start time, Period Length: Do not change. They are for use with other COSmanager applications.
-And not in list Only schedule the job for dates that are defined above and NOT included in the datelist selected.
-Schedule command Enter a command which will return ‘true’ only if today’s date is when the job should be run.
+;And in list: Only schedule the job for dates that are defined above AND are in the datelist selected.
-Use with cron Must be ‘yes’ for task3G.
-Press Accept to complete the definition of the scheduled time.
+;And not in list: Only schedule the job for dates that are defined above and NOT included in the datelist selected.
+;Schedule command: Enter a command which will return ‘true’ only if today’s date is when the job should be run.
+;Use with cron: Must be ‘yes’ for task3G.
+Press {{cnav|Accept}} to complete the definition of the scheduled time.
 <br>
 === Synchronising crontab ===