FS
Documentation

Backup3G/User Guide/Appendix C - Backup Methods and Drivers

From Documentation

Jump to: navigation, search

A backup method is the ‘how to’ of a backup step. It contains the command(s) that will write the backup data. backup3G comes with a number of standard backup methods and driver scripts.

These scripts use standard UNIX utilities to provide a uniform set of facilities for backing up and recovering files across a network. In effect the drivers ‘top up’ the basic facilities provided by cpio, dump, and tar.

This appendix describes:


Appendix D—Defining Backup Methods describes how to create a new backup method.


Contents

Multi-part Backup Methods: How They Work

The recommended method for multi-part backup of filesystems and lists of files and directories is ‘full cpio - MP’. The recommended backup method for multi-part backup of raw partitions and large files is ‘image copy - MP’.

Multi-part backup methods based on FSmcpio call a program FSbusplit to split the backup. FSbusplit uses the size of the file or filesystem and the part size and capacity of the backup medium to calculate how many parts and volumes to split the backup over. FSbusplit uses an algorithm that tries to minimize the number of partitions needed.

By default, FSmcpio can back up files larger than the part size but smaller than the capacity of the volume. FSmcpio simply writes the file to a single larger than usual part. Files larger than the volume’s capacity are rejected with a warning message. To reject files larger than the part size, specify the -B flag to FSmcpio in the Options field of the backup item.

The multi-part methods automatically issue a “change media” request after writing the last part on each volume. The method used to change volumes depends on the logical drive definition. If the last partition on the previous volume was incomplete it is rewritten in full at the start of the new volume.

Although it isn’t mandatory, it is best if multi-part backup steps start writing at the beginning of a volume (backup3G assumes this when it does its capacity calculations). If the multi-part step starts half-way through a volume, it’s likely to write up to the physical end-of-volume, and so the last part will have to be restarted on the next volume.

To sum up, if a multi-part step is not the first in a job to write data, you should insert a ‘change media’ backup step before it to force backup3G to start writing at the start of a new volume.

Each of the multi-part backup methods has a corresponding ‘single-part’ method.

For example:

‘full cpio’
writes a backup step in cpio format to a single tape file or part.
‘full cpio - MP’
writes a backup step in cpio format, if necessary in more than one part or over more than one volume.


We recommend that you use a multi-part backup method where possible, particularly where the directory being backed up:


For backups that are guaranteed not to exceed the capacity of a single volume, the advantage of the single-part methods is that they will start up marginally faster.


Capacity and Part Size

You can tune the Multi-part size and Capacity parameters for a media type to make your backups run more efficiently. The following principles should be used as a guide.


If the Multi-part size field is left blank, then the Capacity field will be used in its place. That is, each volume will contain one whole partition.

If both Capacity and Multi-part size are defined, backup3G calculates how many partitions can fit on the output volume, and requests a media change after it has written the last full partition or if an I/O error (End of Media) occurs.

If Capacity is not defined, it defaults to unlimited capacity. This means that partitions will be written to the volume until an I/O error occurs. backup3G assumes this means that end-of-volume has been reached, though it could also be due to some other cause such as a physical defect on the tape.



Caution
By default, FSmcpio will back up a file larger than the part size. However, if Capacity is null, the partitioning algorithm can’t reject files larger than the real capacity of the volume. The backup step will keep requesting a new volume, trying to write the file in full, until it runs out of scratch volumes or is cancelled.
To avoid this, either exclude such files from the backup; set a realistic value for capacity; or specify the -B flag in the Options field.


Passing Options to a Backup Method

Each backup item includes an Options field, with which you can pass optional flags and arguments to the backup command as command line parameters.

For example, the cpio backup methods supplied with backup3G accept several flags that change the way backups are done. Some of these can be specified through fields in the backup item. One example is the Index field. This passes the -I flag that instructs FScpio to create an online index. Other flags have no field set aside for them in the ‘Backup item’ prompt form. Some examples:

-s <files>  back up this list of files
-x <files>  exclude from the backup this list of files
-d          perform a deleting backup
-z          use the alternate index method ls_index


Example: exclude unwanted files from backup

You have a backup item to backup the filesystem /user2 on host mama using the backup method ‘full cpio - MP’. You want to exclude core and object files and certain temporary files of the form tmpxxxx. To exclude these files from the backup, you would:

  1. Select Maintain backup jobs from the backup3G configuration menu.
  2. Select Items > Maintain.
  3. Select the backup item from the list displayed.
  4. Select Maintain > Change.
  5. In the Options field, enter -x "*/*.o */core */tmp*"
Figure 45 — Passing arguments to the backup method

These options will be appended to the cpio command at run time.



Note
Backup3G does not validate any of the options when you create or change the backup item. If any of the options are unrecognized or the syntax is incorrect, the backup step will fail when the backup job tries to execute the command.


You can specify more than one flag. The only limit is the length of the Options field.

If you have added your own backup methods to the set provided with backup3G, use Options to take advantage of any flags and features that are recognized by your backup command but aren’t built in to the backup item prompt form.


Backup3G Driver Scripts

Backup3G includes several backup drivers. These drivers use the standard utilities cpio, dump, and tar, and provide a uniform set of facilities to backup and recover files across a network and to create online indexes. In effect the drivers ‘top up’ the basic facilities provided by cpio, dump, and tar to a common level.

Another driver, FSimage, performs an image copy of a raw disk partition or very large file.

Table 10 shows what features are supported by each driver. Refer to the manual page for more information. Generally we recommend that you use a backup method based on FSmcpio. For particular uses other drivers may be more suitable. See below for notes on working with particular FS backup drivers.


Table 10 — Attributes of FS backup drivers
Driver Data Format Create Index When the Backup is Created? Create Index From an Existing Backup? Remote Backup? Multi-part Backup? Include/Exclude Selected Files
FScpio cpio
FSmcpio cpio
FSimage image1
FSmimage image
FSdump dump2 3
FStar tar
FSfilesys filesystem4


1writes a 1 KB header followed by a byte-by-byte copy of the data.
2dump is not available under AIX or SCO.
3FSdump uses the ls_index method to create the index when the backup first starts.
4copies the specified files/directories to another directory using cpio -p.


FScpio

FScpio can’t back up a read-only filesystem

FScpio is not suitable for backing up a read-only filesystem.

There are two problems. First, cpio will not be able to reset the access times of the files. Second, the .FSbackup file cannot be created as FScpio cannot write to the filesystem.

To back up a read-only filesystem you should use a different backup method, one not based on FScpio.

Backing up symbolic links with FScpio

The FScpio -y flag allows backup of data pointed to by symbolic links, whereas normally only the links themselves are backed up.

FScpio uses find(1) to gather the list of files to be backed up. The -y flag tells find to follow symbolic links.

Note that if the file being pointed to is in another filesystem, you must also use the -m flag (‘cross mount points’) to allow FScpio to back up the file.


FSdump

FSdump backup methods and online indexes

backup3G generates an online index by extracting directory information about the files being backed up, using options on the underlying backup command. For example, both cpio and tar have flags -tv that list file details. The backup drivers FScpio and FStar use these flags (with a little help from sed) to generate output similar to that produced by ls -l.

However, the BSD dump command doesn’t have any options to produce equivalent file information. FSdump gets around this by doing an ls command just before the backup. However, there are a few points you should be aware of:

If this is a problem, the safest method is to mount the filesystem as readonly before running the backup, then re-mount it as read-write afterwards. Alternatively, you can define backup methods that will perform the mounts.

FSimage

Recovering a disk partition using FSimage

Note that there can be a potential problem on some systems when restoring an image backup:

If the size of the disk partition is not a multiple of the tape block size, some tape drives will pad out the incomplete last block. When the partition is restored the last block is written in full back to disk. The result is that the restored partition is larger than the original. Most operating systems handle this correctly, but success cannot be guaranteed for all tape drives and all systems.

The image copy methods based on FSimage can write a 1 KB header. This is used in the recovery to ensure that the file, when recovered, has exactly the same size, owner, group, and permissions. backup3G writes the header by default; we recommend that you accept this behavior to avoid the padding problem.

If you choose not to have backup3G write a header, another solution is to make sure that the size of the disk partition is a multiple of the tape block size. For example, with a 200 MB disk partition (204,800 KB), possible tape block sizes are:

Block size No. of blocks
32 KB 6400 Correct
20 KB 10240 Correct
27 KB 7585.2 INCORRECT

Appendix D — Defining Backup Methods

If you wish to keep using your own backup software or a third-party product, you can define the commands as a new backup method.

This appendix describes how to define a new backup method, and lists the variables maintained by backup3G that you can use to customize how a backup method works.

How to Define Backup Methods

Each step in a backup job names a data object and the method that will be used to back it up. A method can also implement a related step that doesn’t write data. Any command or script that could be run from the UNIX prompt can be used in a backup method.

backup3G maintains several variables that you can use in a backup command — see Shell Variables in Backup Commands.

Here is the definition of one common method, ‘full cpio’.

Format
cpio
Backup Command
FScpio -f -I \"$Indexfile\" $Directory
Description
Full backup of directory using cpio

FScpio is a driver (supplied with backup3G) which writes the data in cpio format. The flags -f and -I mean that a full backup is to be done and an online index will be created. ‘full cpio’ is a convenient and descriptive way to refer to this command for use with many different backup jobs. If the command format or site requirements ever change, only the method needs to be changed in one place. Every backup job that uses that method will automatically be updated.

Note The FS backup drivers provide a consistent front-end to the underlying archiving commands. 
They do not write data in a proprietary format. 
For example, FScpio uses the standard cpio command and writes data in standard cpio format.

To define a backup method

  1. Select backup3G configuration > Maintain tables
  2. Select the ‘Backup Method’ table, then select Table > Maintain to see the backup methods that have already been defined.
  3. Select Maintain > Add, and enter the following fields:
Backup method
Enter a descriptive name. Include some attributes of the method,such as the name of the command or format, whether it will do full or incremental backups, and whether it supports multi-part backups. Example: incr cpio - MP
Description
Ideally this is an expansion of the method name. This description will help other users when choosing a backup method from the list of available methods.
Format
Select a backup format. This determines the recovery method. Leave Format blank if you are defining a non-backup method (a method that doesn’t write data).
Backup command
Enter the command that you want this method to execute.
Note Any double-quotes (") in the backup command must be protected by a backslash character, that is \". 
If this is not done the command will be incorrectly parsed by the shell.

If you left the Format field blank, the next four fields are set to default values. If you selected a backup format, the information in these three fields helps backup3G to process the backup command.

Multipart support?
Select ‘yes’ if your method supports multipart backups. (See $APPL_HOME/bin/FSmcpio for an example of how to

implement support for multi-part backups and online indexes.)

Uses STDOUT?
Select ‘yes’ if your backup command writes to standard output(stdout). Select ‘no’ if it writes directly to the device. If you select ‘yes’, backup3G uses devio(1) to read stdout and write the data to the device, using the block size specified for this media type in the media table.
Index support?
Select ‘yes’ if your method supports online indexes. Generally this will be ‘yes’ if you use one of the standard backup drivers (FScpio, FSfilesys, FSdump or FStar), otherwise ‘no’.
Remote support?
Select ‘yes’ if your method supports backup and recovery on a remote host. This doesn’t mean that all backups using your method must be run remotely. It just tells backup3G to give users the option when defining a backup.
If Uses STDOUT? is ‘yes’, the method automatically supports remote backup.
Run on host
Governs on which host the method will run. For remote backups, select ‘drive-host’ if the method only accesses the backup drive without writing data, otherwise use ‘file-host’.

Select Accept to save the changes and return to the methods table.


Shell Variables in Backup Commands

Table 11 lists the variables maintained by backup3G that you can refer to in your backup command. All the variables that are data amounts (Blocksize, Capacity, and Part_size) are in kilobytes (KB). The syntax for referring to variables is similar to shell syntax: $<variablename>.

Note You can use the fields status and maxstatus for dependency tests.

Table 11 — Variables available to be used in backup commands

Variable Description
maxstatus The highest exit status of the backup steps run so far.
status The exit status from the previous backup step.
Blocksize The block size of the backup volume.
Capacity The data storage capacity of the backup volume.
Comments Comments from the backup item relating to current backup step.
Days The number of days to retain the backup volume.
Density The density of the backup volume.
Device The rewind-on-close device file for the backup drive.
Device_nr The ‘no rewind-on-close’ device file for the backup drive.
Directory The directory which is to be backed up.
Drive The logical backup drive name.
Drivehost The host to which the backup drive is connected.
Drivename The physical backup drive name.
File The file number on volume being written or about to be written.
Format The format in which the backup is written.
Host The host on which the data lives (the file host).
Indexfile The file name for the online index. NULL means ‘do not generate an index’.
Item The backup item number referred to by the current backup step.
Job Backup job name.
Length The length of the backup volume.
Method The backup method used by the backup item.
Mode The backup job mode (scheduled, automatic or at-request).
Number The media number of the volume currently being written to.
Part_size The maximum size of each part in a multi-part backup.
Pool Group of logical drives
Queuename Name of the physical drive or drive pool to which the job is queued. Once the job starts running, Queuename = Drivename

Retention The retention period for the backup media.

Seq The sequence number of this volume in the media set produced by this job.
Step Backup step number.
Time The time of day that the backup was scheduled to run.
Type The backup media type.