Backup3G/User Guide/Media
This page was last modified 10:38, 6 March 2008.From Documentation
| Revision as of 08:17, 20 April 2006 Moff (Talk | contribs) (→Scratch Media) ← Previous diff |
Current revision Moff (Talk | contribs) (Backup3G 5.1/User Guide/Media moved to Backup3G/User Guide/Media) |
||
| Line 127: | Line 127: | ||
| #Define a special media type, with the same details as the media normally used for this backup job. Name it (for example) exabyte-dly | #Define a special media type, with the same details as the media normally used for this backup job. Name it (for example) exabyte-dly | ||
| #Add enough media to complete exactly one cycle—e.g. for a twelve-month cycle add twelve tapes of type exabyte-yrly.<br>Label each tape electronically with the media number, and physically with a sticky label showing the media number and month—for example: | #Add enough media to complete exactly one cycle—e.g. for a twelve-month cycle add twelve tapes of type exabyte-yrly.<br>Label each tape electronically with the media number, and physically with a sticky label showing the media number and month—for example: | ||
| - | 501 January | + | #:501 January |
| - | 502 February | + | #:502 February |
| - | 503 March | + | #:503 March |
| - | … | + | #:… |
| #In the backup job, select this media type. Choose a retention period so that each volume expires the day before it is due to be reused. | #In the backup job, select this media type. Choose a retention period so that each volume expires the day before it is due to be reused. | ||
| Line 136: | Line 136: | ||
| <br> | <br> | ||
| + | |||
| === Media Life Cycle === | === Media Life Cycle === | ||
| + | |||
| A typical backup tape might go through the following stages. | A typical backup tape might go through the following stages. | ||
| Line 150: | Line 152: | ||
| *When a backup job requests a tape, backup3G selects from the scratch pool a tape of the required type which has the oldest ‘date last written’, and prompts the user to load it. | *When a backup job requests a tape, backup3G selects from the scratch pool a tape of the required type which has the oldest ‘date last written’, and prompts the user to load it. | ||
| - | *When the backup job finishes successfully the usage type for each tape in the media set is changed to ‘backup’, and the ‘Description’ field is set to the backup job description. backup3G calculates an expiry date and stores it in | + | *When the backup job finishes successfully the usage type for each tape in the media set is changed to ‘backup’, and the ‘Description’ field is set to the backup job description. backup3G calculates an expiry date and stores it in the media table. The name of the base directory for each backup step is stored in the media contents table. |
| - | the media table. The name of the base directory for each backup step is stored in the media contents table. | + | |
| *The user is prompted to unload the tape, usually as part of a daily scheduled duty to acknowledge completion of all overnight jobs. | *The user is prompted to unload the tape, usually as part of a daily scheduled duty to acknowledge completion of all overnight jobs. | ||
| Line 168: | Line 169: | ||
| <br> | <br> | ||
| + | |||
| == Media Support Tables == | == Media Support Tables == | ||
| Line 213: | Line 215: | ||
| '''Media label type''' | '''Media label type''' | ||
| - | Media should be electronically labeled with the media number. This helps to avoid volumes containing live data from being accidentally overwritten. backup3G supports several label formats: | + | Media should be electronically labeled with the media number. This helps to avoid volumes containing live data from being accidentally overwritten. Backup3G supports several label formats: |
| - | COSstdtape writes a short file containing the media number and ‘date last written’ | + | {| border="1" cellpadding="8" cellspacing="0" |
| - | to the start of a tape, and an end-of-data marker after the last | + | |COSstdtape |
| - | file on the tape. This is recommended for tapes that will be used | + | |writes a short file containing the media number and ‘date last written’ to the start of a tape, and an end-of-data marker after the last file on the tape. This is recommended for tapes that will be used for appending backups. |
| - | for appending backups. | + | |- |
| - | COSstdBOT writes the media number and ‘date last written’ to the start of a | + | |COSstdBOT |
| - | tape, but doesn’t write an end-of-data marker. | + | |writes the media number and ‘date last written’ to the start of a tape, but doesn’t write an end-of-data marker. |
| - | end_label writes the label to the end of a diskette. This prevents the label | + | |- |
| - | being overwritten when a new backup is done. | + | |end_label |
| - | sV_filesys System V filesystem labels | + | |writes the label to the end of a diskette. This prevents the label being overwritten when a new backup is done. |
| + | |- | ||
| + | |sV_filesys | ||
| + | |System V filesystem labels | ||
| + | |} | ||
| <br> | <br> | ||
Current revision
An essential element of the backup and recovery mechanism is effective media management. This section explains some media concepts and provides examples of good practice. Managing Removable Media on page 115 describes how to manage and use media through backup3G.
Backup3G’s facilities include:
- a comprehensive media database with support for any media type
- flexible retention periods and balancing of media usage
- tracking of media movement on- and offsite
- advanced search facilities
- media usage reports and volume labeling.
- Note
- References to tapes and tape drives in backup3G and in this manual generally apply to any type of removable media.
Contents |
Media Database
The media database is kept on your backup3G master host, and stores information about all your removable media – both used and unused – and the data they contain. The main components of the database are:
- media details
- information about all the media which can be used for backup and recovery—e.g. media type, usage, location
- media contents
- information about data stored on the media, such as online indexes and lists of base directories that have been backed up
- media support tables
- information that describes how media are used in your organization—e.g. media types, locations, label types.
Figure 9 — Media contents listing
The database is set up when you first configure the backup3G master host (see Add Media to the Media Library on page 148). These details are automatically updated as each backup is run and when you use the options in the Media Management module.
Media Management Concepts
Removable media are used in backup3G to store data for backups and other purposes. Typically, high-capacity tapes are used (e.g. SDLT, LTO, DLT), but low capacity legacy tapes (e.g. Exabyte, DAT), diskettes, and optical media can equally well be used.
Electronic labels
Media are identified by a unique number. Tapes can be electronically labeled with their media number to prevent live data being accidentally overwritten.
Media tracking
You can use the media management module to track details of your media over their whole life-cycle. For example, you can enter details of a new batch of tapes, display media contents, search through online indexes for backed-up files, and record when media are moved between locations.
Media can be stored in several locations, for example different tape libraries and offsite storage areas. Jukeboxes and silos are also types of media location.
Media usage
Backup tapes have a retention period which is defined in the backup job. Once the expiry date has passed, the data is not deleted, but the tape becomes available again. Unused and expired tapes are called scratch tapes, and the set of media available to be used is called the scratch pool.
Backup3G tries to balance media usage, by selecting the least-recently used scratch volume.
How backup3G Tapes are Structured
Figure 10 shows the layout of a backup tape created by backup3G. The first file is a short (1 KB) file containing the electronic label. Subsequent files contain the backup data, followed by an end-of-data marker.
| File # | Backup of | Method/Format |
|---|---|---|
| 2 | / (root) | Full cpio |
| 3 | /opt | Full cpio |
| 4 | /var | Full cpio |
| 5 | $APPL_HOME/db | Full cpio |
- Figure 10 — Sample layout of a backup3G tape
Media Set
A multi-volume backup is written to a set of media. A media set is identified by a ‘Set ID’, which is the media number of the first volume in the set. Each volume also has a ‘Sequence Number’, which shows its position in the set.
Example: a backup job writes to three tapes, whose media numbers are 2012, 2038 and 2001. The media set is identified as follows:
| Media No. | Set ID | Sequence No. |
|---|---|---|
| 2012 | 2012 | 1 |
| 2038 | 2012 | 2 |
| 2001 | 2012 | 3 |
Scratch Media
Scratch media are tapes and disks that are available to be written by backup jobs. A tape may be included in the scratch pool for two reasons.
- It has a usage type of ‘scratch’ in the media database.
- The data it contains has passed its expiry date.
A volume is determined to be scratch according to information held about it in the media database, not by any mark or record on the tape or diskette itself. Until they are overwritten, scratch or expired tapes still contain the last files written to them.
A media set can also be manually scratched via the Maintain > Scratch option in the Media module. When you do this, each volume in the set is marked as ‘scratch’ and the link between them (the Set ID) is removed. Note that backup3G’s record of the contents of that volume is also lost. If you might need to search for or recover files from old backups, it’s best to leave them as expired and not to mark them as scratch.
A volume with a null expiry date never expires; this equates to a retention period of ‘forever’. To reuse a volume like this you must manually mark it as ‘scratch’ as described above.
- Caution
- Don’t try to force a volume to expire by editing its expiry date. This is not a safe method of scratching media, because the volume may be part of a media set containing active data, and if there is an online index it will not be removed.
Media Balancing and Backup Cycling
Some sites use a few tapes to do all their regular backups. For example, each week they may cycle through a set of tapes physically labeled ‘Monday’, ‘Tuesday’ and so on. The disadvantage of this scheme is that media containing vital backup data are used the most heavily and are subject to a higher risk of wear and damage. Exabyte tapes, for instance, carry a manufacturers recommended maximum of 40 uses.
Instead, backup3G balances tape usage by prompting the operator to load from the scratch pool the tape with the earliest ‘date last written’.
The scratch pool method has other advantages over tape cycling.
- Designing your backup strategy is much simpler – you don’t have to worry about allocating certain tapes to certain backups
- You can support any number of tapes in your library, whereas cycling forces you to use a set number of tapes
- You can use the same system for doing irregular or one-off archives as for scheduled backups
- Flexibility – it’s much easier to change your backup strategy after implementation.
However, should you wish to, you can still implement cycling of backup media within backup3G as follows.
- Define a special media type, with the same details as the media normally used for this backup job. Name it (for example) exabyte-dly
- Add enough media to complete exactly one cycle—e.g. for a twelve-month cycle add twelve tapes of type exabyte-yrly.
Label each tape electronically with the media number, and physically with a sticky label showing the media number and month—for example:- 501 January
- 502 February
- 503 March
- …
- In the backup job, select this media type. Choose a retention period so that each volume expires the day before it is due to be reused.
Each time the backup job runs, only one tape of the correct type will ever be available—the one from the same point in the previous cycle that has just expired.
Media Life Cycle
A typical backup tape might go through the following stages.
Recording media on the system
- A batch of 10 tapes is bought and registered to backup3G via the Media > Maintain > Add batch option. The next ten available media numbers are assigned to these tapes and recorded in the media database.
- The tapes are electronically labeled with the media number, either when the batch is entered or at the time when each tape is used for the first time.
- The new tapes are marked in the database as scratch, that is, available to be used. tapes available for use make up the scratch pool.
Using media for backup or archive
- When a backup job requests a tape, backup3G selects from the scratch pool a tape of the required type which has the oldest ‘date last written’, and prompts the user to load it.
- When the backup job finishes successfully the usage type for each tape in the media set is changed to ‘backup’, and the ‘Description’ field is set to the backup job description. backup3G calculates an expiry date and stores it in the media table. The name of the base directory for each backup step is stored in the media contents table.
- The user is prompted to unload the tape, usually as part of a daily scheduled duty to acknowledge completion of all overnight jobs.
Tracking tape location
- If the backup job specifies that the tape is to be stored offsite, the user is automatically prompted to do so when acknowledging that the job has completed. If auto-acknowledge is set to ‘always’ or ‘on success’, successful jobs will be acknowledged, the tape will be unloaded and marked as stored offsite.
- If the tape is moved some time after the job has completed, its new location can be recorded via the Move > Offsite option.
Recovery or expiry
- The user may browse the media contents table and online indexes prior to recovering files from the backup.
- The tape becomes usable again when its expiry date passes, or when it is marked as scratch on the system. The data on the tape isn’t actually overwritten until the next time it is used.
- A record can be kept when maintenance procedures (such as tape cleaning or a low-level format) are carried out, either as part of a regular maintenance schedule, or after a fault is detected.
- Eventually the tape will fail or be ‘retired’, and its media number reused or removed from the media database.
Media Support Tables
The media database comprises the media table plus several support tables, which contain information about the way media are used at your site. These tables are updated whenever backup and recovery jobs run. You can customize the media support tables through backup3G configuration > Maintain tables.
Site/Media Location
A media location is a place where media can be stored. There are several different types of location:
- onsite
- the main media library or tape vault
- offsite
- another branch or office, or a long-term archive facility
- load_device
- a silo or jukebox, from where tapes are loaded automatically into a drive
- load_area
- a place where media are stored temporarily before being loaded manually into a drive—for example a trolley.
You should add to the location table any places where media may be kept, including temporary locations and auto-loading devices such as jukeboxes. Any location that will not be used for media storage should be defined as location type ‘other’.
A tape stacker or trolley is in effect a temporary staging area from which tapes can be loaded. However a load device can also be a semi-permanent storage location, for example a jukebox containing up to several hundred tapes, where most tapes are just left in the unit.
Tapes allocated to an auto-loading location are assigned slot numbers. If backup3G detects that a tape has been loaded in the wrong slot, it corrects the slot number in the media table.
Backup3G has facilities to track media movements between locations, for example to register that archive tapes have been sent offsite, and to list which tapes are due to be returned onsite. Under the Media module, you can View media by location and Move selected media sets on- or off-site.
An offsite location is a place away from the main onsite location where media can be stored indefinitely, for example in a vault on another floor or in another building.
Offsite media must be returned onsite before they can be used in a backup or recovery job or allocated to a load device.
Usage type
Describes what a volume is currently used for. For example, tapes containing backup data have type ‘backup’, and tapes containing archive data have type ‘archive’. Unused media are set to type ‘scratch’.
Maintenance Procedures
You can keep a history of corrective or preventive maintenance done to media, such as tape cleaning and low-level formatting. You can add your own site’s procedures to the ‘Media Maint Procedure’ table.
Media type
Each size and/or density of storage medium is stored as a unique media type. You can specify which media type a backup should request. For example, you could use the same drive to back up a large filesystem to a 5-gigabyte Exabyte tape, and smaller backups to 2-gigabyte tapes. You can add new formats to the ‘Media Type’ table.
The media type table also includes the part size and capacity, which are used when writing multi-part backups.
Media label type
Media should be electronically labeled with the media number. This helps to avoid volumes containing live data from being accidentally overwritten. Backup3G supports several label formats:
| COSstdtape | writes a short file containing the media number and ‘date last written’ to the start of a tape, and an end-of-data marker after the last file on the tape. This is recommended for tapes that will be used for appending backups. |
| COSstdBOT | writes the media number and ‘date last written’ to the start of a tape, but doesn’t write an end-of-data marker. |
| end_label | writes the label to the end of a diskette. This prevents the label being overwritten when a new backup is done. |
| sV_filesys | System V filesystem labels |
Media Table Locking
Processes that update media details (such as backup and recovery jobs, labelling a volume, and adding a media batch) lock the media table briefly while the new details are written to the database.
If the media table is temporarily locked when a second process tries to update it, the lock will usually have been released by the time the second process tries again. Interactive users are prompted to press a button to try again; unattended processes such as backup jobs retry automatically.
If a process locks out a backup for an excessive length of time, backup3G will kill that process rather than hold up a backup. For example, if a user has left open the ‘Change media details’ prompt form, backup3G will cancel the user’s process to release the lock and allow backups to continue.