Understanding Policy Based Retention with TSM

1. Understanding Policy Based Retention with TSM Daniel Curtis

2. Agenda • Overview • Policy structure • Retention of Archives • Retention of backups • Exceptions

3. Overview • TSM provides two basic means of storing files, images, etc ...., as a general label I will call anything stored in TSM an OBJECT. The two classes of objects are • BACKUPS, a backup object is subject to retention based on the settings in the management class to which it is bound. Backup retention, or "how long will this object be saved ?" is governed by the concepts of Incremental forever and versioning. • ARCHIVES, and archive object is subject to a much more narrower set of settings in the management class to which it is bound. Archive retention is governed by a simple life time and is not subject to versioning. • The Policy structure that TSM uses to control when an object stored in TSM is deleted from the TSM database is hierarchical and allows control down to single object level. • There are exceptions where objects stored in TSM are not governed by either the backup retention settings or the Archive retention settings and we will cover those.

4. POLICY DOMAIN Policy Structure There are 4 nested levels of policy objects in TSM, they are: • POLICY DOMAIN • This is the top most structure, and the simplest in terms of options. The Policy Domain allows for the logical grouping of client nodes, such as by client platform. For example all AIX clients in the AIX domain and all Windows clients in the Windows domain, or by type all backup archive clients in a domain and all TDP for Domino clients in a second domain. Grouping can be arbitrary, such as a domain for all clients in the finance department and another for all clients in the legal department.

5. POLICY DOMAIN Policy Structure • POLICY DOMAIN continued • There are only three options or settings in a policy domain. • DESCRIPTION, is exactly what it says, a text description of why the domain exists. • BACKRETENTION, is the parameter that comes into play when all policy structures below the Domain that pertain to backups have been deleted, BACKRETENTION is set in days with 30 being the current default. If all other policies for backed up objects are deleted then those objects will be governed by this setting. • ARCHRETENTION, is the parameter that comes into play when all policy structures below the Domain that pertain to archives have been deleted, ARCHRETENTION is set in days with 365 being the current default. If all other policies for archived objects are deleted then those objects will be governed by this setting.

6. POLICY DOMAIN POLICY SET Policy Structure • POLICY SET • The second tier of Policy objects is the Policy Set, it is the simplest in terms of possible settings but it does introduce complexity into the equation by how multiple copies of the policy set are created, and handled.The only option when defining a policy set is DESCRIPTION. • The complexity comes from the fact that ONLY ONE (1) policy set can be ACTIVE in a domain at any one time. There can be multiple Policy Sets Defined to the Policy Domain but only 1 of them is ACTIVE. The ACTIVE policy set contains the management classes and the associated copy groups that contain the policy settings that are enforced on the objects stored under the Policy Domain. • A Policy Set can be DEFINED, COPIED, VALIDATED, and ACTIVATED, also the Policy Set description can be UPDATED.

7. POLICY DOMAIN POLICY SET Policy Structure • POLICY SET Continued • When a Policy set is ACTIVATED the contents of that policy set are copied to the ACTIVE policy set and they become the parameters in that newly ACTIVE policy set are what governs objects stored under that Domain. It is possible to define as many policy sets with unique names as you wish, but none of them will be used to control the retention of objects until activated. A good practice to use when an update of the active policy set is required, would be to COPY the policy set so that you are sure you are working on the same set of policies that is currently in use. After the copy, make your modifications to the copy, then modify and activate the copy. The activate will copy the values in the modified policy set to the ACTIVE policy set those will become the controlling set of policies. • An alternative would be to use the description field to identify which policy set was last activated, however the policy set may have been modified in the interim since the activation and may no longer correctly reflect the correct policy set.

8. POLICY DOMAIN POLICY SET MANAGEMENT CLASS Policy Structure • MANAGEMENT CLASS • The third tier of the policy structure is the MANAGEMENT CLASS, there can be as many management classes as desired, each management class can have either a BACKUP COPYGROUP or an ARCHIVE COPYGROUP, it may also contain both types or neither. In order to ACTIVATE a policy set, it must contain at least one (1) Management Class, and there must be one (1) management class that is defined as the DEFAULT Management Class. If there is no DEFAULT Management Class then a validate and/or activate of the policy set will fail. The management class contains several options that control the backup behavior for Hierarchical Storage Management files.

9. POLICY DOMAIN POLICY SET MANAGEMENT CLASS Policy Structure • MANAGEMENT CLASS continued • DESCRIPTION • SPACEMGTECHNIQUE, specifies which type of HSM migration is allowed for files bound to this management class, AUTOMATIC, SELECTIVE, NONE are the possible values. • AUTOMIGNONUSE, Specifies the number of days that must elapse since a file was last accessed before it is eligible for automatic migration. The default value is 0,and the maximum is 9999. • MIGREQUIRESBKUP, Specifies whether a backup version of a file must exist before a file can be migrated. The default is YES. • MIGDESTINATION Specifies the primary storage pool where the server initially stores files migrated from HSM clients.

10. POLICY DOMAIN POLICY SET MANAGEMENT CLASS Policy Structure • MANAGEMENT CLASS continued • As noted earlier a Management Class is allowed to NOT contain any copy groups, if this is a case files that are bound to that management class will not be backed up or archived. If a Management Class contains a BACKUP copy group, then files that are bound to the Management Class during backup will be stored by the TSM server. If an ARCHIVE copy group exists then files that are bound to the Management Class during an ARCHIVE operation will be stored by the server, if the correct type of copy group is not in the specified management class then the BACKUP or ARCHIVE of that file will fail. • Files are BOUND to a Management Class through an INCLUDE statement during the client BACKUP or ARCHIVE client operation. Specific construction of the INCLUDE statement is beyond this presentation but the basic format is INCLUDE file-spec MANAGMENTCLASSNAME • Any file that matches the file-spec is BOUND to the named Management Class and is governed by the copy group settings in that Management Class, this is the mechanism that allows control of the retention of files on an individual level of granularity.

11. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Policy Structure • COPY GROUP • The final level of the Policy structure, is the COPY GROUP. There are two (2) types of COPYGROUP, BACKUP and ARCHIVE. Each type carries the actual options that control how long an object is kept by the TSM server and when it can be deleted by the expiration process. All copy groups are named STANDARD. They are differentiated by the Management Class, Policy Set, and Policy Domain to which they are associated. • BACKUP COPYGROUP The Backup Copygroup determines the original physical destination for a backup object and contains other options that govern how long that object lives. The effects of the retention options will be discussed later, the options are:

12. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Policy Structure • COPY GROUP continued • DESTINATION, Specifies the primary storage pool where the server initially stores backup data. You cannot specify a copy storage pool as the destination. • FREQUENCY, Specifies how frequently Tivoli Storage Manager can back up a file. The FREQUENCY value is used only during an incremental backup operation. This value is ignored during selective backup or partial incremental backup. You can specify an integer from 0 to 9999. The default value is 0, meaning that Tivoli Storage Manager can back up a file regardless of when the file was last backed up.

13. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Policy Structure • COPY GROUP continued • VEREXISTS, Specifies the maximum number of backup versions to retain for files that are currently on the client file system. The default value is 2. You can specify an integer from 1 to 9999, or NOLimit • VERDELETED, Specifies the maximum number of backup versions to retain for files that have been deleted from the client file system after being backed up using Tivoli Storage Manager. The default value is 1. You can specify an integer from 0 to 9999, or NOLimit.

14. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Policy Structure • COPY GROUP continued • RETEXTRA, Specifies the number of days to retain a backup version after that version becomes inactive. The default value is 30 days.You can specify an integer from 0 to 9999,or NOLimit. • RETONLY, Specifies the number of days to retain the last backup version of a file that has been deleted from the client file system. The default value is 60. You can specify an integer from 0 to 9999, or NOLimit.

15. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Policy Structure • COPY GROUP continued • MODE, Specifies whether Tivoli Storage Manager backs up a file only if the file has changed since the last backup, or whenever a client requests a backup. The default value is MODIFIED. Possible values are MODified, or ABSolute • SERIALIZATION, Specifies how Tivoli Storage Manager handles files or directories when they are modified during backup processing. The default value is SHRSTATIC. Possible values are: SHRSTatic, STatic, SHRDYnamic, or DYnamic.

16. POLICY DOMAIN POLICY SET MANAGEMENT CLASS COPY GROUP Policy Structure • COPY GROUP continued • ARCHIVE COPYGROUP The Archive Copygroup determines the original physical destination for an Archive object and contains other options that govern how long that object lives. The effects of the retention options will be discussed later, the options are: • DESTINATION, Specifies the primary storage pool where the server initially stores the archive copy. You cannot specify a copy storage pool as the destination. • RETVER, Specifies the number of days to keep an archive copy. The default value is 365. You can specify an integer from 0 to 30000, or NOLimit. • SERIALIZATION, Specifies how Tivoli Storage Manager handles files that are modified during archive. The default value is SHRSTATIC. Possible values are: SHRSTatic, STatic, SHRDYnamic, or DYnamic.

17. Retention of Archives • ARCHIVE RETENTION • An ARCHIVE in TSM usage is an object or group of objects that are saved for a specific number of days after they are stored in TSM, the ARCHIVE does not affect the number of backed up versions of an object, and will not change its expiration date due to another subsequent archive of the same object. • The setting in an archive copy group that governs the retention of the object is RETVER, the retain version setting is in days, it can be set for any value between 0 and 30,000. When an object is archived an entry in the expiring objects table is made for that object. In the expiring objects table entry is the management class to which it is bound. When the expiration process runs the management class is checked and the retver value taken from the copy group, the retver number of days is added to the date the object was archived and the resulting date is checked against the current date. Thus an object that was archived on June 6th and bound to an archive copy group with RETVER 15 would be expired on the first expiration run on or after June 21st. • Rebinding, there is no rebinding of archived objects, because the original object on the client machine is never looked at again in regards to that specific archived object. If the retver value in the copy group is updated and the changed policy set is activated then on the next expiration run the calculation will be made using the new value. If the management class to which the archive is bound is deleted the archived objects will be subject to the policy domain ARCHRETENTION setting.

18. Archived directories • One retention consideration for archives that is different from the above is the case of archived directory objects. When an object is archived for the first time, a copy is made of all the directories in the path that lead to that object. If an ARCHMC has been specified for the client then the ARCHMC overrides all other factors and the directory objects are bound to the specified management class. If the ARCHMC is not specified (or an INCLUDE MC is used) then the directory objects are bound to the default management class. For a complete description of this process please see DCF #1152060 and #1198242 (also listed in the references section). • If a an archive copy of the same directory already exists with in TSM storage, and the description is NOT unique, then TSM does not make a new copy. If the description is unique or the permissions of the directory have changed a new copy will be taken. For a detailed discussion of this please see DCF #1198242, and if you are having problems with a large number of extra directories being saved please see flash 10210 or 10258, (links are in the references).

19. Retention of Backups • BACKUP RETENTION • In comparison to archives, the retention behavior of backup objects is much more complex. The complexity comes from the interplay of the concepts of INCREMENTAL FOREVER, and VERSIONING controlled by the 4 backup retention settings. • The most recent backup copy of an object is always the ACTIVE version, an ACTIVE version is never eligible for expiration, there is no expiring objects table entry for ACTIVE objects. • Once a newer VERSION of an object is backed up, the old ACTIVE version is changed to INACTIVE. When the object is inactivated an entry is created for it in the expiring objects table with the management class to which it is bound. • During an INCRAMENTAL backup the TSM Backup Archive client compares the currently ACTIVE copy of an object to the copy currently on the client machine. If the copy on the client machine meets the criteria for a new copy to be taken a new version is made and becomes the ACTIVE version. If the object stored in TSM no longer exists on the client, or there exists an EXCLUD statement that matches the object, the ACTIVE version is deactivated and no new copy is made. • An inactive version of a backup object is subject to the retention settings, VEREXISTS, VERDELETED, RETEXTRA, RETONLY. If an ACTIVE version of the object exists the VEREXISTS and RETEXTRA apply. If there is no ACTIVE version then VERDELETED and RETONLY apply.

20. Retention of Backups • BACKUP RETENTION continued • For an object with an ACTIVE copy VEREXISTS determines the total number of versions that may exist, so a setting of VEREXISTS 4, means that at any time TSM may retain 4 copies of the object 1 ACTIVE version and 3 INACTIVE versions. RETEXTRA 10 places a life span of 10 days on INACTIVE versions of objects. So for an object that exists on the client machine, changes every day and has a backup taken each day, the two settings interact like this:

21. C - ACTIVE A - ACTIVE A - INACTIVE B - INACTIVE A - INACTIVE Retention of Backups • BACKUP RETENTION continued • Day 1, Backup made of a new file, VERSION-A is the ACTIVE version, will never expire • Day 2, Backup made of the file, VERSION-B is the ACTIVE version, will never expire • VERSION-A is deactivated, entry made in the expiring objects table, version will expire in 10 days • Day 3, Backup made of the file VERSION-C is the ACTIVE version, will never expires • VERSION-B is deactivated, entry made in the expiring objects table, version will expire in 10 days • VERSION-A is inactive, version will expire in 9 days B - ACTIVE

22. E - ACTIVE C - INACTIVE D - INACTIVE C - INACTIVE B - INACTIVE A - INACTIVE B - INACTIVE A - EXPIRE Retention of Backups • BACKUP RETENTION continued • Day4, Backup made of the file, VERSION-D is the ACTIVE version, will never expire • VERSION-C is deactivated, entry made in the expiring objects table, version will expire in 10 days • VERSION-B is inactive, version will expire in 9 days • VERSION-A is inactive, version will expire in 8 days • Day5, Backup made of the file, VERSION-E is the ACTIVE version, will never expire • VERSION-D is deactivated, entry made in the expiring objects table, version will expire in 10 days • VERSION-C is inactive, version will expire in 9 days • VERSION-B is inactive, version will expire in 8 days • VERSION-A is inactive, version is number 5, it will expire today D - ACTIVE

23. F - ACTIVE F - ACTIVE D - INACTIVE D - INACTIVE C - INACTIVE C - INACTIVE E - INACTIVE E - INACTIVE B - EXPIRE Retention of Backups • BACKUP RETENTION continued • Day6, Backup made of the file, VERSION-F is the ACTIVE version, will never expire • VERSION-E is deactivated, entry made in the expiring objects table, version will expire in 10 days • VERSION-D is inactive, version will expire in 9 days • VERSION-C is inactive, version will expire in 8 days • VERSION-B is inactive, version is number 5, it will expire today • This object has reached a point of equilibrium, for each new version taken an old version is removed • Day7, file does not change, no backup, VERSION-F is still the ACTIVE version, will never expire • VERSION-E is deactivated, entry made in the expiring objects table, version will expire in 9 days • VERSION-D is inactive, version will expire in 8 days • VERSION-C is inactive, version will expire in 7 days • Note the days to expiration for the inactive files continue to decrease

24. F - ACTIVE Retention of Backups • BACKUP RETENTION continued • Now skip 10 days • Day17, File does not change, no backups taken, VERSION-F is the ACTIVE version, will never expire • At this point the extra version have all expired, only the most recent version exists, a restore could only be made to the state of the file on Day6.

25. D - INACTIVE E - INACTIVE F - INACTIVE C - EXPIRE Retention of Backups • BACKUP RETENTION continued • Once an object is excluded or deleted from the client file system there is no ACTIVE version and the RETDELETED and RETONLY parameters take over. RETDELETED 3 specifies 3 versions of a deleted file are retained, and RETONLY 15 specifies that the last existent version is kept for 15 days not 10, • starting with the deletion of the file from the client on Day8 above the settings work like this: • Day8, file is deleted from client no backup is taken, VERSION-F is deactivated, entry made in the expiring objects table, version will expire in 10 days • VERSION-E is inactive, version will expire in 8 days • VERSION-D is inactive, version will expire in 7 days • VERSION-C is inactive, and due to RETDEL 3 is version 4 and will expire today

26. F - INACTIVE E - EXPIRE F - EXPIRE Retention of Backups • BACKUP RETENTION continued • Skip 8 days • Day16, VERSION-F is inactive, version will expire in 2 days • VERSION-E is inactive, version will expire today • Day17, VERSION-F is inactive, version is the last remaining version, and is subject to the RETONLY of 15, version will expire in 6 days • Skip 7 days • Day24, File is deleted from client No version remain in TSM storage. NO VERSIONS EXIST

27. Retention of Backups • BACKUP RETENTION continued • Rebinding, if the management class to which backup objects are bound is changed, for example by adding an INCLUDE statement that binds all of the client files to a different management class, on the NEXT full incremental backup of the client ALL of the backup versions of files belonging to that client that matches the include will be bound to the new management class. The settings in the copy group for the new management class will be used to determine the eligibility of a version for expiration. This can have the affect of removing version sooner than they would have been under the old management class. If the management class to which objects are bound is deleted (and no include statement exists to bind them to a new management class) on the NEXT full incremental backup of a client with files bound to that management class, the clients files will be rebound to the DEFAULT management class for the ACTIVE policy set. If the files were bound to the default and the default is deleted then the objects are governed by the Policy Domain BACKUPRETENTION setting. • Also if a client file system is never backed up again, such as if the client machine is removed from the network, the TSM client is uninstalled or disabled, a Unix file system is never again mounted during TSM backups, or a disk drive is removed and no drive with that same label installed. The inactive versions of files will expire per the retention settings, the ACTIVE versions will remain. This may entail a manual delete of a client file space or spaces to free the space used and expire the ACTIVE objects from the TSM database.

28. Retention of Backups • BACKUP RETENTION continued • The final consideration for backup objects is the handling of directory objects. • If no DIRMC is set for the client then TSM binds directories to the management class in the active policy set with the longest retain extra setting. This normally ensures that directory objects do not expire before the files that rely on them. • If a directory object is expired and a restore is required the client GUI will not be able to do the restore. The object can still be restored with the client command line and the directory structure created, however the directories will be created with client system defaults and any custom or extend attributes for the directories will be lost. • To ensure that directories are available to support a point-in-time restore it is recommended that all retentions settings be configured to the minimum number of days back to which a point-in-time will be attempted.

29. EXCEPTIONS • The sections above deal specifically with archived objects and normal backup objects. There are several exceptions where data stored in TSM is not subject to these rules. If the TSM server is enabled for ARCHIVERETENTIONPROTECTION, then there are extra rules for the expiration of archived objects, and that TSM server does not accept backup objects for storage.Image backups, and backup sets have their own specific rules. Data generated by the TSM server such as exported data and TSM database backups also have specific mechanisms other than those above to govern how long they are retained.

30. Q & A

31. REFERENCES • DCF documents for archived directory behavior #1152060 and #1198242 • Flash 10210 and 10258 for cleanup backupgroups • http://www03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10258