Data Domain Operating System (DDOS) 6.0 introduces a new feature known as cloud retention or long term retention (LTR). This feature allows a second tier of object based storage provisioned by a cloud provider to be added to certain models of Data Domain Restorer (DDR) with an associated CLOUD_CAPACITY license.
On systems using LTR files ingested by the DDR are initially written to the active tier (locally attached storage). Data movement policies/age thresholds are then configured on a per mtree basis such that certain files requiring long term retention are later migrated from the active to the cloud tier by the data movement process (a regularly scheduled task).
Files in the cloud tier can be deleted as normal however associated space on cloud/object storage is not immediately reclaimed for use. To remove superflous data from the cloud the cloud tier must be cleaned.
Structure of the cloud tier:
The cloud tier is subdivided into 'cloud units'. Note that:
- The cloud tier can contain up to two cloud units
- Each cloud unit can be as large as the maximum supported active tier size for the given model of DDR
- Each cloud unit can be provisioned from a different object storage provider
For example:
# cloud unit list
Name Profile Status
----------------------- ------------ ------
B-unit LTR-ECS-Ben Active <=== ECS provider
cloud-unit-virtustream1 virtustream1 Active <=== Virtustream provider
----------------------- ------------ ------
Basic concepts of cloud cleaning:
- Cloud cleaning only operates against a single cloud unit during each run - to determine the cloud unit being cleaned the following message can be found in DDFS logs (/ddr/var/log/debug/ddfs.info) - in this case the cloud-unit-virtustream1 cloud unit is being cleaned:
08/12 13:25:07.551 (tid 0x7f22991eb880): gc: Physical Cleaning will run on partition: cloud-unit-virtustream1, select_flags: none, usr: SCHEDULED CLOUD-GC, asm: Yes
Unfortunately this information is not currently available via the Data Domain command line shell (DDSH) for in progress cloud unit cleans.
- If a system has multiple cloud units configured cloud clean will round robin cleaning of these units attempting to clean a single unit each time cloud clean is run
- Cloud clean can be started manually or automatically via a schedule - to start manually the following command is used:
# cloud clean start [cloud unit name]
- Active tier clean and cloud clean cannot run in parallel (due to both using the same memory structures within DDFS)
- If active tier clean is running (started either manually or via its schedule) and cloud clean is attempted to be started it will error, i.e.:
# cloud clean start cloudunit2
Failed to start: activer tier cleaning is currently running. Use 'filesys clean watch' to monitor its progress.
- If cloud clean has been started automatically (i.e. via its schedule) and active tier clean is started then cloud unit clean will be aborted to allow active tier clean to run. This is indicated by the following in DDFS logs::
08/12 13:25:24.532 (tid 0x7f2277e9d210): gc_asm_start: Abort scheduled cloud-GC
- If cloud clean has been started manually and active is attempted to be started then active tier clean will fail to start - cloud clean will be left to run to completion, i.e.:
# filesys clean start
**** Cleaning cannot start since Cloud tier cleaning is in progress. Use 'cloud clean watch' to monitor progress.
- A cloud unit must have experienced a minimum of 1% data 'churn' (i.e. >= 1% of the data currently on the cloud unit must be deemed to be superflous and therefore removable) for cloud clean to start. If this is not the case the following will be displayed on the command line if cloud clean is manually started:
# cloud clean start cloudunit2
**** Failed to start: cloud unit "cloudunit2" does not have sufficient cleanable data.
In addition the following will be displayed in DDFS logs if cloud clean is started manually or via its schedule:
07/26 15:38:58.496 (tid 0x7f7a450fd340): gc: cp: cloudunit2 has 0% churn, minimum churn needed to run gc: 1%
07/26 15:38:58.496 (tid 0x7f7a450fd340): gc: cp: cloudunit2 does not have sufficient churn for GC to run
- If a system contains two cloud units and scheduled cleaning of the first unit fails for some reason (for example insufficient churn) then clean will automatically attempt to start against the second unit (i.e. there is no requirement to wait for the next scheduled run of cloud clean for the second unit to be cleaned)
- Cloud clean can be throttled (similar to active tier clean) to determine what action should be taken when the system is under significant other workload (i.e. ingest/restores/replication).
As with active tier clean the throttle is set as a percentage between 0 and 100:
0%: Cloud clean releases resources quickly to other workload and as a result may run slowly but causes minimal impact to overall system performance
100%: Cloud clean does not release resources to other workload and therefore runs as fast as possible but may cause significant impact to overall system performance
Cloud clean throttle is set to a default of 50%:
# cloud clean throttle show
Cloud tier cleaning throttle is set to 50 percent
To modify the throttle the following command can be used - note that the new throttle value takes effect immediately and there is no requirement to restart DDFS or cloud clean after changing the throttle:
# cloud clean throttle set 75
Cloud tier cleaning throttle set to 75 percent
Scheduling cloud clean:
In DDOS 6.0 and later the way in which active tier clean is scheduled has not changed - by default active tier clean is scheduled to run once a week at 0600 on Tuesday, i.e.:
# filesys clean show schedule
Filesystem cleaning is scheduled to run "Tue" at "0600".
Cloud clean is scheduled, by default, to run after every 4th invocation of scheduled active tier clean. To display the cloud clean schedule the following command should be used:
# cloud clean frequency show
Cloud tier cleaning frequency is set to run after every 4 active tier cleaning cycles.
As a result, on a system with default configuration, cloud clean will be started every 4 weeks - if the system has two cloud units each unit will be cleaned once every 8 weeks.
To change the cloud clean frequency the following command can be used:
# cloud clean frequency set 2
Cloud tier cleaning frequency is set to run after every 2 active tier cleaning cycles.
To reset cloud clean to its default schedule of after every 4 active tier cleans the following command can be used:
# cloud clean frequency reset
Cloud tier cleaning frequency is reset to default (every 4 active tier cleaning cycles).
Note that the cloud clean schedule does not include manually started active tier clean cycles. As a result, on the above system, even if active tier clean were run manually every day cloud tier clean would only start once every 4 weeks.
It is also possible to completely disable scheduled cloud clean using the following command:
# cloud clean frequency set never
Cloud tier cleaning frequency is set to "never".
In this case cloud clean will only run when started manually.
To stop a currently running cloud clean the following command can be used:
# cloud clean stop
To determine when cloud clean last ran the following command can be used:
# cloud clean status
Cloud tier cleaning finished at 2016/08/01 20:54:43.
Cloud clean algorithm:
Cloud clean will use the same clean algorithm as configured for the active tier. In DDOS 6.0 (and later) this defaults to perfect physical garbage collection (PPGC) however this can be changed to physical garbage collection (PGC) via system parameters.
Note that physical garbage collection should not be disabled as using the traditional/full cleaning algorithm to clean a cloud unit may result in a DDFS panic/restart
The algorithm used for cloud clean is displayed in DDFS logs when clean starts, i.e.:
06/28 10:51:56.960 (tid 0x7fc5bccb2d50): gc: gc_start_intern: Algorithm selected: Physical Cleaning <=== PPGC or PGC
07/27 12:21:18.224 (tid 0x7f92b8cfe7e0): gc: gc_start_intern: Algorithm selected: Full Cleaning <=== Traditional GC
Note that from the above output it is not possible to distinguish between PPGC or PGC - the specific algorithm used is apparent due to the number of phases run by clean - in general:
Traditional/full GC: 10 phases
PGC: 12 phases
PPGC: 6 phases
For further information on changing the clean algorithm used on a system contact your contracted support provider
Differences between active tier clean and cloud tier clean copy phases:
The copy phase of clean is the phase where superflous data on a DDR is physically removed/space reclaimed. Note that there are differences between how the copy phase operates against the active and cloud tiers:
Active tier:
- Data written to the active tier of a DDR is contained within 4.5Mb containers
- By default a container will only be considered for 'copy' by clean if it contains <= 92% 'live' (i.e. actively referenced) data
- The live data will be extracted from the container and written to a new container (along with live data from other copied containers) at the end of the file system
- On disk indices are updated to reflect the new container holding the live data
- The original container (holding both live and dead data) is then deleted and underlying disk space made available for use
Cloud tier:
- Data written to the cloud tier of a DDR is structured differently - instead of being placed within 4.5Mb containers individual chunks of data (64Kb compression regions) are written to the cloud unit (NOTE : for DDOS 6.1.2.0 and later, objects stored in the cloud unit unit will be larger, see Data Domain: Large Object Size for Cloud Tier for details)
- Instead of extracting live data from an existing compression region and copying this forwards cloud clean will only consider compression regions which contain solely dead data for deletion
As a result if a compression region contains a single very small amount of data which is still live (referenced by a file) it will not be deleted and dead data within the compression region not removed from disk (i.e. none of the space used by the compression region will be reclaimed)
Compression regions marked for deletion are processed asynchronously by cloud clean - as a result free space on a cloud unit may continue to increase even once cloud clean has completed
This difference is due to the inherent cost involved in reading/writing large amount of data on cloud storage however it does mean that a cloud unit could become artificially full (i.e. contain a large number of compression regions each of which contain a very small amount of live data preventing their removal).
If this situation occurs it is possible to set system parameters forcing a 'defragmentation clean' of the cloud unit - this will copy forwards live data from existing compression regions to consolidate live data in as few compression regions as possible allowing space to be freed.
For further information on running a 'defragmentation clean' please contact your contracted support provider.