Data Domain Restorer and Long-Term Retention to the Cloud: Frequently Asked Questions

• A new feature called LTR was introduced starting with Data Domain Operating System (DDOS) 6.0.
• LTR allows certain models of DDRs to migrate a subset of files or data to an object or cloud storage, known as a cloud tier, from a range of supported public or private cloud providers.
• To physically migrate files or data to object storage, a data movement process is run on the DDR.
• To physically free redundant data from the cloud tier, a cloud tier-cleaning process is run on the DDR.
• LTR is a licensed feature and requires a CLOUDTIER_CAPACITY license.
• LTR requires some local storage for cloud tier metadata.

• As LTR is considered a new feature from DDOS 6.x and later, an elicense is required. 
• The type of elicense required is called a CLOUDTIER_CAPACITY license. An example of a CLOUDTIER_CAPACITY license is as follows:

Capacity licenses:
##   Feature              Shelf Model   Capacity     Mode        Expiration Date
--   ------------------   -----------   ----------   ---------   ---------------
1    CLOUDTIER-CAPACITY   n/a           136.42 TiB   permanent   n/a
--   ------------------   -----------   ----------   ---------   ---------------

• Normal DDRs (without an LTR license) have a single tier known as the active tier.
• The active tier is the traditional tier of storage on all 'standard' DDRs.
• LTR systems have a second tier of storage known as a cloud tier.

Active Tier:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB*
----------------   --------   --------   ---------   ----   --------------
/data: pre-comp           -    36674.6           -      -                -
/data: post-comp    65460.3      585.4     64874.8     1%              0.1
/ddvar                 29.5       24.7         3.3    88%                -
/ddvar/core            31.5        1.1        28.8     4%                -
----------------   --------   --------   ---------   ----   --------------

Cloud Tier
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB
----------------   --------   --------   ---------   ----   -------------
/data: pre-comp           -       33.1           -      -               -
/data: post-comp      912.2       42.3       869.9     5%             4.1
----------------   --------   --------   ---------   ----   -------------

Total:
Resource           Size GiB   Used GiB   Avail GiB   Use%   Cleanable GiB
----------------   --------   --------   ---------   ----   -------------
/data: pre-comp           -    36674.6           -      -               -
/data: post-comp    65460.3      585.4     64874.8     1%             0.1
/ddvar                 29.5       24.7         3.3    88%               -
/ddvar/core            31.5        1.1        28.8     4%               -
----------------   --------   --------   ---------   ----   -------------

• A cloud tier consists of:   
  • Locally held metadata, stored on an enclosure if a physical DDR is used, or a LUN or device if DDVE is used.
  • Object storage providers
• Both of the above are combined into a cloud unit.
• If multiple cloud units are configured, they can share the locally held metadata.
• A maximum of two cloud units can be configured per system. Each cloud unit can be provisioned from a different object storage provider.
• Each cloud unit can be as large as the maximum supported active tier size for the given model of DDR. See the DDOS administration guide for more information.

• All data is initially written to the active tier where it starts to age.
• Short-lived data that reaches its retention period is expired/deleted as on a normal DDR.
• A subset of data requiring long-term retention, however, is migrated out to the cloud tier.
• The file system maintains a single namespace across all tiers so when a file migrates to the cloud, the namespace does not change and as such, is reasonably transparent to the user or backup application.
• For a file that has already been migrated to the cloud tier reaches it's retention period, it is expired/deleted as any other file would.
• The space that a file was using in the cloud tier is not reclaimed immediately, instead cloud tier cleaning must be run.

• Each cloud unit is a stand alone volume meaning that it is a self-contained deduplication unit.
• As a result data written to each cloud unit can only deduplicate against data in the same cloud unit.

• Files and directories have various timestamps associated with them.
• For example, a file or directory has a creation time, last access time, and modification time.
• DDOS has enhanced this further to also include a placement time. The placement time is the date and time that the file migrated from the active tier to the cloud tier.
• Depending upon the DDOS version, the placement time can be seen when examining which tier a file resides on. If the file has migrated to the cloud tier, the placement time is shown, for example:  

sysadmin@dd4500 # filesys report generate file-location
--------------------------------      ---------------------------
File Name                             Location(Unit Name)
--------------------------------      ---------------------------
/data/col1/mtree1/random-data-file-4        cloudunit2           Tue Sep 5 10:17:00 2017
/data/col1/mtree1/random-data-file-5        cloudunit2           Tue Sep 12 15:52:23 2017
/data/col1/mtree1/random-data-file-6        cloudunit2           Tue Sep 13 09:42:55 2017

• ptime is the last field in the above output, although it does not display a field header.

• A process called data movement is responsible for examining files within an MTree that reside in the active tier.
• Data movement starts by creating a snapshot of all MTrees configured for data movement.
• Each file has a modification time which stores the last time a file was written to.
• If a file previously migrated to the cloud tier, an additional time field called a placement time is set. The placement time stores the date and time that the file migrated to the cloud tier. If the placement time is set, this is used instead of the modification time. This is to avoid a file being continually migrating back to the cloud tier if a file is recalled (as recalling a file does not change its modification time).
• The snapshots created above are traversed by data movement.
• If the file being examined has reached a defined threshold value, as set by the data movement policy for the MTree in question, the file is examined to ascertain what data held in that file must migrate from the active tier to cloud tier. A data movement policy is set per MTree.
• The unique segments for the selected file are written or copied to the cloud tier. 
• Once the unique segments have been copied, the file is verified by reading them back to ensure that the migration was successful.
• Once the file has been verified, the meta data is updated to reflect that the file now resides upon the cloud tier.
• The data movement process can be scheduled to run at a certain frequency or can be manually initiated.

• There are three phases associated with data movement, the copy phase, the verify phase and the install phase.
• The copy phase is responsible for identifying segments that must be copied to the cloud and then migrating these segments to the cloud.
• Once the copy phase starts, it is cloud or object storage, and is used as the copy phase copies the segments identified from the active tier to the cloud tier.
• The verify phase is responsible for ensuring that a file's segments were successfully migrated to the cloud.
• The install phase is responsible for updating the metadata, pertaining to file that was migrated, to show that it now resides on cloud or object storage.
• Each file has to complete all three phases in order for data movement to be deemed successful for that file. Therefore, until the install phase completes for a file, the file remains in the active tier.

• Data movement policies can be one of the following:   
  • Age threshold: If a files placement or modification time is greater than the age range set, it is selected for migration to the cloud tier.
  • Age range: If a file placement or modification time falls within a certain range, it is selected for migration to the cloud tier.
  • Application defined: The backup application designates if a file is to be selected for migration to the cloud tier.
• Policies are mutually exclusive, that is an MTree can only have one policy set at a time.

• The command below can be used. For example:   

data-movement policy set <policy name> <policy type values> totier cloud cloud-unit <cloud unit name> mtrees <mtree list>

sysadmin@dd4500 # data-movement policy set age-threshold 14 to-tier cloud cloud-unit cloudunit1 mtrees /data/col1/mtree1
sysadmin@dd4500 # data-movement policy set age-range min-age 14 max-age 100 to-tier cloud cloud-unit cloudunit1 mtrees /data/col1/mtree1
sysadmin@dd4500 # data-movement policy set app-managed to-tier cloud cloud-unit cloudunit1 mtrees /data/col1/mtree1

• The command below provides a list of which MTrees have any data movement policies assigned to them. For example:   

data-movement policy show

sysadmin@dd4500 # data-movement policy show
Mtree               Target(Tier/Unit Name)   Policy      Value      
-----------------   ----------------------   ---------   -----------
/data/col1/mtree1   Cloud/cloudunit1         age-range   14-100 days
-----------------   ----------------------   ---------   -----------

• The data-movement policy for the MTree in question is set to app-managed. This is either done manually, or the backup application performs this using the Data Domain REST API interface.
• The backup application must be LTR aware.
• The backup application must use DDBoost, and the version of DDBoost must be LTR aware and compatible.
• Using the DDBoost library/API, the backup application will set the placement time for the file that must be migrated to the cloud tier is set to a special value indicating that next time data movement runs, the file is to be migrated to the cloud.
• When data movement runs on the Data Domain system, the placement time is checked and if it is set to the special value, as mentioned above, then it migrates the file to the cloud.

• The command below can be used, for example:   

data-movement start

sysadmin@dd4500 # data-movement start
Data-movement started.

• The command below can be used to check the status of data movement. For example:   

data-movement status

sysadmin@dd4500 # data-movement status
Data-movement to cloud tier:
----------------------------
Data-movement is initializing..

Data-movement recall:
---------------------
No recall operations found. 

• If data movement is running, the command below can be used, for example:   

data-movement watch 

sysadmin@dd4500 # data-movement watch
Data-movement: phase 1 of 3 (copying)
   92% complete; time: phase  0:08:04, total  0:08:14
      Copied (post-comp): 3.35 GiB, (pre-comp): 3.29 GiB,B,
      Files copied: 7, Files verified: 3, Files installed: 3

• The command below can be used. For example:   

data-movement stop 

sysadmin@dd4500 # data-movement stop
Data-movement stop initiated. Run the status command to check its status.

• No. Essentially data movement can only migrate data to one cloud unit at a time.

• This is a high-level overview, see the detailed process in the DDOS administration guide.
• Add the appropriate CLOUDTIER_CAPACITY license.
• Set the system passphrase if it is not already set.
• Enable the cloud feature.
• Add the metadata storage for the cloud tier.
• Configure a cloud profile or profile for the appropriate cloud or object storage vendor.
• Add a cloud unit.
• Configure a data movement policy for the MTree or MTrees that require storing data in the cloud.
• Start data movement manually or wait for an automatic or scheduled data movement to start.

• Caution: This destroys any data held on the cloud unit, hence the data is unrecoverable, so proceed with caution.
• See the section in this knowledge base document entitled "How does a user ascertain what tier a file is located on" to understand which files reside on the cloud unit that is going to be deleted.
• These files should either be deleted if they are no longer required, or recalled to the active tier if they must be kept.
• If files must be kept, ensure all files are recalled before continuing.
• There should be no files remaining on the cloud unit that are deleted.
• Reset any data movement policies for the MTree or MTrees that use this cloud unit.
• Disable the file system.
• Delete the cloud unit. This marks the cloud unit in a DELETE_PENDING state, which is as designed.
• Enable the file system.
• Once the file system has started, it asynchronously starts deleting all objects in the cloud or object storage provider that were used by this cloud unit. Once all the objects are deleted, the buckets that this cloud unit used is also deleted. If there are many objects, then the cloud unit can stay in a DELETE_PENDING state for an extended amount of time.
• Once all the objects and buckets have been successfully removed, the cloud unit disappears from the cloud unit list.

• Contact Dell Support for further assistance.

• No. ER and LTR are mutually exclusive features.

• This operates in a similar fashion to files residing on the active tier
• Once a file reaches its retention period, it is deleted from the file system namespace.
• Cloud tier cleaning is scheduled to run. By default cloud tier cleaning is run after every four active tier cleaning sessions.
• For cloud tier cleaning to run, the cloud unit being cleaned must have at least 1% of superfluous or cleanable data in order to start. This is because any cloud network traffic could be chargeable so the DDR tries to limit the network traffic where possible.
• Cloud tier runs with a default of 50% cleaning throttle.
• Both the cloud tier-cleaning schedule and cleaning throttle can be changed.
• Active tier and cloud tier cleaning cannot run in parallel.
• If automatic or scheduled cloud tier cleaning is running, it is preempted by active tier cleaning.
• If a manual cloud tier clean is initiated, active tier cleaning cannot start until cloud tier-cleaning has been completed.
• If a cloud tier has two cloud units, only one cloud unit is cleaned per scheduled or automatic cloud tier clean. The cloud unit's are operated on in a round-robin fashion from a cloud tier-cleaning perspective. When there are two cloud units, it is a requirement to specify the cloud unit to clean when running from the command line (cloud clean start <unit-name>)
• If a cloud tier-clean fails to start on a cloud unit, for example, the current cloud unit does not have enough cleanable data to deem it worthwhile, then the system automatically attempts to clean from the next cloud unit.
• For further information regarding cloud tier cleaning, see the following article Data Domain: An introduction to long-term retention, cloud tier cleaning, and garbage collection on Data Domain Restorers (DDRs)

• The command below can be used. For example:   

cloud clean start <cloud unit> 

sysadmin@dd4500 # cloud clean start cloudunit2
Cloud tier cleaning started for cloud unit "cloudunit2". Use 'cloud clean watch' to monitor progress.

• The command below can be used to check if cloud cleaning is running. For example:   

cloud clean start <cloud unit> 

sysadmin@dd4500 # cloud clean status
Previous cloud tier cleaning attempt was unsuccessful.
 Failure reason:
cloud unit "cloudunit2" did not have sufficient cleanable data.
Cloud tier cleaning finished at 2017/03/15 12:16:06.

• If cloud clean is running, it can be monitored by using the command below:

cloud clean watch

• No. Both active tier cleaning and cloud tier cleaning both use the same common internal shared data structures which require exclusive access.

• The command below can be used to display the current cloud cleaning schedule. For example:   

cloud clean frequency show

sysadmin@dd4500 # cloud clean frequency show
Cloud tier cleaning frequency is set to run after every 4 active tier cleaning cycles.

• The command below is used to change a schedule. For example:  

cloud clean frequency set <value>

sysadmin@dd4500 # cloud clean frequency set 3
Cloud tier cleaning frequency is set to run after every 3 active tier cleaning cycles.

• By default, the cloud tier-cleaning throttle it is set to 50%. The command below can be used to reset it to the default throttle percentage.

cloud clean throttle reset

• The command below can be used to display the current cloud cleaning throttle. For example:   

cloud clean throttle show

sysadmin@dd4500 # cloud clean throttle show
Cloud tier cleaning throttle is set to 28 percent

• The command below can be used to change the cleaning throttle. For example:   

cloud clean throttle set <value> 

sysadmin@dd4500 # cloud clean throttle set 20
Cloud tier cleaning throttle set to 20 percent

• The cloud tier-cleaning throttle operates in a similar fashion to the active tier cleaning throttle, in that throttling limits the I/O and CPU resources that the cloud tier cleaning can use.
• It does not throttle network transfer.

• Cleaning is always considered an estimate. See the following KB articles which describe aspects around this topic as they equally apply to data that resides on the cloud tier:  

• Further to this, there are further specific details relating to how the cloud tier is implemented.
• Various methods have been implemented to limit the amount of network traffic to a cloud or object storage provider as this could come with associated costs.
• As mentioned above, a minimum of 1% of data churn is required in order for clean to run.
• When the file system is traversed to search for files that meet the data movement policy, only local copies of the metadata are examined.
• Any segments held on cloud or object storage which are found to only hold user data are marked for asynchronous deletion.
• Any segments containing at least one live segment are skipped because DDOS does not want to combine small amounts of data due to the network traffic involved.

• Use the command below for an example of the output generated by this command:  

filesys report generate file-location

sysadmin@dd4500 # filesys report generate file-location
--------------------------------      ---------------------------
File Name                             Location(Unit Name)
--------------------------------      ---------------------------
/data/col1/mtree1/random-data-file-1        Active
/data/col1/mtree1/random-data-file-2        Active
/data/col1/mtree1/random-data-file-4        cloudunit2
/data/col1/mtree1/random-data-file-5        cloudunit2
/data/col1/mtree1/random-data-file-6        cloudunit2

• This depends upon the version of DDOS in use along with the cloud provider:   

• Directly restoring files is possible without having to recall a file first. This is known as the 'direct restore' feature and is limited to ECS as the cloud or object provider.
• For more details about "direct restore" from Avamar, check the "Avamar Granular or File Level Restore from Data Domain Cloud Tier" white paper.
• The Avamar GLR/FLR (Direct Restore) feature needs a minimum combination of Avamar 18.1 or DDOS 6.1 with ECS as the Cloud Provider. 

• A file has to be recalled first. That is, the data migrated back from the cloud tier to the active tier.
• The file must be recalled from the cloud tier to the active tier using the data-movement recall command to allow any read from a file or modification to a file that resides on the cloud tier.
• Any attempt to read or modify a file that resides upon the cloud tier results in an I/O error being returned to whomever tries to read the file that is the backup application if the file is not recalled first.
• Some cloud aware backup applications can initiate file recalls, else files must be recalled manually.
• Since DDOS 7.7+:
  • Direct restore lets nonintegrated applications read files directly from the Cloud Tier without going through the Active Tier.
  • Key considerations in choosing to use direct restore include:
  • Direct restore does not require an integrated application and is transparent for nonintegrated applications.
  • Reading from the cloud tier does not require copying first into the active tier.
  • Histograms and statistics are available for tracking direct reads from the cloud tier.
  • Direct restore is supported only for AWS and ECS cloud providers.
  • Applications do experience cloud tier latency.
  • Reading directly from the cloud tier is not bandwidth optimized.

How many files can be recalled in parallel?

• DDOS 6.0 supports four files to be queued and recalled in parallel.
• DDOS 6.1 supports 1000 files to be queued and 4 files in the recall queue to be recalled in parallel.

• Systems with 256 GB of memory or more can recall up to 16 files at one time.
• Systems with less than 256 GB of memory can recall up to eight files at one time.
• DDVE instances can recall up to four files at one time.

• A file can be recalled by using the command below. For example:   

data-movement recall path <path-name> 

sysadmin@dd4500 # data-movement recall path /data/col1/mtree1/file1

• Depending on the DDOS version, all the files in Cloud may be recalled by running a single command such as below:   

sysadmin@dd4500 # data-movement recall mtree /data/col1/mtree1

• Check the "Dell DDOS Command Reference Guide" for your DDOS version for details

• A recall operation can be monitored by using the command below or if a specific file is required. For example:   

data-movement status path all

data-movement status path /data/col1/mtree/file1

sysadmin@dd4500 # data-movement status path /data/col1/mtree1/file1
Data-movement recall:
---------------------
Data-movement for  /data/col1/mtree1/file1 : phase 2 of 3
(Verifying)
80% complete; time: phase XX:XX:XX total XX:XX:XX
Copied (post-comp): XX XX, (pre-comp) XX XX 

• No. If a file is renamed, then it stays on its current tier.

•  Depending upon the DDOS version in use, DDOS supports the following cloud providers:   
• Amazon Web Services (AWS).
• Microsoft Azure Cloud
• Dell Elastic Cloud Storage (ECS)
• Virtustream
• See the DDOS administration guide for more information.

• Yes, encryption is supported on the cloud tier. This does not require an additional license, unlike active tier encryption.
• This can be configured when the cloud feature is enabled or modified after. 
• At the time of writing, only the embedded key manager is supported for cloud tier encryption and only one encryption algorithm can be used for LTR system wide.

• DDOS creates three buckets
• The buckets end with the string:

'-d0'

'-c0' 

'-m0'

• The bucket ending with the string '-d0' is used for data segments.
• The bucket ending with the string '-c0' is used for configuration data.
• The bucket ending with the string '-m0' is used for metadata.
• Prior to DDOS 6.1, whilst three buckets are created, only the bucket ending '-d0' is used. However all three buckets are needed so ensure that they are not removed.
• For more information about the three bucket naming convention, see the following article Data Domain: How to identify which buckets were created by a Data Domain Restorer (DDR) for Long-Term Retention (LTR) in the cloud

• No this is not possible.

• Yes
• If ECS is used, a load balancer is a mandatory requirement. Without a Load Balancer, Data Domain communicates to ECS on a single node and disconnects once multiple requests are made.
• A 1Gb network between the DDR and the cloud provider

• This depends upon the object or cloud provider being used and also the configuration.
• AWS, Virtustream, or Azure requires a certificate. See the DDOS administration guide for more information.
• If ECS is configured using an http endpoint, a certificate is not required.
• If ECS is configured using an https endpoint, a certificate is required. As a load balancer is a mandatory requirement, the certificate required is from the loan balancer system rather than the ECS system. Contact your load balancer provider for further details.
• When importing the certificate, it must be in PEM format. Some providers do not provide the certificate in the PEM format, so it must be converted prior to importing.

• Collection replication is _not_ supported.
• Directory replication is supported, however it can only be used by the '/data/col1/backup' MTree, but this MTree does not support data movement.
• MTree replication is fully supported.
• MFR or VSR replication is fully supported.

• The source system takes a snapshot of the MTree (this snapshot includes details of files on active and cloud tiers).
• The source system replicates the snapshot to the active tier of the destination system.
• Only when the snapshot has been fully replicated is it exposed on the destination system (at which point files become available in the destination systems file system namespace).
• Only after files are exposed can data movement be run on the destination (assuming it is configured for LTR).
• As a result if the destinations active tier is not large enough to hold a complete snapshot from the source, the snapshot is never exposed and replication cannot complete initialization.

• If data which has already been migrated to the cloud tier on a source DDR must be replicated, it automatically recalls from the cloud provider to the active tier before it can be sent over the network.
• Recalling files from the cloud tier to the active tier may incur a cost or delay.

• Due to the inherent way that cloud or object storage works, a Data Domain system has no way to query the physical size of a cloud device as this could be seen as being seemingly infinite.
• However DDOS had to develop a way to display current usage/deduplication statistics from a DDOS perspective.
• Therefore one of two approaches is used:

1. The size of the cloud tier is sized by the CLOUDTIER_CAPACITY license
2. The size of the cloud tier is shown as a multiple of active tier unit sizes for that model type depending upon how many cloud units are configured. See the hardware installation guide for the model in question for more information regarding active tier sizes.

• Ensure that the file system is disabled.
• Disable the cloud unit that is unavailable using the command below:

cloud unit disable <cloud unit name>

• Enable the file system.

• Ensure that the file system is disabled.
• Enable the cloud unit using the command below:

cloud unit enable <cloud unit name>

• Enable the file system.

•  If files were not removed from an MTree before a cloud unit was deleted, the files continue to exist inside the file systems namespace.
• As such, the file location report shows that the files are part of a deleted cloud unit. For example:  

sysadmin@dd4500 # filesys report generate file-location
--------------------------------      ---------------------------
File Name                             Location(Unit Name)
--------------------------------      ---------------------------
/data/col1/mtree1/random-data-file-3  Deleted cloud-unit
/data/col1/mtree1/random-data-file-4  Deleted cloud-unit

• The files can still be seen in the file system namespace by accessing a CIFS/NFS share for this MTree.
• However the files are not readable because the cloud unit where they were located has been deleted.
• Therefore the only option is to delete these files as their data they referenced no longer exists.

• For example, this maybe required if changing from http to https or conversely, migrating to a new load balancer.
• At the time of writing, there is no way for a Data Domain administrator to perform this change. This functionality is being considered for a future DDOS release.
• However this can be performed by support or engineering.
• The file system must be disabled in order to perform this change.
• If this is required, all the configuration outside of the Data Domain system is performed first because once this is changed, when the file system is enabled, it expects to be able to communicate using the updated protocol or port and read the buckets or objects as it did previously.