Table of Contents
TSM dedup on server side
TSM deduplication ratios typically range from 2:1 (50% reduction) to 15:1 (93% reduction)
snappdiff incremental is supported on container snappmirror is not supported on container TOC is not supported on container
Get dedup stats on a specific node
The generate process can be very long to run
Protect: ISPTST> GENERATE DEDUPSTATS CONTAINER1 NODE1 Protect: ISPTEST1>QUERY DEDUPSTATS Date/Time Storage Node Name Filespace FSID Type Total Saving Total Data (MB) Pool Name Name Percentage Protected ------------------- ---------- ---------- ---------- ---- ---- ------------ --------- 02/12/2021 12:03:47 CONTAINER1 NODE1 /var/log/audit 10 Bkup 26.39 13
You can also run this command by adding a filespace name to reduce processing time
Change deduplication file size limit
Files bigger than 300GB won't be deduplicated, by default on client side. Use the CLIENTDEDUPTXNlimit server option to specify the maximum size for a transaction (range between 32 – 2048 GB, default 300 GB).
setopt CLIENTDEDUPTXNlimit 80
Disable server-side deduplication for all objects over 1024 GB (range between 32-2048GB, default 300 GB):
setopt SERVERDEDUPTXNlimit 1024
SPLITLARGEOBJECTS setting. This is introduced in v7.1 and is part of the increased data deduplication and scalability, allowing v7.1 for up to 10 times more ingest rate compared with previous TSM versions.
SPLITLARGEOBJECTS will split (segment) a file that is bigger then 10 GB into smaller pieces (fragments).
SPLITLARGEObjects=Yes or NO. YES is the default. Available as a parameter of the following commands:
register node update node
Note: if you want to maximize the throught of backups that are going directly to tape, disable SPLITLARGEObjects:
update node xxxx SPLITLARGEObjects=no
Deduplication ratio and percent calculation
TSM> select stgpool_name,est_capacity_mb,pct_utilized,space_saved_mb from stgpools
data physically stored in stgpool:
est_capacity_mb x pct_utilized / 100
total data backed up:
space_saved_mb + (est_capacity_mb x pct_utilized / 100)
data not stored, duplicate shrunk:
space_saved_mb
to calaculate the percentage of data not stored:
100 x space_saved_mb / (space_saved_mb + (est_capacity_mb x pct_utilized / 100))
to calaculate the ratio of data not stored <value>:1
value= (space_saved_mb / (est_capacity_mb x pct_utilized / 100))
Dedup technology
| Server side NextGen dedupe in container stgpool | Server side legacy dedupe in file dev stgpool | Client side dedupe with container stgpool | Client side dedupe with legacy file dev stgpool | ||
|---|---|---|---|---|---|
| When is data deduped | Server side Inline | Out-of-Band: Dedupe processing & reclamation required after backups finish | Before client sends data to server | Before client sends data to server | |
| Data transferred from client to server | Fully hydrated data | Fully hydrated data | Dehydrated data | Dehydrated data | |
| Where is data stored | Container Stgpool | File Dev Class Stgpool | Container Stgpool | File Dev Class Stgpool | |
| Storage pool encryption supported | Yes in cloud containers | No | Yes in cloud containers | No | |
| Compression | Yes (7.1.5) | No (client side only) | Yes (not recommended) | Yes (not recommended) | |
| Client side encryption supported | Yes (not deduped) | Yes (not deduped) | No (SSL works with dedupe) | No (SSL works with dedupe) | |
| “protect stgpool” | Yes in directory containers | No | Yes in directory containers | No | |
| (Replicate data only) | |||||
| “repair stgpool” | Yes in directory containers | No | Yes in directory containers | No | |
| “replicate node” | Yes | Yes | Yes | Yes | |
| (Replicate data & meta data) | |||||
| “replicate node recoverdamaged=yes” | No | Yes | No | Yes | |
| Stgpool reclamation required | No | Yes | No | Yes | |
| TSM version supported | 7.1.3+ | 6.1+ | 7.1.3+ | 6.2+ | |
| Special license needed | No | No | No | No | |
| Amount of data processed per day | 80 TB per day | 20 TB per day | 100 TB per day | 30 TB per day | |
| Data managed on a single TSM server | 1 to 4 PB | 100 to 400 TB | 1 to 4 PB | 100 to 400 TB | |
| Max physical capacity on a TSM server | 1 PB | 100TB | 1 PB | 100TB |
