Use the follwin flag in your command
-testflag=VMRESTOR_DONT_FORCE_THICK
Note: In either of these sections a path to free disk space will be needed for tracing.
Tracing of VM operations can be extensive. If the operation you are tracing processes a number of VMs, determine if the behavior can be duplicated on a smaller subset so that data collection is not cumbersome.
vixDiskLib.transport.LogLevel = "6" vixDiskLib.nfc.LogLevel = "6"
traceflag tid pid enter exit general vm tracefile <Path with free disk space>/datamvr.trace
Change the VMCLI_TRACE value to YES Remove the # from in front of VE_TRACE_FILE and VE_TRACE_FLAGS
<Path with free disk space>/datamvr.trace Entire /opt/Tivoli/tsm/tdpvmware/common/logs folder Entire /opt/Tivoli/tsm/tdpvmware/tsmcli folder dsmwebcl.log from BAclient folder datamover dsminfo.txt file
vixDiskLib.transport.LogLevel = “6”
vixDiskLib.nfc.LogLevel = "6" * Edit the optfile used by the datamover(in the BAclient folder). Add the following: traceflag tid pid enter exit general vm tracefile <Path with free disk space>/datamvr.trace * Recreate the problem experienced * Reverse the changes made to prevent tracing from running * Gather the following for review: <Path with free disk space>/datamvr.trace dsmwebcl.log from BAclient folder datamover dsminfo.txt file
Can you provide the following data:
Vcsplugin.log located in the Baclient directory VEprofile folder, path : C:\IBM\SpectrumProtect\webserver\usr\servers\veProfile Screenshot for all the plugin tabs the vCenter host the following : /storage/vsphere-ui/IBMDataProtection/logs/log.txt (this exactly our log-Discussion between our plugin and webserver)
C:\>cd C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts>vmcli.cmd -f inquire_detail -t TSM -q dmverify -d Datacenter01 -o VM_DMNTXC1 #TASK 146173 inquire_detail 20220205222107802 #RUN 146127 20220205222107818 #LANG en_IE #PARAM BACKEND=TSM #PARAM OPERATION_TYPE 6 #PHASE_COUNT 4 #PHASE PREPARE #PARAM BACKUP_TYPE=0 #PARAM TSM_SERVER_NAME=172.11.1.22 #PARAM TSM_SERVER_PORT=1500 #PARAM TSMCLI_NODE_NAME=VMCLI #PARAM VCENTER_NODE_NAME= #PARAM DATACENTER_NODE_NAME=Datacenter01 #PARAM OFFLOAD_HOST_NAME=VM_DMNTXC1 #PARAM PASSWORD_TYPE=CLINODE #PARAM TSM_OPTFILE=C:\Users\TEMP\AppData\Local\Temp\1\T4V7315.tmp #PARAM INPUT_FILE= #PARAM TRACEFILE= #PARAM TRACEFLAGS= #PARAM RUNID=146173 #PARAM SSL= #PARAM SSLREQUIRED= #PARAM SSLACCEPTCERTFROMSERV= #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DM #CHILD hladdress:tsmdm01.test.lu #PARENT peernode:VM_DM #CHILD lladdress:49204 #PARENT peernode:VM_DM #CHILD targetnode:Datacenter01 #PARENT peernode:VM_LOCAL_MP_WIN #CHILD hladdress:tsmdm01.test.lu #PARENT peernode:VM_LOCAL_MP_WIN #CHILD lladdress:49209 #PARENT peernode:VM_LOCAL_MP_WIN #CHILD targetnode:Datacenter01 #PARENT peernode:VM_LOCAL_MP_LNX #CHILD hladdress:linux01.lu #PARENT peernode:VM_LOCAL_MP_LNX #CHILD lladdress:46717 #PARENT peernode:VM_LOCAL_MP_LNX #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DM1 #CHILD hladdress:dm01.test.lu #PARENT peernode:VM_DM1 #CHILD lladdress:46694 #PARENT peernode:VM_DM1 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DM2 ... #CHILD vmhost: #PARENT vmname: #CHILD vmuuid: #PARENT vmname: #CHILD vmtools: #PARENT vmname: #CHILD vcenter:0 #PARENT vmname: #CHILD host:0 #PARENT vmname: #CHILD toolsVersion:0 #PARENT vmname: #CHILD proxyNodePlatform:Windows #PARENT ProxyNode:VM_DMNTXC1 #CHILD errorLogLocation:C:\Program Files\Tivoli\TSM\baclient\dsmerror.VM_DM.log #PARENT ProxyNode:VM_DMNTXC1 #CHILD dmComputerName:TSMDMNTXC1 #PARENT ProxyNode:VM_DMNTXC1 #PARAM STATUS=success #PARAM STATUS=success #END RUN 146127 20220205222124459 #END TASK 146173 #INFO FMM16014I The return code is 0. #END C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts>vmcli.cmd -f inquire_detail -t TSM -q dmverify -d Datacenter01 -o VM_DMNTXC101 #TASK 146175 inquire_detail 20220205222132272 #RUN 146129 20220205222132287 #LANG en_IE #PARAM BACKEND=TSM #PARAM OPERATION_TYPE 6 #PHASE_COUNT 4 #PHASE PREPARE #PARAM BACKUP_TYPE=0 #PARAM TSM_SERVER_NAME=172.11.1.22 #PARAM TSM_SERVER_PORT=1500 #PARAM TSMCLI_NODE_NAME=VMCLI #PARAM VCENTER_NODE_NAME= #PARAM DATACENTER_NODE_NAME=Datacenter01 #PARAM OFFLOAD_HOST_NAME=VM_DMNTXC101 #PARAM PASSWORD_TYPE=CLINODE #PARAM TSM_OPTFILE=C:\Users\TEMP\AppData\Local\Temp\1\T4VD2B9.tmp #PARAM INPUT_FILE= #PARAM TRACEFILE= #PARAM TRACEFLAGS= #PARAM RUNID=146175 #PARAM SSL= #PARAM SSLREQUIRED= #PARAM SSLACCEPTCERTFROMSERV= #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DM #CHILD hladdress:tsmdm01.test.lu #PARENT peernode:VM_DM #CHILD lladdress:49204 #PARENT peernode:VM_DM #CHILD targetnode:Datacenter01 #PARENT peernode:VM_LOCAL_MP_WIN #CHILD hladdress:tsmdm01.test.lu #PARENT peernode:VM_LOCAL_MP_WIN #CHILD lladdress:49209 #PARENT peernode:VM_LOCAL_MP_WIN #CHILD targetnode:Datacenter01 #PARENT peernode:VM_LOCAL_MP_LNX #CHILD hladdress:tsmmntlispr.int.hs.lu #PARENT peernode:VM_LOCAL_MP_LNX #CHILD lladdress:46717 #PARENT peernode:VM_LOCAL_MP_LNX #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DM1 #CHILD hladdress:tsmdmux01.test.lu #PARENT peernode:VM_DM1 #CHILD lladdress:46694 #PARENT peernode:VM_DM1 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DM2 #CHILD hladdress:tsmdmux02.test.lu #PARENT peernode:VM_DM2 #CHILD lladdress:32871 #PARENT peernode:VM_DM2 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DMNTXC1 #CHILD hladdress:tsmdmntxc1.test.lu #PARENT peernode:VM_DMNTXC1 #CHILD lladdress:49723 #PARENT peernode:VM_DMNTXC1 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DMNTXC2 #CHILD hladdress:tsmdmntxc2.test.lu #PARENT peernode:VM_DMNTXC2 #CHILD lladdress:49713 #PARENT peernode:VM_DMNTXC2 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DMNTXC101 #CHILD hladdress:dmntxc101 #PARENT peernode:VM_DMNTXC101 #CHILD lladdress:33942 #PARENT peernode:VM_DMNTXC101 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DMNTXC202 #CHILD hladdress: #PARENT peernode:VM_DMNTXC202 #CHILD lladdress: #PARENT peernode:VM_DMNTXC202 #CHILD targetnode:Datacenter01 #PARENT peernode:VM_DMNTXC201 #CHILD hladdress:dmntxc201 #PARENT peernode:VM_DMNTXC201 #CHILD lladdress:38038 #PARENT peernode:VM_DMNTXC201 #ERROR ANS2821E The CAD service is not running for data mover node named 'VM_DMNTXC101'. #PARAM STATUS=error #PARAM STATUS=error #END RUN 146129 20220205222143663 #END TASK 146175 #INFO FMM16014I The return code is 2. #END
In my case the node has a bad hostname different from dns ! Be careful on hostname solved by TSM server Use always long name !
To force a VM to use VMware snapshots instead of VSS, without rebooting, first stop the following Windows services:
VMtools Volume Shadow Copy SQL Server VSS Writer /or Exchange VSS Writer
Restart the VMware tools installer and click on Modify
uninstall the VSS into the VMware tools
At the end restart the services that are marked as Automatic start
First delete all snapshots, try a consolidate on snapshots.
Else check in on the ESX folder where VM is located if you have bad files like pending snapshots, or old files and cleanup the directory (ask a VMware specialist). A better way is to use a storage VMotion, it will move all important files.
Permissions for backup and recovery using TSM for VE - DP for VMware
On Windows VM use VSS to bring applications in consistent mode. It's supported on Exchange and SQL.
On the datamover node, specify that SQL VM will use VSS snapshot
INCLUDE.VMTSMVSS exchange
On the command line, type:
dsmc set password -type=vmguest vmguestname AdminID AdminPswd
to store the guest VM password. The guest VM user must have permission to create Volume Shadow Copies and to truncate SQL Server logs.
Back up the VM by issuing the dsmc backup vm command.
Additionnaly keep the SQL logs on the VM, to do a more granular recovery of your SQL database.
INCLUDE.VMTSMVSS vmname OPTions=KEEPSqllog
To check in which mode is backup a VM : the Application protection type: VMware or TSM VSS
C:\Program Files\Tivoli\TSM\baclient>dsmc -optfile=dsm_DM.opt -asnode=datacenter1 tsm> q vm sqlserver2 -detail Query Virtual Machine for Full VM backup # Backup Date Mgmt Class Size Type A/I Virtual Machine --- ------------------- ---------- ----------- ------ --- --------------- 1 11/27/2013 09:07:49 STANDARD 58.59 GB IFFULL A sqlserver2 The size of this incremental backup: n/a The number of incremental backups since last full: 0 The amount of extra data: 0 The TSM objects fragmentation: 0 Backup is represented by: 374 TSM objects Application protection type: VMware VMDK[1]Label: Hard Disk 1 VMDK[1]Name: [DATASTORE-1] sqlserver2/sqlserver2.vmdk VMDK[1]Status: Protected tsm> q vm sqlserver -detail Query Virtual Machine for Full VM backup # Backup Date Mgmt Class Size Type A/I Virtual Machine --- ------------------- ---------- ----------- ------ --- --------------- 1 11/28/2013 02:43:12 STANDARD 58.59 GB IFFULL A sqlserver Size of this incremental backup: n/a Number of incremental backups since last full: 0 Amount of extra data: 0 TSM object fragmentation: 0 Backup is represented by: 258 TSM objects Application protection type: TSM VSS Application(s) protected: MS SQL 2008 VMDK[1]Label: Hard Disk 1 VMDK[1]Name: [sqlserver] sqlserver/sqlserver.vmdk VMDK[1]Status: Protected
http://www-01.ibm.com/support/docview.wss?uid=swg21647461
Hardware and Software Requirements: Data Protection for VMware Version 6.4.1
Be careful, no more that 7 virtual disks per virtual SCSI adapter if you plan to use VSS backups, else it will failed. You can add additional SCSI virtual adapters to add more disks. It's a VMware limitation http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1037071
Most common error can be found in
/opt/tivoli/tsm/tdpvmware/common/logs/vmcli.log /opt/tivoli/tsm/tdpvmware/tsmcli/bin64/tsmclierror.log
To enable the traces on SP4VE (vePlugin, vSphere API, HTML connection…)
/opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties
Add these files into dsm.opt, and start a backup:
traceflag vm enter exit options pid tid general tracefile vmbackup.trc
Edit also the file dsmvddk.opt to increase loglevel:
Enable VMware VDDK trivial tracing
# 0-quiet, 1-panic, 2-error, 3-warning, 4-info, 5-verbose, 6-trivia vixDiskLib.transport.LogLevel = "6" (valeur initiale: 3) vixDiskLib.nfc.LogLevel = "6" (valeur initiale: 0)
Pour l’activation des traces SP4VE (VE plug-in, vSphere API, HTML connection…)
/opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties
Most often option used (VE plug-in):
logger.ve.level = ERROR # logger.ve.level = TRACE
Check for trace file, maybe located in /vcsplugin.log:
find / -name vcsplugin.log -exec ls -l {} \;
As TDP for VE v6.4 has an installation problem, the Apache Derby database is corrupt. You need to delete it, it will be automatically recreated.
I reviewed the last elements you sent to us and I can see error like the following when you did “vmcli -f inquire_config”
#WARNING FMM16048W More than one valid TagType value was specified. The first value TagType: id:2, name:PARAM is used. #PARAM OPERATION_TYPE 4 With this symptom and the http error 500 you get with the plugins I can find a recent apar which match the problem IC92720 :
http://www-01.ibm.com/support/docview.wss?crawler=1&uid=swg1IC92720
Local fix :
The following steps will rebuild the derby database:
1) Stop the "Data Protection for VMware vCenter command-line interface" service 2) Stop the "IBM WebSphere Application Server - TSMVEplugin" service 3) Stop any active vSphere sessions you have open 4) Rename the derby folder under the \TDPVMware\VMwarePlugin directory to derbyold. 5) Restart the "Data Protection for VMware vCenter command-line interface" service. You should see a new derby directory created. 6) Restart the "IBM WebSphere Application Server - TSMVEplugin" service. 7) Try the plugin again.
ANS5228E (RC4370) A backup VM operation failed because VMMAXPARALLEL was reduced to 1 and the client still cannot obtain a server mount point Cause
Maximum number of mount points value not high enough
Diagnosing the problem
Review the NODENAME and VMMAXPARALLEL client options used with IBM Tivoli Storage Manager for Virtual Environments. In this case, the following were used :
nodename vmware01-vc1_dc1_dm1 vmmaxparallel 5
In this example, the VMMAXPARALLEL option is set to 5.
Run the QUERY NODE command for the node involved to verify the value of the “maximum mount points allowed”. For example :
tsm: MYSERVER> q node vmware01-vc1_dc1_dm1 f=d Node Name: vmware01-vc1_dc1_dm1 … Maximum Mount Points Allowed: 1
In this example, the “Maximum Mount Points Allowed” attribute is set to 1.
The “Maximum Mount Points Allowed” value for the node defined on the Tivoli Storage Manager server must be must be equal to or greater than the MMAXPARALLEL option used by the client.
In this example, the following command will adjust the value to 5.
update node vmware01-vc1_dc1_dm1 maxnummp=5
This option works together with the vmlimitperhost and vmlimitperdatastore options to optimize the performance of parallel backups, and to reduce the processor load that parallel backs can create on a host in the vSphere infrastructure.
VMware Change Block Tracking (CBT) support for content-aware backups
TSM Full VM backup will use VMware Change Block Tracking (CBT) support if available to enable content-aware (used-block only) backups. VMware CBT requires ESX 4.0 or later host (with virtual hardware 7). TSM will turn on CBT on the first TSM backup of the virtual machine. TSM Full VM backup does support virtual machines that do not support CBT. In this case both used and un-used areas of the disk will be backed up and a informational message will be logged to the dsmerror.log. The TSM GUI Backup dialog or command line
'dsmc show vm all' will display the CBT status.
tsm> show vm all Session established with server TSM1: Windows Server Version 6, Release 3, Level 4.0 Server date/time: 11/28/2013 13:25:09 Last access: 11/28/2013 12:05:06 Accessing as node: DATACENTER_PROD Show All virtual machines on VMware Server. (S) 1.vmName: exchange hostAddress: exchange.local.net tsmNodeName: exchange displayName: exchange ipAddress: 192.168.1.26 datacenter: Datacenter_prod hostSystem: esx01 guestFolder: Machine virtuelle dActectAce guestFullName: Microsoft WindowsA ServerA 2008A R2 (64A bits) altGuestName: guestId: windows7Server64Guest uuid: 42099bdd-03c4-4707-95c9-16b2c2d39e67, moref: vm-53 instance uuid: 5009157e-f23b-5692-6123-d80cc60746bd guestState: running connectionState: connected changeTracking: On vmHWversion: vmx-09 domainKeyword: all-windows domainSelected: Yes cluster: Cluster vApp: resourcePool: Resources VMDK[1]Label: 'Disque dur 1' (Hard Disk 1) VMDK[1]Name: '[EXCHANGE] exchange/exchange.vmdk' VMDK[1]Status: Included
CAUSE:
The CBT (Changed Block Tracking) feature is set to off in the VM settings
Run ‘show vm all’ in dsm client console to confirm the settings – changeTracking: Off
SOLUTION:
Connect to VCenter and right click the target VM (it must be power off), then choose Edit Settings > Options > Advanced/General > Configuration Parameters and set ctkEnabled = “TRUE”
For vSphere Client 5.0 and 5.1, the Data Protection for VMware vCenter plug-in host URL address must be set in your Internet Explorer trusted sites zone. In the Internet Explorer menu bar, go to Tools > Internet Options > Security > Trusted sites. Click Sites and add the host URL address. Make sure to apply your changes. For example: Add this website to the zone:http://myvctrmachine.xyzco.com
In addition, vSphere Client 5.1 requires that the Data Protection for VMware vCenter plug-in host URL address is set as an Internet Explorer compatibility view. In the Internet Explorer menu bar, go to Tools > Comptability View Settings. Add the host URL address to the Websites you've added to Compatibility View field. For vSphere Web Client 5.1, when you are using Internet Explorer on Windows Server 2008 or Windows 2012 Server, Enhanced Security Configuration (IE ESC) must be disabled. Go to the Server Manager and make sure that IE ESC is set to Off. On Windows Server 2008, if you start the Data Protection for VMware vCenter plug-in and then add the address as a trusted site, a JavaScript error denies continued access to the Data Protection for VMware vCenter plug-in. To resolve this issue, close and restart the Data Protection for VMware vCenter plug-in.
ANS1715E A filespace already exists for virtual machine (xxx), but with a different virtual machine UUID (421681ce-751c-5a38-d863-1212e0d86228) than the current virtual machine UUID (564d2250-900f-1420-34c6-40c831c9120c).
1° Solution (Not sure still avalaible in latest TDP VE)
Add the option in dsm.opt from TDP VE datamover
testflag VMBACKUP_UPDATE_UUID
2° Solution
Manually backup the VM with the option: VMBACKUPUPDATEGUID
dsmc backup VM vm1 -vmbackuptype=IFFull -VMBACKUPUPDATEGUID
Errors quiesced snapshot fails like:
Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE1) fails with winerror 1168. hr = 0x80070490, Element not found.
Can not create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine.
Failed to create snapshot for vmname, error -3960 (cannot quiesce virtual machine)
Power off the virtual machine. Log in to the vCenter Server or the ESXi/ESX host through the vSphere Client. Right-click the virtual machine and click Edit settings. Click the Options tab. Navigate to Advanced > General > Configuration Parameters. Add or modify the row disk.EnableUUID with the value FALSE. Click OK to save. Click OK to exit. Reboot the virtual machine for changes to take in effect.
Note: If this change is done through the command line using a text editor, the vim-cmd command to reload the vmx is enough to see the changes. For more information, see Reloading a vmx file without removing the Virtual machine from inventory (1026043).
Alternatively, un-register the virtual machine from the vCenter Server inventory. To un-register, right-click the virtual machine and click Remove from Inventory. Re-register the virtual machine back to the inventory.
Open the C:\ProgramData\VMware\VMware Tools\Tools.conf file in a text editor, such as Notepad. If the file does not exist, create it. Add these lines to the file:
[vmbackup] vss.disableAppQuiescing = true
Save and close the file. Restart the VMware Tools Service for the changes to take effect. Click Start > Run, type services.msc, and click OK. Right-click the VMware Tools Service and click Restart.
Enable tracing:
vixDiskLib.transport.LogLevel = “6”
vixDiskLib.nfc.LogLevel = "6"
(take a note of the original value to change it back after)
traceflag tid pid enter exit general vm tracefile tsmve.trc
Recreate the problem: Once tracing has been enabled, perform the operation that is failing. It can be a backup or a restore. To minimize the size of the trace, only perform the backup or restore of the VM that has the failure.
Once the problem has been recreated, please undo the changes done in step 1 and 2.
For the involved data mover and/or mount proxy node, run the following command :
dsmc query systeminfo -optfile="<path\datamover node client option file name>"
For example :
cd "C:\Program Files\Tivoli\TSM\baclient" dsmc query systeminfo -optfile=dsm.MYVE_DATACENTER_DM.opt
The command will generate the dsminfo.txt and dsmmsinfo.txt files.
VM BACKUPS CAN FAIL WITH ANS4174E & ANS1907E
Virtual machine backups can fail with the following error even when using “testflag GLTEMPNAMECLEANUP”.
ANS4174E Full VM backup of VMware Virtual Machine 'virtual machine name' failed with RC=45 mode=Incremental Forever - Incremental, target node name='IBM', data mover node name='IBM_DM'
Add the option into dsm.sys on linux or dsm.opt on a windows datamover
testflag GLTEMPNAMECLEANUP
To solve the path library issue, please apply the following steps:
1. Use the IBM-installed Java version, located at the Linux installation location for Java: export JAVA_HOME=/opt
/tivoli/tsm/tdpvmware/common/jre/jre
2.Set the relevant environment variables.
a. Ensure that the JAVA_HOME environment variable is exported correctly:
JAVA_HOME=<jre-or-jdk-install-dir>
b. Ensure that the PATH environment variable is exported correctly:
export PATH=$PATH:$JAVA_HOME/jre/bin
3.Then trace the backup, with the following steps: (The above Java failure is not related to Jackson files)
A-Enable the Vcsplugin tracing:
1.Edit the log4j.properties located in "/opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties" with the following options:
logger.ve.level = TRACE rootLogger.level =TRACE
B.Enable tracing for the backup:
1. Enable VDDK debug log level by editing in file C:\Program Files\Common Files\Tivoli\TSM\IBMVDDK\dsmvddk.opt to have :
vixDiskLib.transport.LogLevel = "6" vixDiskLib.nfc.LogLevel = "6"
2. Run the backup VM command with tracing options:
dsmc backup VM <vmname>–optfile=<datamover optfile name> -tracefile=c:\test_vm.trc -traceflags=service,vm > c:\test_vm.out dsmc show VM all –optfile=<datamover optfile name> -tracefile=c:\show_vm.trc -traceflags=service,vm > c:\show_vm.out
3. Generate a new dsminfo.txt for the data mover node
by running this command from the BAclient installation directory:
dsmc query systeminfo –optfile=<datamover optfile name> - Generates the ‘dsminfo.txt’ file
4. Please send :
dsminfo.txt c:\test_vm.out test_vm.trc
d.Vcsplugin.log located in “/opt/tivoli/tsm/client/ba/bin/”
c:\show_vm.out c:\show_vm.tr