This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
tsm:tsm_tdpvmware_pb [2021/08/13 14:02] manu [TDP connection errors] |
tsm:tsm_tdpvmware_pb [2024/03/22 13:47] (current) manu |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Errors on TDP for VE ====== | ====== Errors on TDP for VE ====== | ||
+ | ===== Restore a VM in thin_provisionning ===== | ||
+ | Use the follwin flag in your command | ||
+ | <code> | ||
+ | -testflag=VMRESTOR_DONT_FORCE_THICK | ||
+ | </code> | ||
+ | |||
+ | ===== Main tracing for VE backups ===== | ||
+ | |||
+ | **Note:** In either of these sections a path to free disk space will be needed for tracing. | ||
+ | |||
+ | Tracing of VM operations can be extensive. If the operation you are tracing processes a number of VMs, determine if the behavior can be duplicated on a smaller subset so that data collection is not cumbersome. | ||
+ | |||
+ | ==== For operations which are initiated via the Web user interface ==== | ||
+ | |||
+ | * Edit the dsmvddk.opt (in the BAclient folder). Change the following options to be 6: | ||
+ | |||
+ | vixDiskLib.transport.LogLevel = "6" | ||
+ | vixDiskLib.nfc.LogLevel = "6" | ||
+ | |||
+ | * Edit the optfile used by the datamover(in the BAclient folder). Add the following: | ||
+ | |||
+ | traceflag tid pid enter exit general vm | ||
+ | tracefile <Path with free disk space>/datamvr.trace | ||
+ | |||
+ | * Edit the vmcliprofile (/opt/tivoli/tsm/tdpvmware/common/scripts): | ||
+ | |||
+ | Change the VMCLI_TRACE value to YES | ||
+ | Remove the # from in front of VE_TRACE_FILE and VE_TRACE_FLAGS | ||
+ | |||
+ | * Recreate the problem experienced | ||
+ | * Reverse the changes made to prevent tracing from running | ||
+ | * Gather the following for review: | ||
+ | |||
+ | <Path with free disk space>/datamvr.trace | ||
+ | Entire /opt/Tivoli/tsm/tdpvmware/common/logs folder | ||
+ | Entire /opt/Tivoli/tsm/tdpvmware/tsmcli folder | ||
+ | dsmwebcl.log from BAclient folder | ||
+ | datamover dsminfo.txt file | ||
+ | |||
+ | https://www.ibm.com/support/pages/collecting-data-ibm-spectrum-protect-virtual-environments-data-protection-vmware-linux#generalgather | ||
+ | |||
+ | |||
+ | ==== For operations which are initiated manually from the command line, or during scheduled events ==== | ||
+ | |||
+ | * Edit the dsmvddk.opt(in the BAclient folder). Change the following options to be 6: | ||
+ | vixDiskLib.transport.LogLevel = "6" | ||
+ | vixDiskLib.nfc.LogLevel = "6" | ||
+ | * Edit the optfile used by the datamover(in the BAclient folder). Add the following: | ||
+ | traceflag tid pid enter exit general vm | ||
+ | tracefile <Path with free disk space>/datamvr.trace | ||
+ | * Recreate the problem experienced | ||
+ | * Reverse the changes made to prevent tracing from running | ||
+ | * Gather the following for review: | ||
+ | <Path with free disk space>/datamvr.trace | ||
+ | dsmwebcl.log from BAclient folder | ||
+ | datamover dsminfo.txt file | ||
+ | |||
+ | |||
+ | |||
+ | ===== Communication pb between datamover and Vcenter ===== | ||
https://www.ibm.com/support/pages/known-issues-and-limitations-data-protection-vmware-v81#Data%20Mover%20Common%20Issues%20and%20Limitations | https://www.ibm.com/support/pages/known-issues-and-limitations-data-protection-vmware-v81#Data%20Mover%20Common%20Issues%20and%20Limitations | ||
+ | Can you provide the following data: | ||
+ | |||
+ | * Stop the Webserver service | ||
+ | |||
+ | * Edit log4j file to have "logger.ve.level = ALL" is set in C:\Program Files\Tivoli\TSM\baclient\plugins\vcloudsuite\sdk\log4j.properties | ||
+ | |||
+ | * Restart the webserver service | ||
+ | |||
+ | * Reproduce the error in the plugin (taking a note with the time-fram on which this error is reproduced) | ||
+ | |||
+ | * Then collect: | ||
+ | |||
+ | Vcsplugin.log located in the Baclient directory | ||
+ | VEprofile folder, path : C:\IBM\SpectrumProtect\webserver\usr\servers\veProfile | ||
+ | Screenshot for all the plugin tabs the vCenter host the following : /storage/vsphere-ui/IBMDataProtection/logs/log.txt (this exactly our log-Discussion between our plugin and webserver) | ||
+ | |||
+ | ===== Test a proxy node ===== | ||
+ | |||
+ | <cli prompt='>'> | ||
+ | C:\>cd C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts | ||
+ | |||
+ | C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts>vmcli.cmd -f inquire_detail -t TSM -q dmverify -d Datacenter01 -o VM_DMNTXC1 | ||
+ | #TASK 146173 inquire_detail 20220205222107802 | ||
+ | #RUN 146127 20220205222107818 | ||
+ | #LANG en_IE | ||
+ | #PARAM BACKEND=TSM | ||
+ | #PARAM OPERATION_TYPE 6 | ||
+ | #PHASE_COUNT 4 | ||
+ | #PHASE PREPARE | ||
+ | #PARAM BACKUP_TYPE=0 | ||
+ | #PARAM TSM_SERVER_NAME=172.11.1.22 | ||
+ | #PARAM TSM_SERVER_PORT=1500 | ||
+ | #PARAM TSMCLI_NODE_NAME=VMCLI | ||
+ | #PARAM VCENTER_NODE_NAME= | ||
+ | #PARAM DATACENTER_NODE_NAME=Datacenter01 | ||
+ | #PARAM OFFLOAD_HOST_NAME=VM_DMNTXC1 | ||
+ | #PARAM PASSWORD_TYPE=CLINODE | ||
+ | #PARAM TSM_OPTFILE=C:\Users\TEMP\AppData\Local\Temp\1\T4V7315.tmp | ||
+ | #PARAM INPUT_FILE= | ||
+ | #PARAM TRACEFILE= | ||
+ | #PARAM TRACEFLAGS= | ||
+ | #PARAM RUNID=146173 | ||
+ | #PARAM SSL= | ||
+ | #PARAM SSLREQUIRED= | ||
+ | #PARAM SSLACCEPTCERTFROMSERV= | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DM | ||
+ | #CHILD hladdress:tsmdm01.test.lu | ||
+ | #PARENT peernode:VM_DM | ||
+ | #CHILD lladdress:49204 | ||
+ | #PARENT peernode:VM_DM | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_LOCAL_MP_WIN | ||
+ | #CHILD hladdress:tsmdm01.test.lu | ||
+ | #PARENT peernode:VM_LOCAL_MP_WIN | ||
+ | #CHILD lladdress:49209 | ||
+ | #PARENT peernode:VM_LOCAL_MP_WIN | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_LOCAL_MP_LNX | ||
+ | #CHILD hladdress:linux01.lu | ||
+ | #PARENT peernode:VM_LOCAL_MP_LNX | ||
+ | #CHILD lladdress:46717 | ||
+ | #PARENT peernode:VM_LOCAL_MP_LNX | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DM1 | ||
+ | #CHILD hladdress:dm01.test.lu | ||
+ | #PARENT peernode:VM_DM1 | ||
+ | #CHILD lladdress:46694 | ||
+ | #PARENT peernode:VM_DM1 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DM2 | ||
+ | ... | ||
+ | #CHILD vmhost: | ||
+ | #PARENT vmname: | ||
+ | #CHILD vmuuid: | ||
+ | #PARENT vmname: | ||
+ | #CHILD vmtools: | ||
+ | #PARENT vmname: | ||
+ | #CHILD vcenter:0 | ||
+ | #PARENT vmname: | ||
+ | #CHILD host:0 | ||
+ | #PARENT vmname: | ||
+ | #CHILD toolsVersion:0 | ||
+ | #PARENT vmname: | ||
+ | #CHILD proxyNodePlatform:Windows | ||
+ | #PARENT ProxyNode:VM_DMNTXC1 | ||
+ | #CHILD errorLogLocation:C:\Program Files\Tivoli\TSM\baclient\dsmerror.VM_DM.log | ||
+ | #PARENT ProxyNode:VM_DMNTXC1 | ||
+ | #CHILD dmComputerName:TSMDMNTXC1 | ||
+ | #PARENT ProxyNode:VM_DMNTXC1 | ||
+ | #PARAM STATUS=success | ||
+ | #PARAM STATUS=success | ||
+ | #END RUN 146127 20220205222124459 | ||
+ | #END TASK 146173 | ||
+ | #INFO FMM16014I The return code is 0. | ||
+ | #END | ||
+ | |||
+ | C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts>vmcli.cmd -f inquire_detail -t TSM -q dmverify -d Datacenter01 -o VM_DMNTXC101 | ||
+ | #TASK 146175 inquire_detail 20220205222132272 | ||
+ | #RUN 146129 20220205222132287 | ||
+ | #LANG en_IE | ||
+ | #PARAM BACKEND=TSM | ||
+ | #PARAM OPERATION_TYPE 6 | ||
+ | #PHASE_COUNT 4 | ||
+ | #PHASE PREPARE | ||
+ | #PARAM BACKUP_TYPE=0 | ||
+ | #PARAM TSM_SERVER_NAME=172.11.1.22 | ||
+ | #PARAM TSM_SERVER_PORT=1500 | ||
+ | #PARAM TSMCLI_NODE_NAME=VMCLI | ||
+ | #PARAM VCENTER_NODE_NAME= | ||
+ | #PARAM DATACENTER_NODE_NAME=Datacenter01 | ||
+ | #PARAM OFFLOAD_HOST_NAME=VM_DMNTXC101 | ||
+ | #PARAM PASSWORD_TYPE=CLINODE | ||
+ | #PARAM TSM_OPTFILE=C:\Users\TEMP\AppData\Local\Temp\1\T4VD2B9.tmp | ||
+ | #PARAM INPUT_FILE= | ||
+ | #PARAM TRACEFILE= | ||
+ | #PARAM TRACEFLAGS= | ||
+ | #PARAM RUNID=146175 | ||
+ | #PARAM SSL= | ||
+ | #PARAM SSLREQUIRED= | ||
+ | #PARAM SSLACCEPTCERTFROMSERV= | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DM | ||
+ | #CHILD hladdress:tsmdm01.test.lu | ||
+ | #PARENT peernode:VM_DM | ||
+ | #CHILD lladdress:49204 | ||
+ | #PARENT peernode:VM_DM | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_LOCAL_MP_WIN | ||
+ | #CHILD hladdress:tsmdm01.test.lu | ||
+ | #PARENT peernode:VM_LOCAL_MP_WIN | ||
+ | #CHILD lladdress:49209 | ||
+ | #PARENT peernode:VM_LOCAL_MP_WIN | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_LOCAL_MP_LNX | ||
+ | #CHILD hladdress:tsmmntlispr.int.hs.lu | ||
+ | #PARENT peernode:VM_LOCAL_MP_LNX | ||
+ | #CHILD lladdress:46717 | ||
+ | #PARENT peernode:VM_LOCAL_MP_LNX | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DM1 | ||
+ | #CHILD hladdress:tsmdmux01.test.lu | ||
+ | #PARENT peernode:VM_DM1 | ||
+ | #CHILD lladdress:46694 | ||
+ | #PARENT peernode:VM_DM1 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DM2 | ||
+ | #CHILD hladdress:tsmdmux02.test.lu | ||
+ | #PARENT peernode:VM_DM2 | ||
+ | #CHILD lladdress:32871 | ||
+ | #PARENT peernode:VM_DM2 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DMNTXC1 | ||
+ | #CHILD hladdress:tsmdmntxc1.test.lu | ||
+ | #PARENT peernode:VM_DMNTXC1 | ||
+ | #CHILD lladdress:49723 | ||
+ | #PARENT peernode:VM_DMNTXC1 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DMNTXC2 | ||
+ | #CHILD hladdress:tsmdmntxc2.test.lu | ||
+ | #PARENT peernode:VM_DMNTXC2 | ||
+ | #CHILD lladdress:49713 | ||
+ | #PARENT peernode:VM_DMNTXC2 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DMNTXC101 | ||
+ | #CHILD hladdress:dmntxc101 | ||
+ | #PARENT peernode:VM_DMNTXC101 | ||
+ | #CHILD lladdress:33942 | ||
+ | #PARENT peernode:VM_DMNTXC101 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | |||
+ | #PARENT peernode:VM_DMNTXC202 | ||
+ | #CHILD hladdress: | ||
+ | #PARENT peernode:VM_DMNTXC202 | ||
+ | #CHILD lladdress: | ||
+ | #PARENT peernode:VM_DMNTXC202 | ||
+ | #CHILD targetnode:Datacenter01 | ||
+ | #PARENT peernode:VM_DMNTXC201 | ||
+ | #CHILD hladdress:dmntxc201 | ||
+ | #PARENT peernode:VM_DMNTXC201 | ||
+ | #CHILD lladdress:38038 | ||
+ | #PARENT peernode:VM_DMNTXC201 | ||
+ | #ERROR ANS2821E The CAD service is not running for data mover node named 'VM_DMNTXC101'. | ||
+ | #PARAM STATUS=error | ||
+ | #PARAM STATUS=error | ||
+ | #END RUN 146129 20220205222143663 | ||
+ | #END TASK 146175 | ||
+ | #INFO FMM16014I The return code is 2. | ||
+ | #END | ||
+ | </cli> | ||
+ | |||
+ | FIXME In my case the node has a bad hostname different from dns ! Be careful on hostname solved by TSM server | ||
+ | Use always long name ! | ||
===== Disable VSS snapshots on Virtual Machine ===== | ===== Disable VSS snapshots on Virtual Machine ===== | ||
Line 98: | Line 351: | ||
=== On TDP VE client === | === On TDP VE client === | ||
+ | |||
+ | Most common error can be found in | ||
+ | /opt/tivoli/tsm/tdpvmware/common/logs/vmcli.log | ||
+ | /opt/tivoli/tsm/tdpvmware/tsmcli/bin64/tsmclierror.log | ||
+ | |||
+ | To enable the traces on SP4VE (vePlugin, vSphere API, HTML connection…) | ||
+ | /opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties | ||
+ | |||
Add these files into **dsm.opt**, and start a backup: | Add these files into **dsm.opt**, and start a backup: | ||
Line 330: | Line 591: | ||
Right-click the VMware Tools Service and click Restart. | Right-click the VMware Tools Service and click Restart. | ||
| | ||
- | ===== Trace TDP VE ===== | + | ===== Trace TDP VE 1 ===== |
Enable tracing: | Enable tracing: | ||
Line 359: | Line 620: | ||
The command will generate the dsminfo.txt and dsmmsinfo.txt files. | The command will generate the dsminfo.txt and dsmmsinfo.txt files. | ||
+ | ===== Trace TDP VE 2 ===== | ||
+ | |||
+ | VM BACKUPS CAN FAIL WITH ANS4174E & ANS1907E | ||
+ | |||
+ | Virtual machine backups can fail with the following error even | ||
+ | when using "testflag GLTEMPNAMECLEANUP". | ||
+ | |||
+ | ANS4174E Full VM backup of VMware Virtual Machine | ||
+ | 'virtual machine name' failed with RC=45 mode=Incremental Forever - Incremental, | ||
+ | target node name='IBM', data mover node name='IBM_DM' | ||
+ | |||
+ | Add the option into dsm.sys on linux or dsm.opt on a windows datamover | ||
+ | testflag GLTEMPNAMECLEANUP | ||
+ | | ||
+ | ===== Trace TDP VE 3 ===== | ||
+ | |||
+ | To solve the path library issue, please apply the following steps: | ||
+ | |||
+ | 1. Use the IBM-installed Java version, located at the Linux installation location for Java: export JAVA_HOME=/opt | ||
+ | <code> | ||
+ | /tivoli/tsm/tdpvmware/common/jre/jre | ||
+ | </code> | ||
+ | |||
+ | 2.Set the relevant environment variables. | ||
+ | |||
+ | a. Ensure that the JAVA_HOME environment variable is exported correctly: | ||
+ | <code> | ||
+ | JAVA_HOME=<jre-or-jdk-install-dir> | ||
+ | </code> | ||
+ | |||
+ | b. Ensure that the PATH environment variable is exported correctly: | ||
+ | <code> | ||
+ | export PATH=$PATH:$JAVA_HOME/jre/bin | ||
+ | </code> | ||
+ | |||
+ | 3.Then trace the backup, with the following steps: (The above Java failure is not related to Jackson files) | ||
+ | |||
+ | A-Enable the Vcsplugin tracing: | ||
+ | |||
+ | 1.Edit the log4j.properties located in "/opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties" with the following options: | ||
+ | <code> | ||
+ | logger.ve.level = TRACE | ||
+ | rootLogger.level =TRACE | ||
+ | </code> | ||
+ | |||
+ | B.Enable tracing for the backup: | ||
+ | |||
+ | 1. Enable VDDK debug log level by editing in file C:\Program Files\Common Files\Tivoli\TSM\IBMVDDK\dsmvddk.opt to have : | ||
+ | <code> | ||
+ | vixDiskLib.transport.LogLevel = "6" | ||
+ | vixDiskLib.nfc.LogLevel = "6" | ||
+ | </code> | ||
+ | |||
+ | 2. Run the backup VM command with tracing options: | ||
+ | <code> | ||
+ | dsmc backup VM <vmname>–optfile=<datamover optfile name> -tracefile=c:\test_vm.trc -traceflags=service,vm > c:\test_vm.out | ||
+ | |||
+ | dsmc show VM all –optfile=<datamover optfile name> -tracefile=c:\show_vm.trc -traceflags=service,vm > c:\show_vm.out | ||
+ | </code> | ||
+ | |||
+ | 3. Generate a new dsminfo.txt for the data mover node | ||
+ | |||
+ | >> by running this command from the BAclient installation directory: | ||
+ | <code> | ||
+ | dsmc query systeminfo –optfile=<datamover optfile name> - Generates the ‘dsminfo.txt’ file | ||
+ | </code> | ||
+ | |||
+ | 4. Please send : | ||
+ | <code> | ||
+ | dsminfo.txt | ||
+ | c:\test_vm.out | ||
+ | test_vm.trc | ||
+ | </code> | ||
+ | |||
+ | d.Vcsplugin.log located in "/opt/tivoli/tsm/client/ba/bin/" | ||
+ | <code> | ||
+ | c:\show_vm.out | ||
+ | c:\show_vm.tr | ||
+ | </code> |