This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| 
                    tsm:tsm_tdpvmware_pb [2021/08/13 14:00] manu [TDP connection errors]  | 
                
                    tsm:tsm_tdpvmware_pb [2024/03/22 13:47] (current) manu  | 
            ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Errors on TDP for VE ====== | ====== Errors on TDP for VE ====== | ||
| + | ===== Restore a VM in thin_provisionning ===== | ||
| + | Use the follwin flag in your command | ||
| + | <code> | ||
| + | -testflag=VMRESTOR_DONT_FORCE_THICK | ||
| + | </code> | ||
| + | |||
| + | ===== Main tracing for VE backups ===== | ||
| + | |||
| + | **Note:** In either of these sections a path to free disk space will be needed for tracing.  | ||
| + | |||
| + | Tracing of VM operations can be extensive. If the operation you are tracing processes a number of VMs, determine if the behavior can be duplicated on a smaller subset so that data collection is not cumbersome. | ||
| + | |||
| + | ==== For operations which are initiated via the Web user interface ==== | ||
| + | |||
| + | * Edit the dsmvddk.opt (in the BAclient folder). Change the following options to be 6: | ||
| + | |||
| + | vixDiskLib.transport.LogLevel = "6" | ||
| + | vixDiskLib.nfc.LogLevel = "6" | ||
| + | |||
| + | * Edit the optfile used by the datamover(in the BAclient folder). Add the following: | ||
| + | |||
| + | traceflag tid pid enter exit general vm | ||
| + | tracefile <Path with free disk space>/datamvr.trace | ||
| + | |||
| + | * Edit the vmcliprofile (/opt/tivoli/tsm/tdpvmware/common/scripts): | ||
| + | |||
| + | Change the VMCLI_TRACE value to YES | ||
| + | Remove the # from in front of VE_TRACE_FILE and VE_TRACE_FLAGS | ||
| + | |||
| + | * Recreate the problem experienced | ||
| + | * Reverse the changes made to prevent tracing from running | ||
| + | * Gather the following for review: | ||
| + | |||
| + | <Path with free disk space>/datamvr.trace | ||
| + | Entire /opt/Tivoli/tsm/tdpvmware/common/logs folder | ||
| + | Entire /opt/Tivoli/tsm/tdpvmware/tsmcli folder | ||
| + | dsmwebcl.log from BAclient folder | ||
| + | datamover dsminfo.txt file | ||
| + | |||
| + | https://www.ibm.com/support/pages/collecting-data-ibm-spectrum-protect-virtual-environments-data-protection-vmware-linux#generalgather | ||
| + | |||
| + | |||
| + | ==== For operations which are initiated manually from the command line, or during scheduled events ==== | ||
| + | |||
| + | * Edit the dsmvddk.opt(in the BAclient folder). Change the following options to be 6: | ||
| + | vixDiskLib.transport.LogLevel = "6" | ||
| + | vixDiskLib.nfc.LogLevel = "6" | ||
| + | * Edit the optfile used by the datamover(in the BAclient folder). Add the following: | ||
| + | traceflag tid pid enter exit general vm | ||
| + | tracefile <Path with free disk space>/datamvr.trace | ||
| + | * Recreate the problem experienced | ||
| + | * Reverse the changes made to prevent tracing from running | ||
| + | * Gather the following for review: | ||
| + | <Path with free disk space>/datamvr.trace | ||
| + | dsmwebcl.log from BAclient folder | ||
| + | datamover dsminfo.txt file | ||
| + | |||
| + | |||
| + | |||
| + | ===== Communication pb between datamover and Vcenter ===== | ||
| https://www.ibm.com/support/pages/known-issues-and-limitations-data-protection-vmware-v81#Data%20Mover%20Common%20Issues%20and%20Limitations | https://www.ibm.com/support/pages/known-issues-and-limitations-data-protection-vmware-v81#Data%20Mover%20Common%20Issues%20and%20Limitations | ||
| + | Can you provide the following data: | ||
| + | |||
| + | * Stop the Webserver service  | ||
| + | |||
| + | * Edit log4j file to have "logger.ve.level = ALL" is set in C:\Program Files\Tivoli\TSM\baclient\plugins\vcloudsuite\sdk\log4j.properties | ||
| + | |||
| + | * Restart the webserver service  | ||
| + | |||
| + | * Reproduce the error in the plugin (taking a note with the time-fram on which this error is reproduced) | ||
| + | |||
| + | * Then collect: | ||
| + | |||
| + | Vcsplugin.log located in the Baclient directory  | ||
| + | VEprofile folder, path : C:\IBM\SpectrumProtect\webserver\usr\servers\veProfile  | ||
| + | Screenshot for all the plugin tabs the vCenter host the following : /storage/vsphere-ui/IBMDataProtection/logs/log.txt (this exactly our log-Discussion between our plugin and webserver) | ||
| + | |||
| + | ===== Test a proxy node ===== | ||
| + | |||
| + | <cli prompt='>'> | ||
| + | C:\>cd C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts | ||
| + | |||
| + | C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts>vmcli.cmd -f inquire_detail -t TSM -q dmverify -d Datacenter01 -o VM_DMNTXC1 | ||
| + | #TASK 146173 inquire_detail 20220205222107802 | ||
| + | #RUN 146127 20220205222107818 | ||
| + | #LANG en_IE | ||
| + | #PARAM BACKEND=TSM | ||
| + | #PARAM OPERATION_TYPE 6 | ||
| + | #PHASE_COUNT 4 | ||
| + | #PHASE PREPARE | ||
| + | #PARAM BACKUP_TYPE=0 | ||
| + | #PARAM TSM_SERVER_NAME=172.11.1.22 | ||
| + | #PARAM TSM_SERVER_PORT=1500 | ||
| + | #PARAM TSMCLI_NODE_NAME=VMCLI | ||
| + | #PARAM VCENTER_NODE_NAME= | ||
| + | #PARAM DATACENTER_NODE_NAME=Datacenter01  | ||
| + | #PARAM OFFLOAD_HOST_NAME=VM_DMNTXC1 | ||
| + | #PARAM PASSWORD_TYPE=CLINODE | ||
| + | #PARAM TSM_OPTFILE=C:\Users\TEMP\AppData\Local\Temp\1\T4V7315.tmp | ||
| + | #PARAM INPUT_FILE= | ||
| + | #PARAM TRACEFILE= | ||
| + | #PARAM TRACEFLAGS= | ||
| + | #PARAM RUNID=146173 | ||
| + | #PARAM SSL= | ||
| + | #PARAM SSLREQUIRED= | ||
| + | #PARAM SSLACCEPTCERTFROMSERV= | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DM | ||
| + | #CHILD hladdress:tsmdm01.test.lu | ||
| + | #PARENT peernode:VM_DM | ||
| + | #CHILD lladdress:49204 | ||
| + | #PARENT peernode:VM_DM | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_LOCAL_MP_WIN | ||
| + | #CHILD hladdress:tsmdm01.test.lu | ||
| + | #PARENT peernode:VM_LOCAL_MP_WIN | ||
| + | #CHILD lladdress:49209 | ||
| + | #PARENT peernode:VM_LOCAL_MP_WIN | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_LOCAL_MP_LNX | ||
| + | #CHILD hladdress:linux01.lu | ||
| + | #PARENT peernode:VM_LOCAL_MP_LNX | ||
| + | #CHILD lladdress:46717 | ||
| + | #PARENT peernode:VM_LOCAL_MP_LNX | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DM1 | ||
| + | #CHILD hladdress:dm01.test.lu | ||
| + | #PARENT peernode:VM_DM1 | ||
| + | #CHILD lladdress:46694 | ||
| + | #PARENT peernode:VM_DM1 | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DM2 | ||
| + | ... | ||
| + | #CHILD vmhost: | ||
| + | #PARENT vmname: | ||
| + | #CHILD vmuuid: | ||
| + | #PARENT vmname: | ||
| + | #CHILD vmtools: | ||
| + | #PARENT vmname: | ||
| + | #CHILD vcenter:0 | ||
| + | #PARENT vmname: | ||
| + | #CHILD host:0 | ||
| + | #PARENT vmname: | ||
| + | #CHILD toolsVersion:0 | ||
| + | #PARENT vmname: | ||
| + | #CHILD proxyNodePlatform:Windows | ||
| + | #PARENT ProxyNode:VM_DMNTXC1 | ||
| + | #CHILD errorLogLocation:C:\Program Files\Tivoli\TSM\baclient\dsmerror.VM_DM.log | ||
| + | #PARENT ProxyNode:VM_DMNTXC1 | ||
| + | #CHILD dmComputerName:TSMDMNTXC1 | ||
| + | #PARENT ProxyNode:VM_DMNTXC1 | ||
| + | #PARAM STATUS=success | ||
| + | #PARAM STATUS=success | ||
| + | #END RUN 146127 20220205222124459 | ||
| + | #END TASK 146173 | ||
| + | #INFO FMM16014I The return code is 0. | ||
| + | #END | ||
| + | |||
| + | C:\Program Files\ibm\SpectrumProtect\Framework\VEGUI\scripts>vmcli.cmd -f inquire_detail -t TSM -q dmverify -d Datacenter01 -o VM_DMNTXC101 | ||
| + | #TASK 146175 inquire_detail 20220205222132272 | ||
| + | #RUN 146129 20220205222132287 | ||
| + | #LANG en_IE | ||
| + | #PARAM BACKEND=TSM | ||
| + | #PARAM OPERATION_TYPE 6 | ||
| + | #PHASE_COUNT 4 | ||
| + | #PHASE PREPARE | ||
| + | #PARAM BACKUP_TYPE=0 | ||
| + | #PARAM TSM_SERVER_NAME=172.11.1.22 | ||
| + | #PARAM TSM_SERVER_PORT=1500 | ||
| + | #PARAM TSMCLI_NODE_NAME=VMCLI | ||
| + | #PARAM VCENTER_NODE_NAME= | ||
| + | #PARAM DATACENTER_NODE_NAME=Datacenter01  | ||
| + | #PARAM OFFLOAD_HOST_NAME=VM_DMNTXC101 | ||
| + | #PARAM PASSWORD_TYPE=CLINODE | ||
| + | #PARAM TSM_OPTFILE=C:\Users\TEMP\AppData\Local\Temp\1\T4VD2B9.tmp | ||
| + | #PARAM INPUT_FILE= | ||
| + | #PARAM TRACEFILE= | ||
| + | #PARAM TRACEFLAGS= | ||
| + | #PARAM RUNID=146175 | ||
| + | #PARAM SSL= | ||
| + | #PARAM SSLREQUIRED= | ||
| + | #PARAM SSLACCEPTCERTFROMSERV= | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DM | ||
| + | #CHILD hladdress:tsmdm01.test.lu | ||
| + | #PARENT peernode:VM_DM | ||
| + | #CHILD lladdress:49204 | ||
| + | #PARENT peernode:VM_DM | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_LOCAL_MP_WIN | ||
| + | #CHILD hladdress:tsmdm01.test.lu | ||
| + | #PARENT peernode:VM_LOCAL_MP_WIN | ||
| + | #CHILD lladdress:49209 | ||
| + | #PARENT peernode:VM_LOCAL_MP_WIN | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_LOCAL_MP_LNX | ||
| + | #CHILD hladdress:tsmmntlispr.int.hs.lu | ||
| + | #PARENT peernode:VM_LOCAL_MP_LNX | ||
| + | #CHILD lladdress:46717 | ||
| + | #PARENT peernode:VM_LOCAL_MP_LNX | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DM1 | ||
| + | #CHILD hladdress:tsmdmux01.test.lu | ||
| + | #PARENT peernode:VM_DM1 | ||
| + | #CHILD lladdress:46694 | ||
| + | #PARENT peernode:VM_DM1 | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DM2 | ||
| + | #CHILD hladdress:tsmdmux02.test.lu | ||
| + | #PARENT peernode:VM_DM2 | ||
| + | #CHILD lladdress:32871 | ||
| + | #PARENT peernode:VM_DM2 | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DMNTXC1 | ||
| + | #CHILD hladdress:tsmdmntxc1.test.lu | ||
| + | #PARENT peernode:VM_DMNTXC1 | ||
| + | #CHILD lladdress:49723 | ||
| + | #PARENT peernode:VM_DMNTXC1 | ||
| + | #CHILD targetnode:Datacenter01 | ||
| + | #PARENT peernode:VM_DMNTXC2 | ||
| + | #CHILD hladdress:tsmdmntxc2.test.lu | ||
| + | #PARENT peernode:VM_DMNTXC2 | ||
| + | #CHILD lladdress:49713 | ||
| + | #PARENT peernode:VM_DMNTXC2 | ||
| + | #CHILD targetnode:Datacenter01 | ||
| + | #PARENT peernode:VM_DMNTXC101 | ||
| + | #CHILD hladdress:dmntxc101 | ||
| + | #PARENT peernode:VM_DMNTXC101 | ||
| + | #CHILD lladdress:33942 | ||
| + | #PARENT peernode:VM_DMNTXC101 | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | |||
| + | #PARENT peernode:VM_DMNTXC202 | ||
| + | #CHILD hladdress: | ||
| + | #PARENT peernode:VM_DMNTXC202 | ||
| + | #CHILD lladdress: | ||
| + | #PARENT peernode:VM_DMNTXC202 | ||
| + | #CHILD targetnode:Datacenter01  | ||
| + | #PARENT peernode:VM_DMNTXC201 | ||
| + | #CHILD hladdress:dmntxc201 | ||
| + | #PARENT peernode:VM_DMNTXC201 | ||
| + | #CHILD lladdress:38038 | ||
| + | #PARENT peernode:VM_DMNTXC201 | ||
| + | #ERROR ANS2821E The CAD service is not running for data mover node named 'VM_DMNTXC101'. | ||
| + | #PARAM STATUS=error | ||
| + | #PARAM STATUS=error | ||
| + | #END RUN 146129 20220205222143663 | ||
| + | #END TASK 146175 | ||
| + | #INFO FMM16014I The return code is 2. | ||
| + | #END | ||
| + | </cli> | ||
| + | |||
| + | FIXME In my case the node has a bad hostname different from dns ! Be careful on hostname solved by TSM server | ||
| + | Use always long name ! | ||
| ===== Disable VSS snapshots on Virtual Machine ===== | ===== Disable VSS snapshots on Virtual Machine ===== | ||
| Line 97: | Line 350: | ||
| ===== TDP connection errors ===== | ===== TDP connection errors ===== | ||
| - | Add these files into dsm.opt, and start a backup: | + | === On TDP VE client === | 
| + | |||
| + | Most common error can be found in | ||
| + | /opt/tivoli/tsm/tdpvmware/common/logs/vmcli.log | ||
| + | /opt/tivoli/tsm/tdpvmware/tsmcli/bin64/tsmclierror.log | ||
| + | |||
| + | To enable the traces on SP4VE (vePlugin, vSphere API, HTML connection…) | ||
| + | /opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties | ||
| + | |||
| + | |||
| + | Add these files into **dsm.opt**, and start a backup: | ||
| <cli> | <cli> | ||
| traceflag vm enter exit options pid tid general | traceflag vm enter exit options pid tid general | ||
| Line 103: | Line 366: | ||
| </cli> | </cli> | ||
| - | Edit also the file dsmvddk.opt to increase loglevel: | + | Edit also the file **dsmvddk.opt** to increase loglevel: | 
| Enable VMware VDDK trivial tracing | Enable VMware VDDK trivial tracing | ||
| Line 123: | Line 386: | ||
| </cli> | </cli> | ||
| - | Check for trace file, maybe located in /vcsplugin.log:  | + | Check for trace file, maybe located in **/vcsplugin.log**: | 
| <cli> | <cli> | ||
| find / -name vcsplugin.log -exec ls -l {} \; | find / -name vcsplugin.log -exec ls -l {} \; | ||
| Line 328: | Line 591: | ||
| Right-click the VMware Tools Service and click Restart.  | Right-click the VMware Tools Service and click Restart.  | ||
|  |  | ||
| - | ===== Trace TDP VE ===== | + | ===== Trace TDP VE 1 ===== | 
| Enable tracing: | Enable tracing: | ||
| Line 357: | Line 620: | ||
| The command will generate the dsminfo.txt and dsmmsinfo.txt files. | The command will generate the dsminfo.txt and dsmmsinfo.txt files. | ||
| + | ===== Trace TDP VE 2 ===== | ||
| + | |||
| + | VM BACKUPS CAN FAIL WITH ANS4174E & ANS1907E | ||
| + | |||
| + | Virtual machine backups can fail with the following error even | ||
| + | when using "testflag GLTEMPNAMECLEANUP". | ||
| + | |||
| + | ANS4174E Full VM backup of VMware Virtual Machine | ||
| + | 'virtual machine name' failed with RC=45 mode=Incremental Forever - Incremental, | ||
| + | target node name='IBM', data mover node name='IBM_DM' | ||
| + | |||
| + | Add the option into dsm.sys on linux or dsm.opt on a windows datamover | ||
| + | testflag GLTEMPNAMECLEANUP | ||
| + |  | ||
| + | ===== Trace TDP VE 3 ===== | ||
| + | |||
| + | To solve the path library  issue, please apply the following steps: | ||
| + | |||
| + | 1. Use the IBM-installed Java version, located at the Linux installation location for Java: export JAVA_HOME=/opt | ||
| + | <code> | ||
| + | /tivoli/tsm/tdpvmware/common/jre/jre | ||
| + | </code> | ||
| + | |||
| + | 2.Set the relevant environment variables. | ||
| + | |||
| + | a. Ensure that the JAVA_HOME environment variable is exported correctly: | ||
| + | <code> | ||
| + | JAVA_HOME=<jre-or-jdk-install-dir> | ||
| + | </code> | ||
| + | |||
| + | b. Ensure that the PATH environment variable is exported correctly: | ||
| + | <code> | ||
| + | export PATH=$PATH:$JAVA_HOME/jre/bin | ||
| + | </code> | ||
| + | |||
| + | 3.Then trace the backup,  with the following steps:  (The above Java failure is not related to Jackson files)  | ||
| + | |||
| + | A-Enable the Vcsplugin tracing: | ||
| + | |||
| + | 1.Edit the log4j.properties located in "/opt/tivoli/tsm/client/ba/bin/plugins/vcloudsuite/sdk/log4j.properties"  with the following options: | ||
| + | <code> | ||
| + | logger.ve.level = TRACE | ||
| + | rootLogger.level =TRACE | ||
| + | </code> | ||
| + | |||
| + | B.Enable tracing for the backup: | ||
| + | |||
| + | 1. Enable VDDK debug log level by editing in file C:\Program Files\Common Files\Tivoli\TSM\IBMVDDK\dsmvddk.opt to have : | ||
| + | <code> | ||
| + | vixDiskLib.transport.LogLevel = "6" | ||
| + | vixDiskLib.nfc.LogLevel = "6" | ||
| + | </code> | ||
| + | |||
| + | 2. Run the backup VM command with tracing options: | ||
| + | <code> | ||
| + | dsmc backup VM <vmname>–optfile=<datamover optfile name> -tracefile=c:\test_vm.trc -traceflags=service,vm > c:\test_vm.out | ||
| + | |||
| + | dsmc show VM all –optfile=<datamover optfile name> -tracefile=c:\show_vm.trc -traceflags=service,vm > c:\show_vm.out | ||
| + | </code> | ||
| + | |||
| + | 3. Generate a new dsminfo.txt for the data mover node | ||
| + | |||
| + | >> by running this command from the BAclient installation directory: | ||
| + | <code> | ||
| + | dsmc query systeminfo –optfile=<datamover optfile name> - Generates the ‘dsminfo.txt’ file | ||
| + | </code> | ||
| + | |||
| + | 4. Please send : | ||
| + | <code> | ||
| + | dsminfo.txt | ||
| + | c:\test_vm.out | ||
| + | test_vm.trc | ||
| + | </code> | ||
| + | |||
| + | d.Vcsplugin.log located in "/opt/tivoli/tsm/client/ba/bin/" | ||
| + | <code> | ||
| + | c:\show_vm.out | ||
| + | c:\show_vm.tr | ||
| + | </code> | ||