Symptoms
On the ACM server.log, the following error is observed:
2019-10-23 23:54:56,389 ERROR [pool-76-thread-19]-vi.ViJavaAccess: getAllTask() --> Manage entity not found with given name AVProxy mor type VirtualMachine
2019-10-23 23:54:56,390 ERROR [pool-76-thread-19]-vi.ViJavaAccess: getAllTask() --> Unable to find task for given entity. error Message VirtualMachine AVProxy is not present on vCenter server.
2019-10-23 23:54:56,392 ERROR [pool-76-thread-19]-avadapter.AvamarMcsdkUtil: deployProxyOnHost() Proxy Deployment Failed Unable to find task info from vCenter for AVProxy.
2019-10-23 23:54:56,392 ERROR [pool-76-thread-19]-avadapter.AvamarMcsdkUtil: deployProxy() Failed to deploy the Proxy. Reason: com.emc.vcedpa.common.exception.ApplianceException: Avamar proxy deployment failed.
From the Avamar server, the proxy deployment manager logs are checked:
2019-10-23T12:54:25.142-1100 DEBUG [pool-8-thread-8][ util.ProxyNetworkWaiter#waitForNetworking]: Waiting for proxy networking...
2019-10-23T12:54:30.175-1100 DEBUG [pool-8-thread-8][ util.ProxyNetworkWaiter#waitForNetworking]: IP address determined: x.x.x.x
2019-10-23T12:54:30.175-1100 DEBUG [pool-8-thread-8][ util.ProxyNetworkWaiter#HttpUrlConnection]: HTTPS detected
2019-10-23T12:54:30.191-1100 DEBUG [pool-8-thread-8][ util.ProxyNetworkWaiter#mServiceReachable]: CIM HEAD request response code 501
2019-10-23T12:54:30.191-1100 DEBUG [pool-8-thread-8][ util.ProxyNetworkWaiter#waitForNetworking]: Proxy CIM service is reachable
2019-10-23T12:54:30.220-1100 DEBUG [pool-8-thread-8][ worker.TaskWorker# updateProgress]: Updating task progress: 70
2019-10-23T12:54:30.221-1100 DEBUG [pool-8-thread-8][ events.VCSEvent# send]: Sending event: template: org.springframework.amqp.rabbit.core.RabbitTemplate@7edd5a com.emc.avamar.vmware.vcs.deploymanager.deploy.status_event: com.emc.avamar.vmware.events.VCSEvent@15b7ac98
2019-10-23T12:54:30.222-1100 DEBUG [pool-8-thread-8][ worker.TaskWorker#teTaskDescription]: Updating task description: Registering proxy...
2019-10-23T12:54:30.241-1100 INFO [pool-8-thread-8][ domain.Proxy# register]: Proxy hostname: localhost.localdom
2019-10-23T12:54:30.241-1100 WARN [pool-8-thread-8][ worker.DeployWorker# run]: Deploy failed. null
2019-10-23T12:54:30.241-1100 DEBUG [pool-8-thread-8][ worker.TaskWorker#teTaskDescription]: Updating task description:
2019-10-23T12:54:30.244-1100 INFO [pool-8-thread-8][ worker.TaskWorker# sendTaskError]: Sending fault com.emc.avamar.vmware.vcs.deploymanager.register_hostname_invalid
Cause
When using Linux DNS servers, port 53 should be enabled between the IDPA Proxy and the DNS server. TCP port 53 being blocked either on the DNS server or in network firewall can cause this issue. By default, this port is disabled on Linux based DNS servers.
From the output above, notice that proxy was not able to obtain the correct Hostname and hence, localhost.localdom was set as the hostname which caused the issue.
In a Proxy Deployment workflow, after the IP address gets assigned to the Avamar Proxy, it tries to reach the DNS servers to lookup the hostname for the same. After a successful reverse lookup from the DNS, it then assigns the same hostname to the Proxy. If, for some reason, proxy is unable to reach the DNS servers or if it does not get the response or incorrect response, the proxy hostname gets marked as localhost.local.dom.
This results in the proxy deployment failing and the proxy gets deleted as part of cleanup steps in PDM.
Resolution
Verify if port 53 is blocked between the IDPA Components and the DNS servers
Run the following commands to test if the server is blocked:
- Log in to the ACM CLI as 'root' user.
- Run the following commands:
curl -kv <DNS_Server_IP_Address>:53
The above output should show as Connected for that port. If the connection fails, that confirms the port is blocked.
host -W 10 -T <Avamar_Proxy_VM_IP_Address>
The above command should return the Hostname of Avamar Proxy IP if the DNS server is accessible and Proxy IP is registered in DNS server.
Additional Information
Some Additional Troubleshooting Steps are outlined below:
- Verify the ACM Files have correct DNS server updated:
cat /etc/resolv.conf
cat /usr/local/dataprotection/var/configmgr/server_data/config/commonconfig.xml
Note: Make sure /etc/resolv.conf contains search domain.
- All hostnames should not contain special characters like (_,@). All hostnames should be in lowercase.
- Verify the IP address is not duplicate. When the proxy is not deployed, perform a ping and arping to the IP address to detect a duplicate IP address.
ping <Avamar_Proxy_IP>
If you get a response back when proxy is not deployed, it confirms that this is a duplicate IP address.
Additional tests can be done using arping command:
arping -D <Avamar_Proxy_IP> -c 3
echo $?
PowerProtect DP4400, PowerProtect DP5300, PowerProtect DP5800, PowerProtect DP8300, PowerProtect DP8800, Integrated Data Protection Appliance Family, PowerProtect Data Protection Hardware, Integrated Data Protection Appliance Software