Symptoms
None
Randomly customer seeing their Exchange Plugin backups failing with the error:
2016/06/13-23:15:34.22300 [avexvss_snapup_assist] Root domain context 'DC=xyz1,DC=com', default naming context 'DC=xyz,DC=com'.
2016-06-13 19:16:00 avexvss Error : Connecting to remote server failed with the following error message : The WS-Management service cannot process the request. This user is allowed a maximum number of 25 concurrent shells, which has been exceeded. Close existing shells or raise the quota for this user. For more information, see the about_Remote_Troubleshooting Help topic.
2016-06-13 19:16:00 avexvss Error : Unable to initialize Powershell interface process -- cannot continue. 2016-06-13 19:16:00 avexvss Error : Unable to complete backup. At the same time ps_exec.log also shows the error as:
2016-06-13 19:16:00.213 ps_exec.exe Info :: ERROR: Connecting to remote server failed with the following error message : The WS-Management service cannot process the request. This user is allowed a maximum number of 25 concurrent shells, which has been exceeded. Close existing shells or raise the quota for this user. For more information, see the about_Remote_Troubleshooting Help topic. Customer is using iisreset /noforce command to reset the state so that new backups not impacted.
Cause
Engineering engaged via Escalation process:
Escalation 26591 - avexvss Exchange Backup randomly failing with Powershell limit exceeded errors
It was determined that in case of successful scenario where a PS session gets created without
failures, WinRM logs of the same shows ,
- When new-pssession is called, the WinRM calls WSMan API Initialize and WSMan Session initialize.
PS session get created.
- When remove-pssesion is called, the WinRM calls WSMan API Deinitialize and WSMan Session
Deinitialize(Closing WSMan Session completed successfully).
But during a failure case from the application logs found that,
- When new-pssession is called, the WinRM calls WSMan API Initialize and WSMan Session initialize.
PS session get created.
- When remove-pssesion is called, the WinRM calls WSMan API Deinitialize. But WSMan Session Deinitialize is not being called.
These were Microsoft API calls and they provided a solution as described below
Resolution
Steps to resolve this configuration issue were:
- Disable the MaxShellPerUser group policy in place.
- Upgrade the Windows Management Framework 2.0 to 3.0. The default version of windows management framework with windows 2008 r2 is 2.0. By upgrading, we make sure the remoteshell is upgraded and stabilize the WMF compared to older version. Details at: https://www.microsoft.com/en-in/download/details.aspx?id=3459
Additional Information
Referenced following links that provide useful leads related to this issue:
- http://blog.tiensivu.com/aaron/archives/1864-Exchange-2010-Beta-The-WS-Management-service-cannot-process-the-request.html
- https://social.technet.microsoft.com/Forums/exchange/en-US/c0c062d2-c52f-455a-92be-907e37fd1512/cant-open-management-shell-or-console-reached-limit-of-powershell-concurrent-connections?forum=exchange2010
- 176901 : Avamar: Exchange Backups failing due to PowerShell errors https://support.emc.com/kb/176901
- 186899 : Avmar: Exchange 2010 Public Folder PowerShell Execution Policy https://support.emc.com/kb/186899