The below service plan event, mount phase took more than one hour and failed:
<DATE & TIME>,null,832bd9f8-bfe8-4f7e-a9bc-73f84d915865,MILE_000005,<AppSync Server>,Master Phase phase for sql completed with errors
<DATE & TIME>,Mount copy,7279880d-bb90-41c3-8ff9-6b7d79bf7854,MILE_000005,<AppSync Server>,Mount copy phase for sql completed with errors
<DATE & TIME>,Mount copy,7279880d-bb90-41c3-8ff9-6b7d79bf7854,HST_000011,<AppSync Server>,Mounting file systems on mount host <Mount Host>.
We see "
OutOfMemoryError" in server logs before the mount failure along with the needed amount of Heap memory and existing maximum amount.
<DATE & TIME> WARN [ServerService Thread Pool -- 76] [org.apache.activemq.artemis.core.server] [] [] AMQ222205: OutOfMemoryError possible! There are currently 30 addresses with a total max-size-bytes of 3,145,728,000 bytes, but the maximum memory available is 2,147,483,648 bytes.
<DATE & TIME> WARN [EJB default - 1] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] [<AppSync Server>] [] SQL Error: 0, SQLState: 57014
<DATE & TIME> ERROR [EJB default - 1] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] [<AppSync Server>] [] ERROR: canceling statement due to user request
<DATE & TIME> ERROR [EJB default - 1] [org.hibernate.internal.ExceptionMapperStandardImpl] [<AppSync Server>] [] HHH000346: Error during managed flush [org.hibernate.exception.GenericJDBCException: could not execute statement]
<DATE & TIME> ERROR [EJB default - 4] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] [] [] FATAL: terminating connection due to administrator command
<DATE & TIME> WARN [EJB default - 4] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] [] [] SQL Error: 0, SQLState: 08006
<DATE & TIME> ERROR [EJB default - 4] [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] [] [] An I/O error occurred while sending to the backend.
Service plan failed as AppSync requires more heap memory for the operation than is configured.
Increase the Java Heap Memory size (Xms and Xmx) value depending on the Memory available on AppSync server host and the amount needed.
Xms is the initial amount of Heap memory acquired.
Xmx is the maximum amount that can be acquired if need be.
There is a direct correlation for RAM used regarding the increase in Heap Memory.
In the above example, we see that 3G is needed, therefore setting a maximum amount to 4G resolves the issue.
To perform this change, follow the below steps:
- Log in to AppSync server host.
- Go to the location Installation Drive\EMC\AppSync\jboss\executive\ and edit the application-service.conf file to increase the heap memory value.
The default values are:
wrapper.java.additional.2=-Xms1024m
wrapper.java.additional.3=-Xmx2048m
If we change the value to double, it looks like the below:
wrapper.java.additional.2=-Xms2048m
wrapper.java.additional.3=-Xmx4096m
- Restart the AppSync services.