Question : Deployed successfully but failed on JVM restart

========================================================================
The following Applications will be deployed on CELL cell_webapp in staging :
========================================================================

APPLICATION     CLUSTER
=======         ===========
webapp_fadsys   webapp_fadsys

Please confirm Y/N: y
================================================================
Info: ---- Starting Deployments ...
Info: ---- Checking WAR file ...
Info: ---- webapp_fadsys.war file found in /www/websphere61/deployment/webapp_fadsys on wasdmstgoh1
-rwxrwxr-x   1 root     sbarney  11810621 Sep 29 22:21 /www/websphere61/deployment/webapp_fadsys/webapp_fadsys.war
Info: ---- Checking archive webapp_fadsys.war
Info: ---- WAR file webapp_fadsys.war is good
Info: ---- Creating EAR file webapp_fadsys.ear ...
status: creating /tmp/earmaker_13254/META-INF/application.xml
status: creating /tmp/earmaker_13254/META-INF/ibm-application-ext.xmi
status: creating /tmp/earmaker_13254/META-INF/ibm-application-bnd.xmi
status: building ... webapp_fadsys.ear
added manifest
adding: webapp_fadsys.war(in = 11810621) (out= 11754513)(deflated 0%)
ignoring entry META-INF/
adding: META-INF/application.xml(in = 460) (out= 259)(deflated 43%)
adding: META-INF/ibm-application-ext.xmi(in = 300) (out= 149)(deflated 50%)
adding: META-INF/ibm-application-bnd.xmi(in = 510) (out= 246)(deflated 51%)
status: created /www/websphere61/deployment/webapp_fadsys/webapp_fadsys.ear
Info: ---- EAR file webapp_fadsys.ear is created Successfully
-rw-r--r--   1 root     root     11756090 Sep 30 15:23 /www/websphere61/deployment/webapp_fadsys/webapp_fadsys.ear
Info: ---- Checking archive webapp_fadsys.ear
Info: ---- EAR file webapp_fadsys.ear is good
---------------------------------------------------------------------
Info: ---- Wed Sep 30 15:23:39 EDT 2009 deployment of application webapp_fadsys started on cluster webapp_fadsys
Info: ---- Stopping cluster webapp_fadsys
        Info: ---- Stopping JVM webapp_fadsys_1 on server wasbrdstgoh1
        Info: ---- Command: restart.sh stop webapp_fadsys_1 /opt/WebSphere/AppServer61_webapp/profiles/node_webapp
##############################################################
Stopping webapp_fadsys_1 in /opt/WebSphere/AppServer61_webapp/profiles/node_webapp on wasbrdstgoh1 ...
##############################################################
Process name WSFADSYS
status: Taking thread dump for 14063
Command: /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/bin/stopServer.sh webapp_fadsys_1 -trace
ADMU0116I: Tool information is being logged in file
           /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/logs/webapp_fadsys_1/stopServer.log
Process WSFADSYS with pid=14063 is still running
ADMU0128I: Starting tool with the node_webapp profile
ADMU3100I: Reading configuration for server: webapp_fadsys_1
Process WSFADSYS with pid=14063 is still running
Process WSFADSYS with pid=14063 is still running
ADMU3201I: Server stop request issued. Waiting for stop status.
Process WSFADSYS with pid=14063 is still running
Process WSFADSYS with pid=14063 is stopped
Info: ---- Deleting transaction logs
Info: ---- Command: ./wsadmin.sh -lang jython -f /opt/WebSphere/scripts/was61/tmp/deployApp.py webapp_fadsys webapp_fadsys /www/websphere61/deployment/webapp_fadsys/webapp_fadsys.ear webapp_fadsys_host No_sh_lib /opt/WebSphere/AppServer61_webapp/installedApps SINGLE
WASX7209I: Connected to process "dmgr" on node dmgr_webapp using SOAP connector;  The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[webapp_fadsys, webapp_fadsys, /www/websphere61/deployment/webapp_fadsys/webapp_fadsys.ear, webapp_fadsys_host, No_sh_lib, /opt/WebSphere/AppServer61_webapp/installedApps, SINGLE]"
Verifying cluster name webapp_fadsys
Info: ---  Cluster webapp_fadsys is found for cell cell_webapp
Info: ---  Application webapp_fadsys is found for cell cell_webapp
Info: ---  Virtual Host webapp_fadsys_host is found for cell cell_webapp
Info: ---  Mapping virtial host to web modules
Info: ---  WAS61: Mapping FADS to webapp_fadsys_host
Info: ---  Virtual host mapping : ["FADS" webapp_fadsys.war,WEB-INF/web.xml webapp_fadsys_host]
Info: ---  Application webapp_fadsys will be installed on Cluster webapp_fadsys
Info: ---  Stopping the Cluster webapp_fadsys
Info: ---  Cluster webapp_fadsys is STOPPED
Info: ---  Uninstalling application webapp_fadsys
ADMA5017I: Uninstallation of webapp_fadsys started.
ADMA5104I: The server index entry for WebSphere:cell=cell_webapp,node=wasbrdstgoh1_webapp is updated successfully.
ADMA5102I: The configuration data for webapp_fadsys from the configuration repository is deleted successfully.
ADMA5011I: The cleanup of the temp directory for application webapp_fadsys is complete.
ADMA5106I: Application webapp_fadsys uninstalled successfully.
Info: ---  Installing application webapp_fadsys
Info: ---  Deployment options: -cluster webapp_fadsys -installed.ear.destination /opt/WebSphere/AppServer61_webapp/installedApps -appname webapp_fadsys -deployejb -deployws -MapWebModToVH [["FADS" webapp_fadsys.war,WEB-INF/web.xml webapp_fadsys_host]]
ADMA5016I: Installation of webapp_fadsys started.
ADMA5058I: Application and module versions are validated with versions of deployment targets.
ADMA5018I: The EJBDeploy command is running on enterprise archive (EAR) file /var/tmp/app47528.ear.
Starting workbench.
EJB Deploy configuration directory: /opt/WebSphere/AppServer61_webapp/profiles/dmgr_webapp/ejbdeploy/configuration/
framework search path: /opt/WebSphere/AppServer61_webapp/deploytool/itp/plugins
Debug options:
Creating the project.
Writing output file
Shutting down workbench.
EJBDeploy complete.
0 Errors, 0 Warnings, 0 Informational Messages
ADMA5007I: The EJBDeploy command completed on /opt/WebSphere/AppServer61_webapp/profiles/dmgr_webapp/wstemp/wstemp/app_1240c6b42eb/dpl/dpl_webapp_fadsys.ear
WSWS0041I: Web services deploy task completed successfully.
ADMA5005I: The application webapp_fadsys is configured in the WebSphere Application Server repository.
ADMA5053I: The library references for the installed optional package are created.
ADMA5005I: The application webapp_fadsys is configured in the WebSphere Application Server repository.
ADMA5001I: The application binaries are saved in /opt/WebSphere/AppServer61_webapp/profiles/dmgr_webapp/wstemp/Script1240c6a02e2/workspace/cells/cell_webapp/applications/webapp_fadsys.ear/webapp_fadsys.ear
ADMA5005I: The application webapp_fadsys is configured in the WebSphere Application Server repository.
SECJ0400I: Successfuly updated the application webapp_fadsys with the appContextIDForSecurity information.
ADMA5011I: The cleanup of the temp directory for application webapp_fadsys is complete.
ADMA5013I: Application webapp_fadsys installed successfully.
Info: ---  Application webapp_fadsys is found for cell cell_webapp
Info: ---  Application webapp_fadsys is installed successfully on cell cell_webapp
Info: ---  Classloader policy set to SINGLE
Info: ---  Saving configuration ...
Info: ---  Performing Node sync ...
Info: ---  Force NodeSync on WebSphere:name=nodeSync,process=nodeagent,platform=common,node=wasbrdstgoh1_webapp,diagnosticProvider=true,version=6.1.0.19,type=NodeSync,mbeanIdentifier=nodeSync,cell=cell_webapp,spec=1.0
Info: ---  Waiting for 10 sec to make sure sync is completed ...
Info: ---- Return code from WAS is 0
====================================================================================
Info: ---- Deployment of application webapp_fadsys on cluster webapp_fadsys is Successful
====================================================================================
Info: ---- jvms for cell cell_webapp will be started after deployment
Info: ---- Starting cluster webapp_fadsys
        Info: ---- Starting JVM webapp_fadsys_1 on server wasbrdstgoh1
        Info: ---- Command: restart.sh start webapp_fadsys_1 /opt/WebSphere/AppServer61_webapp/profiles/node_webapp
##############################################################
Starting webapp_fadsys_1 in /opt/WebSphere/AppServer61_webapp/profiles/node_webapp on wasbrdstgoh1 ...
##############################################################
Process name WSFADSYS
Info: ---- Changing permissions on /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/config/temp /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/temp /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/wstemp /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/logs/ffdc /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/logs/webapp_fadsys_1 /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/configuration/org.eclipse.osgi
Info: ---- Deleting transaction logs
ADMU0116I: Tool information is being logged in file
           /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/logs/webapp_fadsys_1/startServer.log
ADMU0128I: Starting tool with the node_webapp profile
ADMU3100I: Reading configuration for server: webapp_fadsys_1
ADMU0111E: Program exiting with error:
           com.ibm.ws.process.exception.OutOfMemoryException: PROC0009E: The
           operating system reported not enough free memory to create the
           process.
ADMU0211I: Error details may be seen in the file:
           /opt/WebSphere/AppServer61_webapp/profiles/node_webapp/logs/webapp_fadsys_1/startServer.log
Info: ---- Wed Sep 30 15:29:24 EDT 2009 deployment of application webapp_fadsys is completed on cluster webapp_fadsys


Can SomeBody Help Please

Answer : Deployed successfully but failed on JVM restart

Well, this tells you what happened:

--------------------------------------------------
ADMU0111E: Program exiting with error:
           com.ibm.ws.process.exception.OutOfMemoryException: PROC0009E: The
           operating system reported not enough free memory to create the
           process.
--------------------------------------------------

  Did you change your application in a way that causes it to allocate memory differently?

  I suggest that you use some of the tools on alphaWorks (http://www.alphaworks.ibm.com/) to help understand what is happening:

- If you enable verbose garbage collection (verbosegc), then this tools can help:
  IBM Pattern Modeling and Analysis Tool for Java Garbage Collector
  http://www.alphaworks.ibm.com/tech/pmat
- I don't think that this tool would help you much
  IBM Thread and Monitor Dump Analyzer for Java
  http://www.alphaworks.ibm.com/tech/jca
- This tool tends to be more useful when the JVM has been running awhile
  HeapAnalyzer
  http://www.alphaworks.ibm.com/tech/heapanalyzer

  From the stuff above though, it looks like the application is quickly allocating lots of large objects, and dieing quite quickly...