Tuesday, 8 September 2020

IBM WebSphere weird error "ORB already created"; startup fails

 To day I got the mysterious error `ORB already created` while starting WebSphere Application server.

The error came twice, the first time; I had changed the host name of the machine without updating in WAS. The entry for localhost was removed in the hosts file.

The second one, I had a corrupted Trust Association Interceptor jar. That jar had some shaded dependency and the exact error is never known from WAS logs. Here's the stack trace in SystemOut.log

com.ibm.ws.exception.RuntimeError: ORB already created
	at com.ibm.ws.runtime.component.ORBImpl.start(ORBImpl.java:366)
	at com.ibm.ws.runtime.component.ContainerHelper.startComponents(ContainerHelper.java:540)
	at com.ibm.ws.runtime.component.ContainerImpl.startComponents(ContainerImpl.java:627)
	at com.ibm.ws.runtime.component.ContainerImpl.start(ContainerImpl.java:618)
	at com.ibm.ws.runtime.component.ServerImpl.start(ServerImpl.java:555)
	at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:312)
	at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:225)
	at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:698)
	at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
	at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:101)
	at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
	at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
	at org.eclipse.core.launcher.Main.run(Main.java:981)
	at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:422)
	at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:179)

SystemErr.log also gives another mysterious error that is not related at all:

[9/8/20 14:42:09:698 AST] 00000001 SystemErr     R com.ibm.websphere.management.exception.AdminException: java.lang.NullPointerException
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.management.MBeanFactoryImpl.deactivateMBean(MBeanFactoryImpl.java:1168)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.batch.endpointsensor.BatchSensorComponent.destroy(BatchSensorComponent.java:100)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.component.ContainerHelper.destroyComponent(ContainerHelper.java:341)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.component.ContainerHelper.destroyComponents(ContainerHelper.java:325)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.component.ContainerImpl.destroyComponents(ContainerImpl.java:607)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.component.ContainerImpl.destroy(ContainerImpl.java:598)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.component.ServerImpl.destroy(ServerImpl.java:501)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.WsServerImpl.bootServerContainer(WsServerImpl.java:321)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.WsServerImpl.start(WsServerImpl.java:225)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.WsServerImpl.main(WsServerImpl.java:698)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.runtime.WsServer.main(WsServer.java:59)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at java.lang.reflect.Method.invoke(Method.java:508)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.wsspi.bootstrap.WSLauncher.launchMain(WSLauncher.java:234)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.wsspi.bootstrap.WSLauncher.main(WSLauncher.java:101)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.wsspi.bootstrap.WSLauncher.run(WSLauncher.java:82)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at java.lang.reflect.Method.invoke(Method.java:508)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.equinox.internal.app.EclipseAppContainer.callMethodWithException(EclipseAppContainer.java:587)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:181)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at java.lang.reflect.Method.invoke(Method.java:508)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.launcher.Main.invokeFramework(Main.java:340)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at org.eclipse.core.launcher.Main.run(Main.java:981)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:422)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:179)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R Caused by: java.lang.NullPointerException
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	at com.ibm.ws.management.MBeanFactoryImpl.deactivateMBean(MBeanFactoryImpl.java:1141)
[9/8/20 14:42:09:714 AST] 00000001 SystemErr     R 	... 36 more

Not only does WebSphere hide the source error, it also gives misleading errors in the ffdc logs. It gives errors that indicates that some port are in use. I did run netstats and verified that errors are wrong. 

I hope this information would help someone save time when he gets this weird error. For me, I order to discover this error, I had to reinstall WebSphere and configure it step by step with restarts in order to know where the exact error is coming from. WebSphere Rocks!!

No comments:

Post a Comment