OutOfMemoryError:PermGen
Wed Jun 04 11:24:00 CDT 2008
Our tests started getting slow, taking a long time to execute, and then they started failing with OutOfMemoryError's. That this started happening with our adoption of Hibernate was not a coincidence. This is not to say that Hibernate is at fault. The PermGen ClassLoader problem with the Sun JVM is now known enough.
In short, the Sun JVM loads Classloader's and Classes directly into the PermGen or permanent generation. This collection of objects is for objects which, in theory, would never need unloading and therefore they are not checked during the sweep. I think the idea is that the gc sweep will go faster if the gc does not bother checking objects that will not ever be eligible. JRockit does not do this, so the PermGen problem will not manifest itself there, though, in the long term, memory leaks around class loaders will eventually show up.
TPTP - Testing and Profiling in Eclipse - Abandoned
I never did get this working, but I made the most progress I ever have on it. While I got some execution time measuring working, I never was able to get memory profiling working interactively. I managed to get the run to dump a stand-alone allocation trace file to disk that was 2G long. I gave up getting Eclipse to import it.
Getting this far involved
- emerging the Compatibility C++ and libc5 and libc6 libraries:
- downloading the stand-alone configuration agent,
- fixing its configuration so it will run on linux (the config script should do this but it does not) and
- Setting and exporting TPTP_AC_HOME
- Setting and exporting JAVA_PROFILER_HOME
- updating the LD_LIBRARY_PATH so that the jvm invocation will be able to load the agent.
All this and my Eclipse install (Version: 3.3.2 Build id: M20080221-1800) and TPTP (TPTP Core Version: 4.4.100.v200709030100-8Q82EeRJEPilTjotXnOr2 Build id: TPTP-4.4.1-200802230953) still would not show me heap usage or allocation hot spots. I lost hours to this.
NetBeans 6.1 - Memory profiling in 5 minutes
The longest part of installing NetBeans 6.1 is was downloading the install, something larger than 100M, but after that, I was up and profiling in about 5 minutes. That is including running into the need to do the calibration step.
I fed the JVM args through ANT_OPTS
which held execution until I connected to NetBeans. The real-time heap usage displays were more explanatory than a heap snapshot at the end.
Memory allocation from the OS for heap sky-rocketed up to the 1G mark set in ANT_OPTS but heap usage remained low. Even with visible dips in heap usage as the result of GC, the JVM continued to allocate more memory, but appeared to not use it. Here is a picture of the illusion.
The appearance of all the allegedly unused heap is frustrating given the inevitable demise of the test run: OutOfMemoryError:PermGen. Let's look at what else NetBeans profiling tells us:
Here we can see a chart that appears to claim that 100% of the allocated objects are in surviving generations.
Finally we see a chart claiming that we have loaded some 23,000 to 24,000 classes
We have 5460 java files in our code base and I'll have to do something else to determine how many distinct classes we pull in from all of our jars.
(I was using 1.5.0_15.)
There are several articles and blog entries about the problem, and describe running into it with the SunJVM, Hibernate, Spring, Tomcat, JBoss. The explanations also explain why switching to JRockit appears to fix it (it merely slows the real problem down if you do have a real classloader leak).
Links: * Google Search * How to enable PermGen sweeping. Comments include good practices for setting up Tomcat with JDBC drivers. * Hubris: * http://www.jroller.com/agileanswers/entry/preventingjavasjavalang * hibernate forum * java forum * spring
Good explanations * Frank Kieviet * Frank Kieviet follow up
Fixes for our problem
Forking for all the junit tests really slows it down. If we can find a better balance, perhaps forking for each batch of tests, that may work.
The other thing we may be able to do is to configure Hibernate in a different way. We could give it an explicit list of classes to inspect for annotations instead of having it inspect all available classes. This iteration through all available classes takes a long time anyway, never mind flooding PermGen with multiple instances of classes (if that is even this problem).