The [Link].
OutOfMemoryError:
PermGen Space error demystified
Edward Chou
Frank Kieviet
SOA/BI
Sun Microsystems
BOF-9982
2007 JavaOneSM Conference | Session BOF-9982 |
Goal
What we'll tell you:
[Link]: PermGen Space:
What is it?
What can you do about it?
2007 JavaOneSM Conference | Session BOF-9982 | 2
A confusing error...
● Out of memory? I have plenty of memory!
● Complexity
● Classes
● Classloaders
● Container code / application code
● Application server at fault?
2007 JavaOneSM Conference | Session BOF-9982 | 3
An often queried error...
● Blog
● About 200 hits per day
● > 50% on entry on memory leaks
● > 80% through Google queries
2007 JavaOneSM Conference | Session BOF-9982 | 4
Agenda
● Basics of memory, GC
● Basics of Classloaders
● Basics of Leaks
● How to fix?
● Demo of tools to find leaks
2007 JavaOneSM Conference | Session BOF-9982 | 5
The basics: garbage collection
void f() {
Object x = new Object();
}
In method f()
After exiting method f()
x GC!
2007 JavaOneSM Conference | Session BOF-9982 | 6
Objects in memory
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 7
Objects in memory
Roots
Reference chain
Root objects
- Bootstrap ClassLoader
- native references
- stack frames
Objects
Reachable
Unreachable
2007 JavaOneSM Conference | Session BOF-9982 | 8
12
GC: first mark all objects as unreachable
Roots
Objects
(simplified model)
2007 JavaOneSM Conference | Session BOF-9982 | 9
11
GC: next, mark reachable objects as reachable
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 10
10
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 11
9
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 12
8
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 13
7
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 14
6
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 15
5
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 16
4
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 17
3
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 18
2
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 19
1
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 20
0
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 21
Lastly GC all unreachable objects
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 22
Memory leaks
● Objects are accidentally referenced
● Objects accumulate
2007 JavaOneSM Conference | Session BOF-9982 | 23
The basics: memory generations
● Young generation
● Newly created objects
● Tenured generation
● Objects that survived several rounds of GC
● Permgen space (permanent generation)
● Class objects
● [Link]()
2007 JavaOneSM Conference | Session BOF-9982 | 24
The basics: classes and classloaders
● Each object is an instance of a class
● A class itself is an object (class object)
● instance of the class Class
● Each class references its classloader
● A classloader references all classes it loaded
● Class objects hold static members
[Link]
statics
* 1
*
[Link] 1 [Link]
*
2007 JavaOneSM Conference | Session BOF-9982 | 25
Why classloaders?
● Containers use classloaders to
● dynamically load applications
● isolate applications from eachother
● dynamically unload applications
2007 JavaOneSM Conference | Session BOF-9982 | 26
Example: empty servlet
package [Link];
import [Link].*;
import [Link].*;
import [Link].*;
import [Link].*;
public class Servlet1 extends HttpServlet {
private static final String STATICNAME = "Simple";
protected void doGet(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
// nothing
}
}
2007 JavaOneSM Conference | Session BOF-9982 | 27
Deployed
Undeployed
2007 JavaOneSM Conference | Session BOF-9982 | 28
Classloader leaks
● A classloader cannot be GC-ed if any of the
instances of any of the classes it loaded are
reachable.
● Such a classloader keeps all its classes with all
their static members in memory.
● Not immediately apparent from a memory dump what
is a leak and what is not.
● Cause of the leak difficult to find.
2007 JavaOneSM Conference | Session BOF-9982 | 29
Example: a leaking servlet
package [Link];
import [Link].*;
import [Link].*;
import [Link].*;
import [Link].*;
public class LeakServlet extends HttpServlet {
private static final String STATICNAME = "Leak!";
static Level CUSTOMLEVEL = new Level("OOPS", 555) {};
protected void doGet(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException {
// Log at a custom level
[Link]("test").log(CUSTOMLEVEL, "x called");
}
}
2007 JavaOneSM Conference | Session BOF-9982 | 30
Deployed
2007 JavaOneSM Conference | Session BOF-9982 | 31
[Link] class
private static List known = new ArrayList();
protected Level(String name, int value) {
[Link] = name;
[Link] = value;
synchronized ([Link]) {
[Link](this);
}
}
2007 JavaOneSM Conference | Session BOF-9982 | 32
A leak...
Undeployed
2007 JavaOneSM Conference | Session BOF-9982 | 33
Reality:
● Hundreds of leaked classes
● Thousands of leaked objects
● Bafflement...
2007 JavaOneSM Conference | Session BOF-9982 | 34
Profilers
Profilers...
● Take memory snapshots
● Find reference chains to root objects
● Most see class objects as root objects
2007 JavaOneSM Conference | Session BOF-9982 | 35
Profilers find reference chains
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 36
Profilers may not find reference chains...
Roots
Objects
Class object
2007 JavaOneSM Conference | Session BOF-9982 | 37
JHat
● A tool that reads hprof memory dumps
● Comes with JDK 6
2007 JavaOneSM Conference | Session BOF-9982 | 38
Steps to find the cause of a memory leak:
● Trigger a heap dump
● -XX:+HeapDumpOnOutOfMemoryError
● -Xrunhprof:heap=dump,format=b
● jmap -dump:format=b,file=[Link] <pid>
● Analyze heap dump using Jhat
● jhat -J-mx1024m [Link]
● [Link]
● Using built-in or custom queries to narrow down leak
suspects
● Identify an object or class in the application
● List reference chains
2007 JavaOneSM Conference | Session BOF-9982 | 39
DEMO
A real life example
2007 JavaOneSM Conference | Session XXXX | 40
4
Multiple reference chains
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 41
3
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 42
2
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 43
1
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 44
0
Roots
Objects
2007 JavaOneSM Conference | Session BOF-9982 | 45
[Link] class
private ClassLoader contextClassLoader;
private AccessControlContext inheritedAccessControlContext;
private void init(ThreadGroup g, Runnable target,
String name, long stackSize) {
[Link] = [Link];
...
...
[Link] =
[Link]();
}
2007 JavaOneSM Conference | Session BOF-9982 | 46
[Link] class
private static ClassLoader contextClassLoader;
static {
contextClassLoader = (ClassLoader)
[Link](
new [Link]() {
public Object run() {
return
[Link]().getContextClassLoader();
}
});
};
2007 JavaOneSM Conference | Session BOF-9982 | 47
[Link] class
private static [Link] writeDrivers =
new [Link]();
public static synchronized void registerDriver(
[Link] driver) throws SQLException {
...
DriverInfo di = new DriverInfo();
[Link] = driver;
[Link] = [Link]();
[Link] = [Link]();
[Link](di);
...
}
2007 JavaOneSM Conference | Session BOF-9982 | 48
Common problems
● Dangling thread
● Commons logging
● [Link]
● Thread context classloader
● new Thread()
● Bean util
● Details
● SQL Driver
2007 JavaOneSM Conference | Session BOF-9982 | 49
Determining the leaked classloader(s)
● Use a known class as a marker
● Does not work in systems with many classloaders
● Identify classloaders with identical sets of classes
● e.g. 10 classloaders out of 100 classloaders load
classes {A, B, C, D}
● Identify “orphaned” classloaders
● i.e. no reference chains to classloader besides classes
2007 JavaOneSM Conference | Session BOF-9982 | 50
“Orphaned” classloader
2007 JavaOneSM Conference | Session BOF-9982 | 51
Summary
● jhat is an effective tool to track down classloader
leaks
● Approach:
● Obtain a dump
● Identify suspect classloader
● Identify a known class
● Duplicate sets of classes
● “Orphaned” classloaders
● Trace reference chains
2007 JavaOneSM Conference | Session BOF-9982 | 52
Q&A
[Link]
[Link]
This picture © 2007 Software Test and Performance Magazine
2007 JavaOneSM Conference | Session XXXX | 53
Optional Demo
● How to write new jhat query?
● Object Query Language (OQL)
● Java
2007 JavaOneSM Conference | Session BOF-9982 | 54