OutOfMemoryError (OOM) is a common issue that we can see in any WebLogic Server environment.
Here are some guidelines you can follow to troubleshoot the issue:
1. Have you tried increasing the heap size?
This is the first thing you need to try to tune your environment, the issue will probably be fixed.
Here is a nice Document that explains memory management for hotspot JDK (it mostly apply to recent versions):
http://www.oracle.com/technetwork/java/javase/memorymanagement-whitepaper-150215.pdf
2. Have you tried the latest JDK of your version?
Even you are using a certified JDK, older versions have several bugs that have been found along the years.
These bugs can be related to GC algorithms, so trying to tune your JVM in an old version, you might not get the expected result.
Also, the latest releases include several security fixes you might wnat to have in your environment.
You can find all Oracle Java releases (including JRockit) here:
All Java SE Downloads on MOS (Doc ID 1439822.1)
3. Are you using 32 or 64 bit JVM ?
If your environment is certified with 64 bit JVM, always consider using 64 bit JVM for scalability.
4. What JVM and Operating System are you using?
Depending on if you are using Hotspot/JRockit/ other JDK and also depending on your OS you can define the way you collect, gclogs, heapdumps, histograms.
FAQ: JRockit and java.lang.OutOfMemoryError related issues (Doc ID 1290251.1)
How to Troubleshoot Java Heap Memory Leak - "java.lang.OutOfMemoryError: Java heap space" (Doc ID 1236373.1)
How to collect a heap dump? - https://blogs.oracle.com/LuzMestre/entry/how_to_collect_a_heap
5. I getting stuck threads and OutOfMemoryError, What to troubleshoot first?
You need to troubleshoot the OOM first. Frequent GC can cause huge GC pauses, then stuck threads can occur.
Resources for Garbage Collection Performance Tuning (Doc ID 1572109.1)
6. I am getting a JVM crash and OutOfMemoryError, What to troubleshoot first?
You need to troubleshoot the OOM. After an OOM, the JVM is unstable, so a JVM crash can occur.
Oracle WebLogic Server Support Pattern: Troubleshooting Out of Memory and Memory Leak Problems (Doc ID 877172.1)
Enjoy!