Code Cache flushing issue from JRE 1.6 to JRE 1.7 - Performance degradation due to JIT stop compiling

During the past few days I have been fighting with an issue happening after migrating from JRE 1.6 (1.6.0_30) to JRE 1.7 (1.7.0_65). 

I want to share this experience with you, so that (hopefully) you don’t have to deal with it ;)

User Experience:

After some time the application was running, it started becoming slow, though the CPU load was low. The user experienced a latency in terms of responsiveness of the component.

Analysis on Code Cache:

After some research and internal discussions with colleagues, I started concentrating on the JIT compilation and the Code Cache:

•  JRE 1.7 enable by default the Code Cache flushing, which should (theoretically) improve performance of your application, since it dispose old JITted code in favour of new one (speculative flushing).

Unfortunately, the Code Cache flushing seems not to work properly in Java 1.7 (while in Java 1.6 it works fine)

I started profiling my application (using JConsole: Memory->Code Cache), monitoring the Code Cache (which default size is 48MB, -XX:ReservedCodeCacheSize=48M) and I have discovered that my application uses the entire Code Cache heap.

•         JRE 1.6: after Code Cache fills, the bytecode compilation will stop (by default) switching into interpreted mode.
•         JRE 1.7: after Code Cache fills, Code Cache flushing starts, marking older methods to be potentially eligible for flushing in favor of newer methods. Actually it seems that after the Code Cache flushing, JIT compilation stops at all causing degradation in performance (neither old or new code is compiled anymore)

Since the Code Cache flushing is disabled by default in jre 1.6, I have enabled it and performed a parallel test (for comparison) with both jre 1.6 and jre 1.7.
Here are the jvm flags used for the test:
   -XX:ReservedCodeCacheSize=48M (default)

   -XX:+UseCodeCacheFlushing (default only in jre 1.7)

Below you can find a test executed with jre 1.6 and jre 1.7:

Note: I have seen that in jre 1.7, after sometimes the code compilation starts again (after reaching a very low level, for example 3MB) 

JRE 1.8:

This issue is not experienced using jre 1.8, in which the flushing algorithm seems to be changed: the flushing starts before getting the code cache full.

Temporary solution:

A workaround for this problem would be increasing the Code Cache size and be sure the Code Cache flushing does not happen . In my application the Code Cache size average seems to be 55MB, so I solved by setting it to 80MB.

The following JVM option can be used:

   -XX:ReservedCodeCacheSize=80M

To enable the Code Cache flushing in JRE 1.6 you can set the following option:

   -XX:+UseCodeCacheFlushing

Solution:

At the end, it seems to be related to the following issue, and fixed in JRE 1.8 (hs25):

http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8012547

I think this is a big compatibility issue as it comes out when migrating from jre6 to jre 7. 

Question: This fix has been done on Apr-13 on jre 1.8. Why at the present (Jul-14) this fix has never been ported into jre 1.7?

I have contacted Oracle and opened a bug report for this to be sure this issue is related to the known one.

References:

Some reference to understand how the Code Cache works can be found here:

1. http://www.oraclejavamagazine-digital.com/javamagazine_open/20130708?pg=42#pg42
2. http://blogs.atlassian.com/2012/05/codecache-is-full-compiler-has-been-disabled/
3. https://blog.codecentric.de/en/2012/07/useful-jvm-flags-part-4-heap-tuning/