Jenkins java.lang.OutOfMemoryError: PermGen space - Facing frequently even after increasing the perm memory

Question

I get the following error very often in Jenkins and as suggested by other answers I restarted Jenkins (Running on Solaris) by increasing the MaxPermSize parameter.

Initially when I faced the error it was 128 MB, I increased it to 512. There were no issues for some weeks. Then again got the error hence increased it to 1024. In a few weeks, same error occurred which made me to increase it again to 2048M. I did not add any extra jobs or processes during this time. Within a week of increasing it to 2048m I'm again facing the issue. A simple restart of jenkins resolves it, but it re occurs soon. What else can be done on this? Is there any way to check the current memory occupied or to really find whether the Perm space is running out in a short time.

The Error:

javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException:          
    jar:file:/proj/ecms-build/www/jenkins/war/WEB-INF/lib/jenkins-core-  1.609.2.jar!/hudson/model/Run/console.jelly:65:27: <j:whitespace> PermGen space
    at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:103)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.commons.jelly.JellyTagException: jar:file:/proj/ecms-build/www/jenkins/war/WEB-INF/lib/jenkins-core-1.609.2.jar!/hudson/model/Run/console.jelly:65:27: <j:whitespace> PermGen space
    at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:745)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:289)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
    at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
    at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
    at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
    at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
    at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
    ... 69 more
Caused by: java.lang.OutOfMemoryError: PermGen space

Solution tried:

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx4096m -XX:MaxPermSize=2048m"

Answer 1

Maybe this one helps as this sounds very familiar to me:

https://stackoverflow.com/a/37664040/5158269

I had to update the JRE for Jenkins itself from 7 to 8. My OutOfMemoryError-Error started, when I have upgraded to Jenkins 2.X and used Java7.

Answer 2

PermGen is used by the JVM to keep the Class objects themselves. This usually happens when some script (looks like JellyScript might be the candidate) is dynamically converting the script into byte code to execute. I do not know much about Jenkins, so I can't provide specific suggestions.

1. Check whether the module that is using JellyScript has an option for turning off the compilation mode. Most scripting languages provide interpret and compile mode.
2. See whether you hive off this task into a separate executable and use it from Jenkins.
3. Just restart the server when no one is watching :(