Is this a critical security issue?
Describe the Bug
Attempting to upgrade past JRuby 9.4.12.1 results in the OpenVox Server test suit failing due to OutOfMemory errors. This was discovered earlier in the year with an attempted upgrade to 9.4.14.0 and subsequent reversion in OpenVoxProject/openvox-server@78105ec.
git bisect has revealed that the issue started with a change to how JRuby handles proxies created to allow Ruby to access Java objects. This commit between the 9.4.12.1 and 9.4.13.0 tags is where the issue starts: jruby/jruby@19e3f6a
The good news is that we can "fix the glitch" by flipping the config bit back to ClassValue.Type.HARD_MAP from ClassValue.Type.STABLE.
However, the upstream change was designed to mitigate memory leaks and there has been a general malaise of increased memory usage hanging over 9.4. We should root cause exactly what is happening here in order to determine if flipping back to ClassValue.Type.HARD_MAP is truly the right fix, or whether there is something in the Puppet code base that is leaking memory and should be fixed.
Expected Behavior
Memory usage is stable after upgrading JRuby past 9.4.13.0. lein test :multithreaded completes in the same amount of time, or faster, as before the upgrade.
Steps to Reproduce
- Clone OpenVoxProject/openvox-server, check out the
8.x branch, and then run the :multithreaded test suite with the jruby-deps version overridden to 9.4.15.0-1:
lein update-in :managed-dependencies conj '[org.openvoxproject/jruby-deps "9.4.15.0-1"]' -- test :multithreaded
Environment
Version 5.4.1
Platform ANY
Additional Context
No response
Relevant log output
$ lein update-in :managed-dependencies conj '[org.openvoxproject/jruby-deps "9.4.15.0-1"]' -- test :multithreaded
Retrieving org/openvoxproject/jruby-deps/9.4.15.0-1/jruby-deps-9.4.15.0-1.pom from clojars
Retrieving org/jruby/jruby-base/9.4.15.0/jruby-base-9.4.15.0.pom from central
Retrieving org/jruby/jruby-parent/9.4.15.0/jruby-parent-9.4.15.0.pom from central
Retrieving com/github/jnr/jnr-enxio/0.32.20/jnr-enxio-0.32.20.pom from central
Retrieving com/github/jnr/jnr-unixsocket/0.38.25/jnr-unixsocket-0.38.25.pom from central
Retrieving com/github/jnr/jnr-posix/3.1.22/jnr-posix-3.1.22.pom from central
Retrieving com/github/jnr/jnr-ffi/2.2.19/jnr-ffi-2.2.19.pom from central
Retrieving com/github/jnr/jffi/1.3.15/jffi-1.3.15.pom from central
Retrieving org/jruby/joni/joni/2.2.5/joni-2.2.5.pom from central
Retrieving org/jruby/jcodings/jcodings/1.0.63/jcodings-1.0.63.pom from central
Retrieving org/jruby/jruby-stdlib/9.4.15.0/jruby-stdlib-9.4.15.0.pom from central
Retrieving org/jruby/jruby-base/9.4.15.0/jruby-base-9.4.15.0.jar from central
Retrieving com/github/jnr/jnr-posix/3.1.22/jnr-posix-3.1.22.jar from central
Retrieving com/github/jnr/jnr-enxio/0.32.20/jnr-enxio-0.32.20.jar from central
Retrieving com/github/jnr/jnr-unixsocket/0.38.25/jnr-unixsocket-0.38.25.jar from central
Retrieving com/github/jnr/jffi/1.3.15/jffi-1.3.15.jar from central
Retrieving com/github/jnr/jnr-ffi/2.2.19/jnr-ffi-2.2.19.jar from central
Retrieving com/github/jnr/jffi/1.3.15/jffi-1.3.15-native.jar from central
Retrieving org/jruby/joni/joni/2.2.5/joni-2.2.5.jar from central
Retrieving org/jruby/jcodings/jcodings/1.0.63/jcodings-1.0.63.jar from central
Retrieving org/jruby/jruby-stdlib/9.4.15.0/jruby-stdlib-9.4.15.0.jar from central
Retrieving org/openvoxproject/jruby-deps/9.4.15.0-1/jruby-deps-9.4.15.0-1.jar from clojars
lein test puppetlabs.general-puppet.general-puppet-int-test
lein test puppetlabs.puppetserver.auth-conf-test
lein test puppetlabs.puppetserver.bootstrap-int-test
lein test puppetlabs.puppetserver.certificate-authority-test
lein test puppetlabs.puppetserver.error-handling-int-test
lein test puppetlabs.puppetserver.ringutils-test
lein test puppetlabs.puppetserver.ruby.http-client-test
lein test puppetlabs.services.ca.certificate-authority-core-test
lein test puppetlabs.services.ca.certificate-authority-disabled-test
lein test puppetlabs.services.certificate-authority.certificate-authority-int-test
lein test puppetlabs.services.config.puppet-server-config-core-test
lein test puppetlabs.services.config.puppet-server-config-service-test
lein test puppetlabs.services.jruby.class-info-test
lein test :only puppetlabs.services.jruby.class-info-test/class-info-test
ERROR in (class-info-test) (:)
class info properly enumerated for changes to environments
Unexpected info retrieved for 'env2'
expected: (= (expected-envs-info "env2") (get-class-info-for-env "env2"))
actual: java.lang.OutOfMemoryError: Java heap space
at [empty stack trace]
lein test :only puppetlabs.services.jruby.class-info-test/class-info-test
ERROR in (class-info-test) (:)
class info properly enumerated for changes to environments
Unexpected info retrieved for 'env3'
expected: (= (expected-envs-info "env3") (get-class-info-for-env "env3"))
^C
Is this a critical security issue?
Describe the Bug
Attempting to upgrade past JRuby 9.4.12.1 results in the OpenVox Server test suit failing due to
OutOfMemoryerrors. This was discovered earlier in the year with an attempted upgrade to 9.4.14.0 and subsequent reversion in OpenVoxProject/openvox-server@78105ec.git bisecthas revealed that the issue started with a change to how JRuby handles proxies created to allow Ruby to access Java objects. This commit between the 9.4.12.1 and 9.4.13.0 tags is where the issue starts: jruby/jruby@19e3f6aThe good news is that we can "fix the glitch" by flipping the config bit back to
ClassValue.Type.HARD_MAPfromClassValue.Type.STABLE.However, the upstream change was designed to mitigate memory leaks and there has been a general malaise of increased memory usage hanging over 9.4. We should root cause exactly what is happening here in order to determine if flipping back to
ClassValue.Type.HARD_MAPis truly the right fix, or whether there is something in the Puppet code base that is leaking memory and should be fixed.Expected Behavior
Memory usage is stable after upgrading JRuby past 9.4.13.0.
lein test :multithreadedcompletes in the same amount of time, or faster, as before the upgrade.Steps to Reproduce
8.xbranch, and then run the:multithreadedtest suite with thejruby-depsversion overridden to9.4.15.0-1:Environment
Version 5.4.1
Platform ANY
Additional Context
No response
Relevant log output