Puppet Server known issues

For information about current known issues, see the following list.

Hiera eyaml lookups fail during Puppet Server catalog compilation

Puppet Server fails to compile catalogs for environments that have eyaml-encrypted Hiera data, returning an error that the hiera_eyaml library is unavailable, although the library is installed.

Copy 
Notice: Requesting catalog from pointed-risk...:8140
Error: Could not retrieve catalog from remote server: Error 500 on SERVER:
  Server Error: Evaluation Error: Error while evaluating a Function Call,
  Function Load Error for function 'eyaml_lookup_key': Lookup using eyaml
  lookup_key function is only supported when the hiera_eyaml library is present

This is caused by an incompatible base64 dependency: the vendored hiera-eyaml 5.0.0 requires base64 ~> 0.3.0, while Puppet Server provides version 0.2.0.

Client-side commands such as puppet lookup and the eyaml CLI continue to work because they use the Puppet agent Ruby environment. PA-9165

Workaround

As a workaround, install hiera-eyaml 5.0.1.

  1. Install hiera-eyaml 5.0.1 into the Puppet Server JRuby gem home:

    Copy 
    sudo /opt/puppetlabs/server/bin/puppetserver gem install hiera-eyaml -v 5.0.1 --no-document

  2. Restart Puppet Server:

    Copy 
    sudo systemctl restart puppetserver

EC private keys in SEC1 format can cause catalog compilation failures

Affected versions:

  • Puppet Core 8.19.0 with Puppet Server 8.9.8 and later versions of Puppet Core 8
  • Puppet Core 9.0.0 with Puppet Server 9.0.2

Not affected:

  • Puppet Core 8.18.0 and earlier with Puppet Server 8.9.7 and earlier

These releases ship an earlier version of Bouncy Castle and are not affected by this issue.

Issue

In affected versions of Puppet Core, Puppet Server can fail to load elliptic curve (EC) private keys that are stored in SEC1 format (-----BEGIN EC PRIVATE KEY-----).

This issue is caused by an incompatibility between the version of Bouncy Castle shipped in affected releases and the version of jruby-openssl bundled with Puppet Server. As a result, Puppet Server can fail to load SEC1-format EC private keys during catalog compilation.

This issue can occur when server-side code, such as a custom Puppet function, attempts to load an EC private key by using OpenSSL functionality. When the key is loaded, Puppet Server can return a NoSuchMethodError, causing catalog compilation to fail and preventing agents from retrieving catalogs.

This issue affects SEC1-encoded EC private keys. PKCS#8-encoded EC private keys (-----BEGIN PRIVATE KEY-----) are not affected.

Example error:

Copy 
Error: Could not retrieve catalog from remote server: Error 500 on SERVER:
Internal Server Error:
java.lang.NoSuchMethodError: 'org.bouncycastle.asn1.ASN1Primitive
org.bouncycastle.asn1.sec.ECPrivateKey.getParameters()'

PUPDOC-6102

Workarounds

Use one of the following workarounds:

Reformat EC private keys to PKCS#8

Convert SEC1-format EC private keys to PKCS#8 format:

Copy 
openssl pkcs8 -topk8 -nocrypt -in sec1.pem -out pkcs8.pem

This workaround requires no Puppet Server changes and remains in place after upgrades.

Move EC key parsing to the agent

Move EC key parsing from a server-side function to an agent-side implementation, such as:

  • A custom fact
  • A deferred function

Agent-side code runs outside Puppet Server's JRuby environment and is not affected by this issue.

Override the bundled jruby-openssl gem

Install the newer version of the jruby-openssl gem on every Puppet Server that compiles catalogs — the primary server and any compilers. If you run a single Puppet Server, apply it there. Then restart Puppet Server service.

  1. Install a newer version of the gem.

    Copy 
    sudo /opt/puppetlabs/bin/puppetserver gem install jruby-openssl -v 0.16.0 --platform java

  2. Restart Puppet Server on the primary server and any compilers:

    Copy 
    sudo systemctl restart puppetserver

This workaround is manual and unsupported. It must be re-applied on the primary server and any compilers after Puppet Core upgrades.

Resolution

A future release will resolve this issue by updating the version of jruby-openssl bundled with Puppet Server.

Potential JAVA ARGS settings

If you're working outside of lab environment, increase ReservedCodeCache to 512m under normal load. If you're working with 6-12 JRuby instances (or a max-requests-per-instance value significantly less than 100k), run with a ReservedCodeCache of 1G. Twelve or more JRuby instances in a single server might require 2G or more.

Similar caveats regarding scaling ReservedCodeCache might apply if you are managing MaxMetaspace.

tmp directory mounted noexec

In some cases, if the /tmp directory is mounted as noexec, Puppet Server might fail to run correctly, and you might see an error in the Puppet Server logs similar to the following:

Copy 
Nov 12 17:46:12 fqdn.com java[56495]: Failed to load feature test for posix: can't find user for 0
Nov 12 17:46:12 fqdn.com java[56495]: Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32
Nov 12 17:46:12 fqdn.com java[56495]: Puppet::Error: Cannot determine basic system flavour

This is caused by JRuby containing some embedded files which need to be copied somewhere on the filesystem before they can be executed (see this JRuby issue). To work around this issue, you can either mount the /tmp directory without noexec, or you can choose a different directory to use as the temporary directory for the Puppet Server process.

Either way, you need to set the permissions of the directory to 1777. This allows the Puppet Server JRuby process to write a file to /tmp and then execute it. If permissions are set incorrectly, you get a massive stack trace without much useful information in it.

To use a different temporary directory, you can set the following JVM property:

Copy 
-Djava.io.tmpdir=/some/other/temporary/directory

When Puppet Server is installed from packages, add this property to the JAVA_ARGS and JAVA_ARGS_CLI variables defined in either /etc/sysconfig/puppetserver or /etc/default/puppetserver, depending on your distribution. Invocations of the gem, ruby, and irb subcommands use the updated JAVA_ARGS_CLI on their next invocation. The service needs to be restarted in order to re-read the JAVA_ARGS variable.