Puppet Server known issues

For information about current known issues, see the following list.

EC private keys in SEC1 format can cause catalog compilation failures

Affected versions:

  • Puppet Core 8.19.0 with Puppet Server 8.9.8 and later versions of Puppet Core 8
  • Puppet Core 9.0.0 with Puppet Server 9.0.2

Not affected:

  • Puppet Core 8.18.0 and earlier with Puppet Server 8.9.7 and earlier

These releases ship an earlier version of Bouncy Castle and are not affected by this issue.

Issue

In affected versions of Puppet Core, Puppet Server can fail to load elliptic curve (EC) private keys that are stored in SEC1 format (-----BEGIN EC PRIVATE KEY-----).

This issue is caused by an incompatibility between the version of Bouncy Castle shipped in affected releases and the version of jruby-openssl bundled with Puppet Server. As a result, Puppet Server can fail to load SEC1-format EC private keys during catalog compilation.

This issue can occur when server-side code, such as a custom Puppet function, attempts to load an EC private key by using OpenSSL functionality. When the key is loaded, Puppet Server can return a NoSuchMethodError, causing catalog compilation to fail and preventing agents from retrieving catalogs.

This issue affects SEC1-encoded EC private keys. PKCS#8-encoded EC private keys (-----BEGIN PRIVATE KEY-----) are not affected.

Example error:

Copy 
Error: Could not retrieve catalog from remote server: Error 500 on SERVER:
Internal Server Error:
java.lang.NoSuchMethodError: 'org.bouncycastle.asn1.ASN1Primitive
org.bouncycastle.asn1.sec.ECPrivateKey.getParameters()'

Workarounds

Use one of the following workarounds:

Reformat EC private keys to PKCS#8

Convert SEC1-format EC private keys to PKCS#8 format:

Copy 
openssl pkcs8 -topk8 -nocrypt -in sec1.pem -out pkcs8.pem

This workaround requires no Puppet Server changes and remains in place after upgrades.

Move EC key parsing to the agent

Move EC key parsing from a server-side function to an agent-side implementation, such as:

  • A custom fact
  • A deferred function

Agent-side code runs outside Puppet Server's JRuby environment and is not affected by this issue.

Override the bundled jruby-openssl gem

Install the newer version of the jruby-openssl gem on every Puppet Server that compiles catalogs — the primary server and any compilers. If you run a single Puppet Server, apply it there. Then restart Puppet Server service.

  1. Install a newer version of the gem.

    Copy 
    sudo /opt/puppetlabs/bin/puppetserver gem install jruby-openssl -v 0.16.0 --platform java

  2. Restart Puppet Server on the primary server and any compilers:

    Copy 
    sudo systemctl restart puppetserver

This workaround is manual and unsupported. It must be re-applied on the primary server and any compilers after Puppet Core upgrades.

Resolution

A future release will resolve this issue by updating the version of jruby-openssl bundled with Puppet Server.

Access CA endpoint to update CRLs

Puppet Server 7.2.0 and 6.16.0 include the following API endpoint: PUT /puppet-ca/v1/certificate_revocation_list. To update this endpoint, you must update the endpoint's rule to be type regex instead of path in the configuration file at /etc/puppetlabs/puppetserver/conf.d/auth.conf.

Cipher updates in Puppet Server 6.5

Puppet Server 6.5 includes an upgrade to the latest release of Jetty's 9.4 series. With this update, you may see "weak cipher" warnings about ciphers that were previously enabled by default. Puppet Server now defaults to stronger FIPS-compliant ciphers, but you must first remove the weak ciphers.

The ciphers previously enabled by default have not been changed but are considered weak by the updated standards. Remove the weak ciphers by removing the cipher-suite configuration section from the webserver.conf. After you remove the cipher-suite, Puppet Server uses the FIPS-compliant ciphers instead. This release includes the weak ciphers for backward compatibility only.

The FIPS-compliant cipher suites, which are not considered weak, will be the default in a future version of Puppet. To maintain backwards compatibility, Puppet Server explicitly enables all cipher suites that were available as of Puppet Server 6.0. When you upgrade to Puppet Server 6.5.0, this affects you in in two ways:

  1. The 6.5 package updates the webserver.conf file in Puppet Server's conf.d directory.
  2. When Puppet Server starts or reloads, Jetty warns about weak cipher suites being enabled.

This update also removes the so-linger-seconds configuration setting. This setting is now ignored and a warning is issued if it is set. See Jetty's so-linger-seconds for removal details.

Note: On some older operating systems, you might see additional warnings that newer cipher suites are unavailable. In this case, manage the contents of the webserver.cipher-suites configuration value to be those strong suites that available to you.

Potential JAVA ARGS settings

If you're working outside of lab environment, increase ReservedCodeCache to 512m under normal load. If you're working with 6-12 JRuby instances (or a max-requests-per-instance value significantly less than 100k), run with a ReservedCodeCache of 1G. Twelve or more JRuby instances in a single server might require 2G or more.

Similar caveats regarding scaling ReservedCodeCache might apply if users are managing MaxMetaspace.

tmp directory mounted noexec

In some cases (especially for RHEL 7 installations) if the /tmp directory is mounted as noexec, Puppet Server may fail to run correctly, and you may see an error in the Puppet Server logs similar to the following:

Copy 
Nov 12 17:46:12 fqdn.com java[56495]: Failed to load feature test for posix: can't find user for 0
Nov 12 17:46:12 fqdn.com java[56495]: Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32
Nov 12 17:46:12 fqdn.com java[56495]: Puppet::Error: Cannot determine basic system flavour

This is caused by the fact that JRuby contains some embedded files which need to be copied somewhere on the filesystem before they can be executed (see this JRuby issue). To work around this issue, you can either mount the /tmp directory without noexec, or you can choose a different directory to use as the temporary directory for the Puppet Server process.

Either way, you'll need to set the permissions of the directory to 1777. This allows the Puppet Server JRuby process to write a file to /tmp and then execute it. If permissions are set incorrectly, you'll get a massive stack trace without much useful information in it.

To use a different temporary directory, you can set the following JVM property:

Copy 
-Djava.io.tmpdir=/some/other/temporary/directory

When Puppet Server is installed from packages, add this property to the JAVA_ARGS and JAVA_ARGS_CLI variables defined in either /etc/sysconfig/puppetserver or /etc/default/puppetserver, depending on your distribution. Invocations of the gem, ruby, and irb subcommands use the updated JAVA_ARGS_CLI on their next invocation. The service will need to be restarted in order to re-read the JAVA_ARGS variable.

Puppet Server Fails to Connect to Load-Balanced Servers with Different SSL Certificates

SERVER-207: Intermittent SSL connection failures have been seen when Puppet Server tries to make SSL requests to servers via the same virtual ip address. This has been seen when the servers present different certificates during the SSL handshake.