Prevent deployment timing issues when updating custom facts

When updating a custom fact and the Puppet code that consumes it, deploy manifest changes that support both the current and new fact values before deploying the updated custom fact. After all nodes have received and are reporting the new fact value, you can remove support for the previous value if it is no longer required.

During an agent run, Puppet performs operations in stages. Pluginsync, which distributes custom facts and other plugins to agents, occurs before catalog compilation. Pluginsync and catalog compilation resolve deployed code independently, so if a code deployment occurs between these stages, an agent can evaluate facts against one version of the code while compiling its catalog against another version.

As a result, an agent can temporarily resolve a previous version of a custom fact if its run begins before the updated fact has been synchronized, even when newer Puppet code has already been deployed. To avoid catalog compilation issues during a deployment, design manifests to remain compatible with both old and new fact values during the transition period.

This timing window can be more noticeable on nodes that require additional time to resolve facts.

For example, assume a custom fact returns the value web:

Facter.add(:app_tier) do
setcode do
'web'
end
end

The manifest uses that fact to manage a web server:

if $facts['app_tier'] == 'web' {
package { 'nginx':
ensure => installed,
}
service { 'nginx':
ensure => running,
enable => true,
require => Package['nginx'],
}
}

Later, the fact is modified to return webserver instead:

Facter.add(:app_tier) do
setcode do
'webserver'
end
end

Before deploying the updated fact, update the manifest to support both values:

if $facts['app_tier'] in ['web', 'webserver'] {
package { 'nginx':
ensure => installed,
}
service { 'nginx':
ensure => running,
enable => true,
require => Package['nginx'],
}
}

Deploy the updated manifest first. After the manifest changes have been deployed, deploy the updated custom fact. This approach ensures that nodes using either version of the custom fact continue to receive the expected configuration during and immediately after deployment. It also helps ensure that agent runs spanning different code versions can continue to compile successfully, regardless of whether they resolve the old or new fact value.

After confirming that all nodes have received and are reporting the new fact value, you can remove support for the legacy value if it is no longer required.

Recommended deployment order

  1. Deploy manifest changes that support both the current and new fact values.

  2. Deploy the updated custom fact.

  3. Verify that nodes are reporting the new fact value.

  4. Remove support for the legacy value when it is no longer needed.

Best practices

  • Treat changes to custom fact names or values as compatibility-sensitive changes.

  • Deploy manifest changes that support both old and new fact values before deploying updated custom facts.

  • Allow sufficient time for all nodes to receive updated plugins before removing compatibility code.

  • Use safe defaults and defensive logic when consuming custom facts to ensure catalogs compile successfully if a fact is missing, unexpected, or temporarily differs from the expected value.