Puppet orchestrator technical overview

The orchestrator uses pe-orchestration-services, a JVM-based service in Puppet Enterprise (PE), to execute on-demand Puppet runs on agent nodes in your infrastructure. The orchestrator uses Puppet Execution Protocol (PXP) agents to orchestrate changes across your infrastructure.

The orchestrator (part of pe-orchestration-services) controls the functionality for the puppet job, puppet task, and puppet plan commands, and it also controls the functionality for jobs and single-node runs in the PE console.

The orchestrator is comprised of several components, each with their own configuration and log locations.

Puppet orchestrator architecture

The orchestrator's functionality derives from the Puppet Execution Protocol (PXP), the Puppet Communications Protocol (PCP), and the Agentless Catalog Executor (ACE) Server.

Puppet Execution Protocol (PXP)

A message format used to request that a task be executed on a remote host and receive responses on the status of that task.

Used by pe-orchestration-services to run Puppet on agents.

PXP agent

A system service in the agent package that runs PXP.

Puppet Communications Protocol (PCP)

The underlying communication protocol that describes how PXP messages get routed to an agent and back to the orchestrator.

PCP broker

A JVM-based service that runs in pe-orchestration-services on the primary server and in the pe-puppetserver service on compilers.

PCP brokers route PCP messages, which declare the content of the message (via message type) and identify the sender and intended recipient.

PCP brokers on compilers connect to the orchestrator, and the orchestrator uses the brokers to direct messages to PXP agents connected to the compilers. When using compilers, PXP agents running on PE components (which includes the primary server, PuppetDB, and the PE console) connect directly to the orchestrator, but all other PXP agents connect to compilers via load balancers.

Agentless Catalog Executor (ACE) service

A Ruby service that enables you to execute tasks, plans, and Puppet runs on remotely on agentless targets. Refer to PE ACE server configuration for more information.

Bolt vs ACE: Orchestrator uses both ACE and Bolt to run tasks and plans. While both can act on agentless targets, the primary difference is that Bolt server works with agentless nodes over WinRM or SSH, whereas ACE works with agentless devices, like network switches and firewalls, over other transports. Go to PE Bolt server configuration to learn about how Bolt works in PE and configuring the Bolt server.

What happens during an on-demand run from the orchestrator ?

Several PE services interact when you Run Puppet on demand from the orchestrator.

  1. You use the puppet job command to create a job in orchestrator.
  2. The orchestrator validates your token with the PE RBAC service.
  3. The orchestrator requests environment classification from the node classifier for the nodes targeted in the job. It also queries PuppetDB for the nodes.
  4. The orchestrator creates the job ID and starts polling nodes in the job to check their statuses.
  5. The orchestrator queries PuppetDB for the agent version on the nodes in the job.
  6. The orchestrator tells the PCP broker to start runs on the nodes in the job, and Puppet runs start on those agents.
  7. Agents send run results to the PCP broker.
  8. The orchestrator receives run results and requests node run reports (also called agent run reports) from PuppetDB.

What happens during a task run from the orchestrator?

Several services interact during task runs. Because tasks are Puppet code, they must be deployed into an environment on the primary server. Puppet Server then exposes the task metadata to the orchestrator. When a task runs, the orchestrator sends the PXP agent a URL indicating where to fetch the task from (on the primary server) and the task file's checksum. The PXP agent downloads the task file from the supplied URL and caches it for future use. The file is validated against the checksum before every execution. This process is comprised of the following steps:

  1. The PE client sends a task command.
  2. The orchestrator checks if the user is authorized.
  3. The orchestrator fetches the node target from PuppetDB (if the target is a query) and returns the list of targeted nodes.
  4. The orchestrator requests task data from Puppet Server.
  5. Puppet Server returns task metadata, file URIs, and file SHAs.
  6. The orchestrator validates the task command and then sends the job ID back to the client.
  7. The orchestrator sends task parameters and file information to the PXP agent.
  8. The PXP agent sends a provisional response to the orchestrator, checks the SHA against the local cache, and requests the task file from Puppet Server.
  9. Puppet Server returns the task file to the PXP agent.
  10. The task runs.
  11. The PXP agent sends the result to the orchestrator.
  12. The client requests events from the orchestrator.
  13. The orchestrator returns the result to the client.

Notes about configuring the orchestrator and related components

Various files contain configuration and tuning settings for orchestrator components.

pe-orchestration-services

This is the underlying service for the orchestrator.

The main configuration file is located at: /etc/puppetlabs/orchestration-services/conf.d

Additional configuration for large infrastructures can include tuning the pe-orchestration-services JVM heap size, increasing the limit on open file descriptors for pe-orchestration-services, and tuning ARP tables.

PCP broker

Part of the pe-puppetserver service.

The main configuration file is located at: /etc/puppetlabs/puppetserver/conf.d

The PCP broker requires JVM memory and file descriptors. These resources scale linearly with the number of active connections. Specifically, the PCP broker requires:

  • Approximately 40 KB of memory (when restricted with the -Xmx JVM option)
  • One file descriptor per connection
  • An approximate baseline of 60 MB of memory and 200 file descriptors

For example, for a deployment with 100 agents, expect to configure the JVM with at least -Xmx64m and 300 file descriptors. Message handling requires minimal additional memory.

PXP agent

Configuration is managed in the agent profile class (puppet_enterprise::profile::agent).

The PXP agent is configured to use Puppet’s SSL certificates and point to one PCP broker endpoint. If you've configured disaster recovery, the agent points to additional PCP broker endpoints in the case of failover.

If you reuse an existing agent with a new orchestrator instance, you must delete the pxp-agent spool directory, which is located at:
  • /opt/puppetlabs/pxp-agent/spool on *nix systems
  • C:\ProgramData\PuppetLabs\pxp-agent\var\spool on Windows systems

Related information

Debugging the orchestrator and related components

If you need to debug the orchestrator or any of its related components, the following log locations might be helpful.

pe-orchestration-services

The main log file is located at:

/var/log/puppetlabs/orchestration-services/orchestration-services.log

PCP brokers

The main log file for PCP brokers on the primary server is located at:

/var/log/puppetlabs/orchestration-services/pcp-broker.log

The main log file for PCP brokers on compilers is located at:

/var/log/puppetlabs/puppetserver/pcp-broker.log

You can configure logback in your Puppet Server configuration.

You can also enable an access log for messages.

PXP agent

The main log file location varies by OS, and it can be configured if necessary. On *nix systems, the default location is:

/var/log/puppetlabs/pxp-agent/pxp-agent.log

On Windows systems, the default location is:

C:/ProgramData/PuppetLabs/pxp-agent/var/log/pxp-agent.log

Metadata about Puppet runs triggered through the PXP agent are kept in the spool-dir for 14 days (by default). The spool-dir location varies by OS.

  • For *nix: /opt/puppetlabs/pxp-agent/spool
  • For Windows: C:/ProgramData/PuppetLabs/pxp-agent/var/spool

ACE server

The main log file is located at:

/var/log/puppetlabs/ace-server/ace-server.log

Related information