Provision and enable a replica
Provisioning a replica duplicates specific components and services from the primary server to the replica. Enabling a replica activates most of its duplicated services and components, and instructs agents and infrastructure nodes how to communicate in a failover scenario.
- Ensure you have completed the steps outlined in the Configure disaster recovery section.
- Configure infrastructure agents to connect orchestration agents to the primary
server.
- In the console, click Node groups, and in the PE Infrastructure group, select the PE Agent > PE Infrastructure Agent group.
- If you manage your load balancers with agents, on the
Rules tab, pin load balancers to the
group.
Pinning load balancers to the PE Infrastructure Agent group ensures that they communicate directly with the primary server.
- On the Classes tab, find the
puppet_enterprise::profile::agent class and
specify these parameters:
Parameter Value manage_puppet_conf Specify true
to ensure that your setting forserver_list
is configured in the expected location and persists through Puppet runs. This is the default value.pcp_broker_list Hostname for your primary server. Hostnames must include port 8142, for example ["PRIMARY.EXAMPLE.COM:8142"]
.primary_uris Hostname for your primary server, for example ["PRIMARY.EXAMPLE.COM"]
. This setting assumes port 8140 unless you specify otherwise withhost:port
.server_list Hostname for your primary server, for example ["PRIMARY.EXAMPLE.COM"]
. This setting assumes port 8140 unless you specify otherwise withhost:port
. - Remove any values set for pcp_broker_ws_uris.
- Commit changes.
- Run Puppet on all agents classified into the PE Infrastructure Agent group.
- On the primary server, as the root user, run
puppet infrastructure provision replica <REPLICA NODE NAME> --enable
The default
replica --enable
command adds the replica to your PE Agent node group's server list, which causes allPuppet.conf
files to include the new server. However, if you include the--skip-agent-config
flag, the replica is added to the server list of the PE Infrastructure Agent node group (which is a child of the PE Agent node group); this, by extension, impacts only thePuppet.conf
files on your infrastructure nodes (including compilers).In installations with compilers, use the
--skip-agent-config
flag with--enable
if you want to:- Upgrade a replica without needing to run Puppet on all agents.
- Add disaster recovery to an installation without modifying the configuration of existing load balancers.
- Manually configure which load balancer agents communicate with in multi-region installations. See Managing agent communication in multi-region installations.
- Copy your secret key files from the primary server to the replica.
The secret key files are located at:
/etc/puppetlabs/orchestration-services/conf.d/secrets/keys.json
/etc/puppetlabs/orchestration-services/conf.d/secrets/orchestrator-encryption-keys.json
/etc/puppetlabs/console-services/conf.d/secrets/keys.json
If you do not copy your secret key files onto your replica, the replica generates new secret key files when you promote it. This prevents you from accessing LDAP, and prevents services from accessing encrypted information in PE databases. - Verify that the contents of the global layer Hiera file on the new replica,
located at
/etc/puppetlabs/puppet/hiera.yaml
, match the contents of the global layer Hiera file on the primary server.- If necessary, update
hiera.yaml
on the replica to matchhiera.yaml
on the primary server. - If you use code to manage the contents of
hiera.yaml
on the primary server, ensure that the new replica is also classified to manage the contents of its ownhiera.yaml
file.
- If necessary, update
- Optional: Verify that all services running on the primary server are also running on the
replica:
- From the primary server, run
puppet infrastructure status --verbose
to verify that the replica is available. - From any managed node, run
puppet agent -t --noop --server_list=<REPLICA HOSTNAME>
. If the replica is correctly configured, the Puppet run succeeds and shows no changed resources.
- From the primary server, run
- Optional: Deploy updated configuration to agents by running Puppet, or wait for the next scheduled Puppet run.
If you used the
--skip-agent-config
option, you can skip this step.If you use the direct Puppet workflow, where agents use cached catalogs, you must manually deploy the new configuration by running:puppet job run --no-enforce-environment --query 'nodes {deactivated is null and expired is null}'
- Optional: Perform any tests you feel are necessary to verify that Puppet runs continue to work during failover. For
example, to simulate an outage on the primary server:
- Prevent the replica and a test node from contacting the primary server.
For example, you might temporarily shut down the primary server or use
iptables
with drop mode. - Run
puppet agent -t
on the test node. If the replica is correctly configured, the Puppet run succeeds and shows no changed resources. Runs might take longer than normal when in failover mode. - Reconnect the replica and test node.
- Prevent the replica and a test node from contacting the primary server.
For example, you might temporarily shut down the primary server or use