Disaster recovery

It is important to prepare your system and regularly capture full snapshots. This backs up your data and makes it easier to restore your system if disaster recovery is needed.

Before you begin:
Ensure you are running a version of the module that matches the version of the backup being used for the migration and that the target has been updated in Hiera to reference the new target system. Update the leaf certificate in ssl_cert_chain in Hiera to point to the new resolvable_hostname.

Prepare your system to support future disaster recovery

To make sure your system is equipped to help you recover from a potential system failure, you must:

  1. Track the Continuous Delivery Bolt project in version control and push the project to this repo whenever changes are made and applied to the Continuous Delivery target host.

    2. Any private keys in the ./keys directory should be stored separately. Do not include them in version control.

  2. Create a backup of the Continuous Delivery application using bolt plan run cd4peadm::backup on a regular basis and copy the resulting backup artifact to a secure location, such as an NFS share, S3 bucket, distributed file system, cloud storage, etc.

Disaster recovery process

Depending on the nature of the disaster you may need to follow different processes. This section covers a few disaster recovery scenarios.

Loss of the Continuous Delivery Bolt project

  1. Restore the Bolt project from version control.

  2. Restore the private key from its separate secure storage.

Loss of the Continuous Delivery host

If you lose the Continuous Delivery host, you can restore the application from a backup file.

If you are restoring Continuous Delivery onto a new host, update the following items in your Bolt project before running the install plan:

  • Update the resolvable_hostname in data/common.yaml to the new hostname.

  • Update inventory.yaml to the new target host information.

  • Update the leaf certificate in ssl_cert_chain in Hiera to point to the new resolvable_hostname.

  1. Create a new Continuous Delivery installation. From the Continuous Delivery Bolt project directory, run:

    Copy 
     bolt plan run cd4peadm::install

  2. Copy the tarball of the most recent backup into /var/lib/puppetlabs/cd4pe/backups on the new target host.

  3. Restore the backup using:

    Copy 
     bolt plan run cd4peadm::restore backup=<backup name>.zip

    Include the .zip file extension in the backup name.

  4. After the restore, update any webhooks that reference the previous hostname. For more information, see Troubleshooting Continuous Delivery (CD).

  5. Align the database passwords using:

    Copy 
     bolt plan run cd4peadm::apply_configuration 

    This ensures that the database passwords are updated to match the configuration values in the data/common.yaml file.

Loss of both the Continuous Delivery Bolt project and the Continuous Delivery host

Follow the process described in both scenarios, first the Continuous Delivery Bolt recovery then the Continuous Delivery recovery.