Disaster recovery

It is important to prepare your system and regularly capture full snapshots. This backs up your data and makes it easier to restore your system if disaster recovery is needed.

Prepare your system to support future disaster recovery

To make sure your system is equipped to help you recover from a potential system failure, you must:

  1. Track the Continuous Delivery Bolt project in version control and push the project to this repo whenever changes are made and applied to the Continuous Delivery target host.

    2. Any private keys in the ./keys directory should be stored separately. Do not include them in version control.

  2. Create a backup of the Continuous Delivery application using bolt plan run cd4peadm::backup on a regular basis and copy the resulting backup artifact to a secure location, such as an NFS share, S3 bucket, distributed file system, cloud storage, etc.

Disaster recovery process

Depending on the nature of the disaster you may need to follow different processes. This section covers a few disaster recovery scenarios.

Loss of the Continuous Delivery Bolt project

  1. Restore the Bolt project from version control.

  2. Restore the private key from its separate secure storage.

Loss of the Continuous Delivery host

If you lose the Continuous Delivery host, you can restore the application from a backup file.

If you are restoring Continuous Delivery onto a new host, update the following items in your Bolt project before running the install plan:

  • Update the resolvable_hostname in data/common.yaml to the new hostname.

  • Update inventory.yaml to the new target host information.

  1. Create a new Continuous Delivery installation. From the Continuous Delivery Bolt project directory, run:

     bolt plan run cd4peadm::install

  2. Copy the tarball of the most recent backup into /var/lib/puppetlabs/cd4pe/backups on the new target host.

  3. Restore the backup using:

     bolt plan run cd4peadm::restore <backup name>

  4. Align the database passwords using:

     bolt plan run cd4peadm::apply_configuration

    This ensures that the database passwords are updated to match the configuration values in the data/common.yaml file.

Loss of both the Continuous Delivery Bolt project and the Continuous Delivery host

Follow the process described in both scenarios, first the Continuous Delivery Bolt recovery then the Continuous Delivery recovery.