Migrate 4.x data to 5.x

The Continuous Delivery 5.x series uses Bolt for installation and management rather than Puppet Application Manager, so you must migrate your 4.x series data to the new installation.

What is migrated?

The migration task copies and migrates to 5.x:

  • The database, which contains all your Continuous Delivery 4.x installation's information, users, integrations, history, and artifacts (except job commands and job logs).
  • The object store, which contains all of your 4.x installation's job commands and job logs.
  • Any of these configuration settings, if in use on 4.x:
    CD4PE_REPO_CACHE_RETRIEVAL_TIMEOUT_MINUTES
    CD4PE_LDAP_GROUP_SEARCH_SIZE_LIMIT
    CD4PE_ENABLE_REPO_CACHING
    CD4PE_HTTP_CONNECTION_TIMEOUT_SEC
    CD4PE_HTTP_READ_TIMEOUT_SEC
    CD4PE_HTTP_WRITE_TIMEOUT_SEC
    CD4PE_HTTP_REQUEST_TIMEOUT_SEC
    CD4PE_INCLUDE_GIT_HISTORY_FOR_CD4PE_JOBS
    CD4PE_JOB_GLOBAL_TIMEOUT_MINUTES
    CD4PE_JOB_HTTP_READ_TIMEOUT_MINUTES

The migration task does not copy or move to 5.x

  • The 4.x root user password, you are prompted to create a new root user password when installing 5.x.
  • The JVM_ARGS configuration setting, if in use on 4.x.
  • These backup tables created for you by Continuous Delivery during the 4.x series:
    • app-pipelines-3-0-backup.pfi, which was created during the upgrade from 3.x to 4.x
    • pe-ia-node-results-cve-2020-7944-backup.pfi and pe-ia-resource-results-cve-2020-7944-backup.pfi, which were created as part of the CVE 2020-7944 remediation in version 3.4.0

Migration cautions

You must have upgraded Continuous Delivery to version 4.26.0 or later before you can migrate to version 5.x.

A super user or the root user must perform this process.

Migrations from installations with external databases are not supported.

Once you initiate the migration task, any new data created in the 4.x installation may not be migrated to the 5.x installation. To avoid losing data, restrict access to Continuous Delivery during migration.

Run the migration task

The migration task uses the cd4peadm::install_from_v4 plan to migrate data from existing 4.x PAM-based installation to a new 5.x install.

If you have already installed Continuous Delivery and would like to add a license file to it, please see Add or update a license key.
Before you begin:

You must have upgraded Continuous Delivery to version 4.26.0 or later before you can migrate to version 5.x.

Unlike version 4.x, Continuous Delivery 5.x uses Bolt for installation, configuration, and administration instead of PAM. To install Continuous Delivery, you need a Forge API token to download the cd4peadm module.

Before migrating, please make sure the Linux system you plan to install Bolt on has internet access as well as SSH access to both the system on which you intend to install Continuous Delivery 5.x as well as the existing 4.x installation from which you are migrating. In addition, because Bolt is installed on one or more systems and used to manage one centralized Continuous Delivery installation, it may be a good idea to maintain the Bolt project in its own VCS repo.

If noexec is set on the /tmp filesystem on the system on which you intend to install Continuous Delivery 5.x, you need to use Bolt’s tmpdir setting in inventory.yaml to point at a directory on a filesystem that does not have noexec set.

Once you've completed the above preparations, you're ready to run the migration task:

  1. Install Bolt version 3.27.4 or later on a Linux jumphost. This can be the intended Continuous Delivery 5.x host, or any other system.
  2. Create the Continuous Delivery Bolt project and switch to that directory.
    mkdir cd4pe-bolt-project 
    cd cd4pe-bolt-project
    bolt project init cd4pe_bolt_project
  3. Edit the bolt-project.yaml file to specify the module to install and your Forge API token. Change the modules and module-install sections to:
    # bolt-project.yaml
    modules:
      - name: puppetlabs/cd4peadm
        version_requirement: 5.y.z
    
    module-install:
      forge:
         authorization_token: 'Bearer <your API token>'
         baseurl: https://forgeapi.puppet.com
    
  4. Optional: Enable log append mode for Bolt logging to simplify debugging efforts in the case of problems. Add the following to bolt-project.yaml:
    # bolt-project.yaml
    log:
      bolt-debug.log:
        append: true
        level: debug			
    Enabling logging causes bolt-debug.log to grow over time. You can safely delete this file to free space as needed.
  5. Install the cd4peadm module using the command: bolt module install.
  6. Create an inventory.yaml file with two targets: One for the host on which Continuous Delivery 5.x is to be installed and another with kubectl access to your Continuous Delivery 4.x install. If your 4.x instance is not installed in the default namespace, add a var to the target called kubernetes_namespace and set it to the namespace you are using. For example:
    ---
    targets:
    - name: cd4pe-5-node
      uri: <IP address or DNS name>
      config:
        transport: ssh
        ssh:
          host-key-check: false
          native-ssh: true
          ssh-command: <Path to SSH command>
          user: root
    - name: cd4pe-4-node
      uri: <IP address or DNS name>
      config:
        transport: ssh
        ssh:
          host-key-check: false
          native-ssh: true
          ssh-command: <Path to SSH command>
          user: root
        vars:
          kubernetes_namespace: 'cd4pe'

    If you are installing on a localhost, use the following connection information instead:

    - name: local-install-host
      uri: localhost
      config:
        transport: local
        local:
          bundled-ruby: false
    
    If you need to change your resolvable hostname for some reason, see Update your resolvable hostname.
  7. Optional: If you have a license file, create a directory named files and copy the license file to files/license.lic. A license file is required to access the premium features of Continuous Delivery.
  8. Run the migration plan using: bolt plan run cd4peadm::install_from_v4. The plan extracts your 4.x settings via kubectl, installs Continuous Delivery 5.x with those settings, and migrates data from the 4.x database to the new 5.x database.
    Migrating the database data creates a database dump in /tmp on the host that is running kubectl. Please ensure that there is sufficient space for that data on the host. You can check the size of your database using the command:
    kubectl exec -it postgres-0 -- du -skh /var/lib/postgresql/data/pgdata
Results
You can now log into the application at the resolvable hostname you provided during the installation, using the same root username as your 4.x install and the new root password you supplied.

Refresh ADO OAuth integration

Migrating Continuous Delivery from 4.x to 5.x typically requires installing the app on a new node. If this results in the URL of the application changing, OAuth integrations with ADO Cloud need to be recreated, both within Continuous Delivery and in ADO. This does not affect any control repos or modules you have set up; it refreshes the credentials and supplies the new application URL to ADO.

To recreate the integration, take the following steps:

  1. In Continuous Delivery’s root console, go to Settings → Integrations and delete the ADO integration.

  2. In any workspace that has been integrated with ADO, go to Settings → Source Control and delete any ADO integrations.

  3. Follow the instructions to integrate Azure Devops to create a new application and register it with Continuous Delivery, using the new authorization callback URL from the root console of your migrated instance.

  4. Re-add the workspace integrations.

  5. If you are not using your old ADO OAuth app (for instance, if you have shut down your 4.x instance that it was previously integrated with), it is safe to delete it now from the ADO console.

Test the new 5.x installation

You must thoroughly test the 5.x installation to make sure the data was properly migrated and all functionality is working as expected.

  1. Test the data migration to make sure your 4.x installation's data is present in your 5.x installation. For each workspace in your 5.x installation:
    1. Review event logs for all control repos and modules.
    2. Check that all jobs and hardware connections are present.
    3. Check that all users and user groups are present.
  2. If any data is missing or looks incorrect, rerun the migration task.
  3. Test manual pipeline runs to make sure your 5.x installation is functioning properly. For each workspace in your 5.x installation, trigger a manual pipeline run for all control repos and modules. To do this, select Trigger pipeline from the Manual actions menu above each pipeline.
What to do next
Once you've confirmed your 4.x data is present in your 5.x installation and all manual pipeline runs succeed, Update webhooks.

Update webhooks

As the final step in the 4.x to 5.x data migration process, update the webhooks that connect Continuous Delivery to your source control system.

Follow these steps if you're updating webhooks after migrating data from 4.x to 5.x. There are separate instructions to Update webhooks in other scenarios.
  1. In the Continuous Delivery 4.x root console, navigate to Config > Optional Configuration, select View options for using a proxy or external load balancer, and copy the backend service endpoint. Use the default value if a custom value has not been defined.
  2. In the Continuous Delivery 5.x root console, click Settings > Webhooks, and enter the backend service endpoint you copied from the 4.x root console.
  3. Click Update webhooks.
Results
Continuous Delivery updates all webhooks to point to the current installation.
After migrating data from 4.x to 5.x, once you update webhooks to point to your 5.x installation, do not create new data in the 4.x installation, and do not run the migration task again.

Post-migration guidance and troubleshooting

After migrating Continuous Delivery data from 4.x to 5.x, thouroughly testing your 5.x installation, and updating webhooks, you're ready to begin using 5.x with your team. However, do not immediately decommission your 4.x installation. You may need to roll back to 4.x if issues arise.

Keep your 4.x installation intact until you are fully confident that all features of the new 5.x installation are working as expected. As a precaution, we strongly advise maintaining your inactive 4.x installation for the first two months of using 5.x.

Maintaining your dormant 4.x installation for an extended test period allows you to roll back to 4.x and prevent production environment disruptions if an issue arises with your 5.x migration.

Rolling back to 4.x

If necessary, you can temporarily roll back to your 4.x installation by switching the active webhooks from the 5.x installation to the 4.x installation and generating new access tokens for all PE instances connected with the 4.x installation.

  1. In your Continuous Delivery 4.x installation's web UI, navigate to a control repo's pipeline and click Manage pipelines.
  2. Select Manage in the web UI and locate the Automation webhook section where the webhook (including token) for your control repo is shown.
  3. Copy the webhook and navigate to the corresponding repository in your source control system. Add the 4.x webhook to the repository and remove the 5.x webhook.
  4. Repeat this process for each control repo and module repo in your 4.x installation.
  5. Generate a new access token for each PE instance connected to the 4.x installation. In the Continuous Delivery web UI, navigate to Settings > Puppet Enterprise and click More actions > Edit integration.
  6. Select Basic authorization or API token and enter the required information:
    • For Basic authorization, enter the username and password for your "Continuous Delivery" user. Continuous Delivery uses this information to generate an API token for you. The username and password are not saved.
    • For API token, generate a PE access token for your "Continuous Delivery" user using puppet-access or the RBAC API v1 service, and paste it into the API token field. For information about generating PE access tokens and the RBAC API v1 service, see Token-based authentication in the PE documentation.
      To avoid unintended service interruptions, create access tokens with a multi-year lifespans.

Once the 4.x webhooks and new PE access tokens are in place, you can resume using the 4.x installation.

When you're ready to return to 5.x, repeat the migration process, starting with Run the migration task.