Skip to content

Operations runbooks

These procedures complement the versioned behavior. They are intended for maintenance jobs in GitLab with an authorized agent context. They were not executed against a cloud account while this documentation was prepared.

1. Verify a delivery

Record the pipeline, application SHA, image and manifest repository commit. Check the target before inspecting resources; AWS and GCP can use identical namespaces and resource names.

kubectl --context "$KUBE_CONTEXT" -n app get deployments,pods,services,ingresses
kubectl --context "$KUBE_CONTEXT" -n app rollout status deployment/rest --timeout=180s
kubectl --context "$KUBE_CONTEXT" -n app rollout status deployment/web --timeout=180s
kubectl --context "$KUBE_CONTEXT" -n app get deployment rest web   -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}'

When delivering only one service, verify that Deployment and then its interaction with the other service's existing version. The timeout is an operational proposal, not a guarantee that every startup finishes in three minutes.

Once HTTP exposure is configured and APP_BASE_URL identifies that target, check / and a read request to /api/v1/users. Alongside the HTTP status, verify that the interface loads and the browser queries the correct backend. The snapshot does not automate these checks.

2. Recover an application version

  1. Temporarily pause new deliveries for the service and select the last known good image. Confirm it is still in the registry.
  2. Review whether schema/data changes allow returning to the previous version. ddl-auto=update does not provide reverse migrations.
  3. Create a reviewed revert or change in the manifest repository fixing the selected image. Keep the resulting commit as MANIFEST_COMMIT.
  4. Apply that exact commit through an authorized maintenance job. A commit in the manifest repository does not trigger its own pipeline.
  5. Wait for rollout, test the read route and record the cause and restored version.

Example job core, after supplying CD_PROJECT_PATH, KUBE_CONTEXT and a reviewed SHA:

test -n "$MANIFEST_COMMIT"
git clone --quiet "$CD_PROJECT_PATH" rollback-manifests
git -C rollback-manifests checkout --detach "$MANIFEST_COMMIT"
kubectl --context "$KUBE_CONTEXT" -n app apply -f rollback-manifests/

Use a fresh working directory per job and protected secret variables. Complete the example with the checks from the first runbook. Rerunning the complete application pipeline may write that application's commit image again, undoing the selected recovery; do not confuse it with applying the manifest commit.

An emergency rollout undo changes the cluster without changing Git. If used, reconcile the manifest repository afterwards and verify the effective image so that the next pipeline does not overwrite the recovery.

3. Check registry access

kubectl --context "$KUBE_CONTEXT" -n app get secret registry   -o jsonpath='{.type}{"\n"}'
kubectl --context "$KUBE_CONTEXT" -n app describe deployment rest
kubectl --context "$KUBE_CONTEXT" -n app get events --sort-by=.lastTimestamp

The registry Secret is an external prerequisite in this snapshot. If absent, create it through the environment's credential procedure using a registry read token; do not use an ephemeral job password as a permanent cluster secret. Also check that the image path and tag exist. Do not publish .dockerconfigjson contents in logs or issues.

Proposed registry Secret bootstrap

In an authorized manual maintenance job, store a valid dockerconfigjson in a File-type CI variable named REGISTRY_AUTH_FILE. Its credential must read images from both projects, since rest and web use the same Secret. This procedure supplements the snapshot and is not part of the original pipeline.

test -s "$REGISTRY_AUTH_FILE"
kubectl --context "$KUBE_CONTEXT" -n app create secret generic registry \
  --type=kubernetes.io/dockerconfigjson \
  --from-file=.dockerconfigjson="$REGISTRY_AUTH_FILE" \
  --dry-run=client -o yaml |
  kubectl --context "$KUBE_CONTEXT" -n app apply -f -

Run this after the namespace exists, with the tools and context available. Keep the file out of Git, artifacts and debug traces. The first command's output feeds directly into the second; do not add a Secret dump to the job log. When rotating the credential, confirm new pods can pull both images.

4. Rotate database credentials

Identify the state managing the database account and the layer generating the rest Secret. Prepare provider-side credential rotation and apply a consistent change in database and platform, respecting their dependencies. Restart consumers that read the Secret as environment variables in a controlled manner, then verify a read query.

With one application account, a password change may interrupt connections during the transition. The repository implements neither dual-credential rotation nor automatic reload; agree on the procedure and change window. Restore an old value only if it is also still valid in the database.

5. Recover an interrupted infrastructure pipeline

First inspect the last operation, state name, still-running jobs and existing resources. Compare the saved plan with current state and generate a new plan if the old one expired or became stale. Resume at the first pending layer only after checking outputs of earlier layers.

force_unlock_gcp_database_private_service_access removes a state binding; it does not release a lock. It must not be run merely to clear a visually blocked pipeline. State and teardown explain its effects and limitations.

6. Change the group namespace

Inventory references before the change: includes, manifest Git URLs, registries, agent contexts and ci_access, state backends and documentation links. Keep numeric project IDs to distinguish a path change from copying to new projects.

Afterwards, validate includes and permissions, build a test image, inspect a context and update a manifest through a controlled delivery. Refresh source-manifest.json when the new location is reviewed. This site uses the Pages URL supplied by CI so the group's historical path is not hardcoded as the publication URL.

Sources and traceability