Skip to content

Teardown, retention and recovery

The pipeline implements destruction through saved plans and manual approval, in reverse provisioning order. Retiring a lab requires accounting for state-managed resources as well as objects created by Kubernetes controllers or external procedures.

Before planning teardown

  1. Identify the account/project, provider, ENVIRONMENT, all four state names and resource owner. Check them against actual resources, not just a pipeline name.
  2. Stop concurrent application deployments and new infrastructure runs for that environment.
  3. Decide which data must survive. Create and verify required SQL backups/exports before deleting the database.
  4. Retain protected copies of state and effective configuration, including resolved provider and chart versions. Do not store them in this public site.
  5. Remove controller-created ingress resources while their controllers are still running. On AWS, review the application Ingress and wait for ALB, targets and associated resources to be removed before retiring the controller.
  6. Check that unrelated resources do not share namespaces being deleted. Deleting app also removes objects inside it that Terraform does not directly manage.

SQL passwords, tokens and state need access controls, but they are not backups of the data. A state backup describes resources; it is not an operational restoration of application tables.

Normal job behavior

Run the infrastructure pipeline on main with PIPELINE_ACTION=destroy, an explicit CLOUD_PROVIDER and the same ENVIRONMENT identifying the correct states. Review each plan before triggering its manual apply.

Step Plan and apply Dependency that must remain available
1 plan_destroy_platform → apply_destroy_platform Cluster API and access credentials
2 plan_destroy_database → apply_destroy_database Network and completed data-retention decisions
3 plan_destroy_kubernetes → apply_destroy_kubernetes Network and cloud permissions
4 plan_destroy_network → apply_destroy_network Dependent resources already removed

Do not change state name or provider between plan and apply. Do not reuse a plan after state changes, manual repairs or changes from another pipeline. If the one-week artifacts expire, generate and review a new plan.

destroy is not an application-version rollback. To return to a previous image, use the deployment process and manifests; destroying infrastructure removes databases and compute capacity.

Configured retention and its boundaries

Service Teardown-related defaults Operational consequence
RDS deletion_protection=false, skip_final_snapshot=true, 1-day automated retention Code permits instance deletion without a final snapshot; preserve required data before apply
Cloud SQL deletion_protection=false, backups and binary logs enabled No export or independent final-retention procedure is declared
GKE deletion_protection=false This additional deletion barrier is absent from the code
Namespaces Terraform manages app and gitlab-agent Deleting them affects all contained objects
AWS certificate/DNS External to the inspected roots Review separately; do not assume destroy manages them

For RDS, setting only skip_final_snapshot=false does not complete a final-snapshot strategy here: the resource defines no final_snapshot_identifier. If adopting that policy, add and validate the required configuration before starting teardown. Do not present an isolated variable change as an already tested mechanism.

Exceptional case: GCP Private Service Access

force_unlock_gcp_database_private_service_access removes google_service_networking_connection.private_service_access from state. Its name is misleading: it does not run tofu force-unlock. state rm behavior is to forget the binding between the resource address and remote object; the remote object may remain.

This can leave untracked peering and cause a later plan to try to recreate it. The job also differs from normal mutual exclusion: it fixes resource_group: dev-gcp-database while backend selection uses the chosen ENVIRONMENT. Its shell expression additionally ends with a “not present” message in some failure cases, so job status is not a substitute for inspecting state.

It is not a routine cleanup step. If Private Service Access removal fails:

  1. Capture the actual error and check whether instances or services still use the connection.
  2. Confirm Cloud SQL deletion has really completed and consult provider diagnostics.
  3. Preserve a protected state backup and stop other plans/applies.
  4. Decide whether to wait, remove an actual dependency, repair/import the binding or perform an exceptional state operation. Record the reason and affected remote object.
  5. After any state change, discard previous plans and generate a new one. Inventory the Google Cloud object even if it no longer appears in state.

The repository does not document the original error that motivated this workaround. This documentation retains the observable fact and leaves its rationale pending confirmation with the author.

Historical evidence and post-teardown checks

Pipeline 2604650152, dated 16 June 2026, records successful apply_destroy_platform, apply_destroy_database, apply_destroy_kubernetes, apply_destroy_network and Private Service Access jobs. It also contains two plan_destroy_database executions. It ran on an earlier commit than the analyzed HEAD.

These job statuses establish a historical teardown run; they do not prove current cloud inventory or the absence of orphaned resources. Close a teardown by comparing state with account/project inventory, including indirectly created objects and objects removed from state.

Resources to check for residual cost

AWS GCP
EKS cluster and EC2 nodes GKE cluster and Compute Engine nodes
RDS and retained backups/snapshots Cloud SQL and retained backups/exports
NAT Gateway and associated addresses Cloud NAT, Router and reserved addresses
ALB, targets and controller resources Externally created load balancers or addresses
EBS volumes and out-of-root resources Disks and out-of-root resources

No cost estimate is provided: cost depends on region, use, quotas, retention and actual resources, which were not inspected in the cloud accounts. The inventory aims to close teardown without confusing a green pipeline with absence of all billable resources.

Sources