Troubleshooting¶
Start with the first failing job or resource and preserve its SHA, target and error message. These checks connect symptoms with code contracts; they do not claim that every symptom occurred in the original environment.
CI and artifacts¶
| Symptom | Check first | Action |
|---|---|---|
| No build/deploy jobs appear | Reference protection, literal main, feature flags |
Inspect effective rules; custom jobs exclude unprotected feature branches |
| Include cannot be found | cicd-templates namespace, ref, permissions and GitLab version |
Validate expanded composition with CI Lint |
| Code Quality stops working | Deprecated CodeClimate dependency and runner | Replace the analyzer with compatible export; do not hide failure with an empty report |
| Docker cannot connect | DinD service, privileges, host and runner TLS | Configure the runner for the selected mode |
target/app.jar is absent |
Maven job, final name and artifact download | Repair the build or rebuild the expired artifact |
| Scanner cannot find image | Path, slug, SHA and CS_IMAGE/registry variables |
Align scanning and build; verify registry access |
| Manifest push denied | CD identity and main protection |
Supply valid write permissions; Deploy Token read_repository is insufficient |
Conflict while pushing to main |
Concurrent pipelines | Serialize deliveries and revalidate the revision; avoid automatic force push |
| Missing needed job in IaC | needs and apply exclusions outside main |
Validate the DAG for actual variables/branch; align rules |
Connectivity and Kubernetes¶
| Symptom | Check first | Action |
|---|---|---|
| Agent context missing | Exact name, ci_access, agent connection |
Correct project authorization and allow propagation |
Forbidden with correct context |
Effective identity and RBAC | Run kubectl auth can-i for the verb/resource/namespace |
| Kubernetes provider fails while agent connects | Direct runner-to-API access during platform provisioning | Review provider networking/authentication; the agent does not replace that route |
ImagePullBackOff |
registry Secret, image path/tag and connectivity |
Create/rotate the external credential and verify the image |
Pod Pending |
Available memory, requests, nodes and events | Adjust capacity or scheduling; EKS min/max does not install an autoscaler |
rest CrashLoopBackOff |
Database variables, private networking, permissions, logs | Validate the Secret contract without dumping values |
| Filesystem write error | Required temporary path and read-only root | Add a bounded writable volume if the application needs it |
| Green pipeline, unready service | Missing rollout wait and application checks | Inspect Deployment conditions, pods, endpoints and probes |
| Objects removed from Git remain in cluster | kubectl apply semantics |
Explicitly retire objects owned by the project |
Initial read-only commands, using an already authorized context:
kubectl --context "$KUBE_CONTEXT" -n app get pods -o wide
kubectl --context "$KUBE_CONTEXT" -n app get events --sort-by=.lastTimestamp
kubectl --context "$KUBE_CONTEXT" -n app describe deployment rest
kubectl --context "$KUBE_CONTEXT" auth can-i patch deployments -n app
Exposure and database¶
| Symptom | Check first | Action |
|---|---|---|
| GKE Ingress has no address | alb class in the only public manifest |
Define a GKE variant/controller; AWS annotations are not automatically portable |
| AWS TLS or host error | ACM certificate, region, DNS and ALB controller | Adapt environment references; DNS/certificate are not provisioned here |
| UI loads but API fails | Actual browser request, /api/v1, services |
Confirm origin/routing; internal rest is not a public hostname |
| MySQL connection refused/timeout | Endpoint, port, SG/firewall, peering and nodes | Verify private networking in the selected cloud |
| MySQL authentication error | Database/Secret consistency and pod restart | Complete credential rotation |
| Old version fails after rollback | Schema changed by Hibernate | Use compatible data recovery rather than repeatedly changing images |
State and teardown¶
An active lock is not resolved by removing resources from state. Check the lock owner and still-running jobs. If a plan is stale, regenerate it against the same state. If teardown fails on dependencies, inspect controller-created and external resources, such as the load balancer or private service peering.
The Private Service Access recovery job changes state and may leave existing resources unmanaged. Using it requires understanding the original failure, backing up state and reconciling resources. See teardown.
Documentation site¶
For Pages pipeline failures, see publishing. Typical causes in this repository are missing dependencies, an unpaired translation, broken internal links or an incorrect base URL. Diagnosing it does not require infrastructure credentials.