PR #29920 merged at 19:43 UTC and the service was verified live in production at 00:13 UTC — staging was clean the whole way, while the prod bring-up surfaced a wedged PreSync job, missing database provisioning, and a deploy-approval staleness trap. All diagnosed and resolved; no user impact (net-new service, no traffic).
chore(share): share service and OG cards — merge commit 5c75319d. Images build green on main.
Workflow opens deploy PR #30555 pinning prod tag. Staging needs no pin — it tracks image digests via ArgoCD Image Updater.
phantom/infra #6851 adds the share database, client, and role to all three CockroachDB Pulumi stacks. Its production deploy run parks at the environment approval gate.
#30560 wires the service.cockroachdb block (DATABASE_URL) into the overlays; #30566 gives pods their own service account.
Prod values pin global.deployment.tag: sha-14a8076. ArgoCD app backend-share created.
PreSync job backend-share-migrate fails: it was created by the first sync before any tag existed, ran a pre-merge image, and yarn workspace share did not exist in it. The dead Job then blocks every later sync.
staging-api.phantom.app/share/* answers with the service's own 404 — pods up, DB migrated, routing good.
Fresh syncs now surface the deeper failure: ExternalSecret backend-share-cockroachdb.client.share → "Secret does not exist" (33 failures since 13:48). Prod has no cockroachdb/client/share secret in AWS Secrets Manager.
Gustavo approves the 3-hour-old Pulumi run; all three production deploy jobs refuse: "Refusing stale deployment: current main moved." Staging had deployed earlier while the commit was still head.
workflow_dispatch from current main for cockroachdb/databases, clients, and roles (production) — one project per dispatch, three approvals.
share role + database + grants created, client cert generated, cockroachdb/client/share written to Secrets Manager with cross-region replicas.
ESO materializes the secret, migrate job passes, pods start at 11311d1.
api.phantom.app/share/* returns {"message":"Share not found"} — a real CRDB-backed response through the prod gateway.
The first ArgoCD sync ran before any image tag was pinned, so the PreSync migrate Job was created from a pre-merge image with no share workspace. Kubernetes Jobs are immutable — the dead Job wedged every subsequent sync until manually deleted.
RESOLUTION Delete the stuck Job; later syncs recreate it from the pinned image.
cockroachdb/client/share (and the prod share database/role themselves) did not exist. The infra PR that provisions them was correct — but its production Pulumi deploy sat unapproved for 3 hours.
RESOLUTION Approve + deploy the infra stacks; ESO picks up the secret within a minute.
By the time the approval came, infra main had moved, and the deploy guard refused the stale commit — silently turning an approved run into a no-op failure.
RESOLUTION Re-dispatch fresh runs from current main (one per project), re-approve, deploy.
| Check | Result |
|---|---|
Prod probe — GET api.phantom.app/share/AAAAAAAAAA | service 404 — DB-backed response |
Staging probe — GET staging-api.phantom.app/share/* | service 404 |
ArgoCD rollout backend-share | Healthy @ 11311d1 |
| Presync migration job | passed — 3 tables in prod CRDB |
cockroachdb/client/share secret (ca-central-1 + replicas) | created, ESO synced |
| Error logs since rollout | only startup-probe noise during boot |
| Datadog monitors/SLOs (3 routes, tier1) | active |
| ArgoCD autosync | still disabled — re-enable via pham infra → [a] |