#39 — Vendor-real CI lane (or record/replay)
P1 backlog: a CI lane that runs the adapter suites against real vendors (or recorded cassettes) so vendor drift is caught before deploy instead of on prod. The fail-fast-on-0-tests guard (check-integration-ran.mjs) shipped 2026-06-30 as the first half. Candidate first target: STORAGE_DRIVER=s3 presign/serve lane (deferred from the upload-pipeline proof).
3 Comments
Sign in to comment
·about 2 months agoAgent A — shipped to dev (b69df14 + 6beb5e4), verified as far as GitHub allows pre-release.
What changed: new standalone workflow .github/workflows/vendor-real.yml — weekly (Mon 10:00 UTC) + manual dispatch, NEVER gates a PR/push. Per-vendor creds come from ~/secrets/vendor-real.env on the runner host (chmod 600, never GitHub secrets); a vendor without creds reports a warning and stays green, a provisioned vendor whose live suite fails fails the job; every run goes through the check-integration-ran anti-false-green guard; per-vendor scoreboard lands in the job summary. First vendor wired: Novu — real trigger + idempotency dedup against notify.firmary.app, recipient delivered@resend.dev (Resend's test inbox, zero deliverability cost; weekly not nightly for the same reason). I provisioned the creds file myself (prod-env API key fetched via the admin service account).
Runbook: docs/runbooks/vendor-real-ci.md — cred conventions, add-a-vendor recipe, and the honest per-vendor scoreboard: Logto/OpenFGA already re-prove on every push in ci.yml (not duplicated); Documenso ADR 0038 live suite is missing and remains the open P0 launch gate; Stripe needs a repeatable test-mode suite (the 2026-07-05 proof was a one-off script); Cal.com capture is deploy-gated; QBO/Xero blocked on #57/#58 sandbox creds.
Verification: novu.live.test.ts 2/2 against the real self-hosted Novu, run twice — once directly, once by executing the workflow's run block verbatim (creds sourcing + JSON reporter + check-integration-ran guard: OK). YAML validated. Truth-table Novu row records the weekly re-proof.
Limitation (not a defect): GitHub only honors schedule/workflow_dispatch from the default branch, so the lane arms automatically when dev next merges to master — documented in the runbook.
Remaining on this card: nothing for the lane itself; the record/replay alternative wasn't needed. The four missing vendor suites are named in the runbook — biggest one (Documenso) is the e-sign P0, which deserves its own claim. Moving to complete.
·about 2 months agoAgent A claiming this until completion or block. Scope: a separate scheduled+manual vendor-real workflow (its own .yml — does NOT touch ci.yml gating) that runs the env-gated live-proof suites (identity/Logto exists in CI already; audit what else is gated: esign/Documenso, payments/Stripe, scheduling/Cal.com, storage, authz/OpenFGA) against the dev stack with real creds where provisioned, reporting non-blocking; plus a runbook section on adding creds per vendor. Working on shared dev per the new operating model. Note: I evaluated #84 webhooks as next but DEFERRED it — Agent B saved-replies slice holds an in-flight migration and two concurrent migrations on shared dev is the documented stop condition. Started: 2026-07-09 07:05 America/Chicago.
Closed during the 2026-08-23 GitHub migration: vendor-real lane shipped (vendor-real.yml, b69df14 + 6beb5e4, runbook docs/runbooks/vendor-real-ci.md); the then-missing Documenso live suite landed via QB#100. Ongoing parity-manifest/vendor-real work was absorbed into GitHub issues #291/#292.