We killed the primary region. The memory survived.
Survivability is easy to claim and hard to show. So we built a nine-node, three-region cluster and drained every node in the database’s primary region while thousands of memories were in place — then asked the agent to keep working.
What actually happened.
Thousands of memories, three regions
Incidents, runbooks, and the agent’s thought stream distributed evenly across us-east, eu-west, and ap-south, each pinned to its home region.
Primary region drained
Every node in the primary region taken offline. Reads of rows homed there kept answering from surviving replicas; recall never returned an error.
New memory to a dead region
A memory homed in the downed region was written and committed — quorum reached from the two surviving regions. Zero data loss on restore.
Recall is served by the distributed vector index.
The query plan shows a vector search over the C-SPANN index, fanning across a prefix span per region — distributed approximate nearest-neighbour, not a full scan.
> EXPLAIN SELECT id FROM incidents ORDER BY embedding <-> $1 LIMIT 5;
• top-k (k: 5)
└── • lookup join (incidents@incidents_pkey)
└── • vector search
table: incidents@incidents_embedding_idx
prefix spans: [/'aws-ap-south-1'] [/'aws-eu-west-1'] [/'aws-us-east-1']Reproducible from the repo: a scripted `cockroach demo --demo-locality` rig with a driver that drains real nodes on command.
What’s real, stated plainly.
Per-node kill is demonstrated on a local nine-node rig, where raw node liveness is observable. On managed CockroachDB Cloud, node-level control isn’t exposed to tenants — there, survivability is a property of the database configuration (SURVIVE REGION FAILURE), and the console’s failure drill runs a live exclusion query to show surviving regions still answer. Both are shown; neither is faked.
Three regions. One answer. Geography picks the podium.
One query, embedded once, raced against all three regional gateways of the same cluster simultaneously. Every leg is a follower read served from that region’s local replicas, and gateway_region() confirms where each answer entered. The latencies differ by an ocean; the answers never differ at all.
“checkout-api p99 latency spike, connection pool exhausted”