Architecture

Not a vector store you could swap for anything.

BlackBox is designed around capabilities only CockroachDB brings together in a single system of record. This is the answer to the question every judge asks: why this database?

REGIONAL BY ROW
Data residency, by row.

Each memory is pinned to its home region via crdb_region. An EU incident’s data physically stays in the EU; local reads stay fast. One logical database, per-row domiciling — no second system.

SURVIVE REGION FAILURE
Memory that outlives the outage.

Lose an entire cloud region and the memory database stays readable and writable from surviving replicas, strongly consistent, with no data loss. The reason this runs here and not on a single-region vector store.

VECTOR INDEX — C-SPANN
Distributed approximate nearest-neighbour.

Region-prefixed vector indexes keep each region’s k-means tree co-located with its data. Semantic recall over thousands of incidents, answered locally and survivably, using pgvector-compatible operators.

Managed MCP Server
The agent reads its own database.

A managed endpoint exposes read-only SQL over the Model Context Protocol. The agent introspects the live cluster it operates — schema, health, running queries, its own memory counts.

System

How the pieces fit.

OperatorConsole · Next.js — chat, incident timeline, survivability panel
AgentReason ↔ recall ↔ act loop · Claude on Amazon Bedrock, typed tool use
EmbeddingsTitan Text Embeddings v2 · 1024-dim vectors for every memory
MemoryCockroachDB Cloud — 3 regions · REGIONAL BY ROW · SURVIVE REGION FAILURE · distributed vector indexes. Introspected by the agent via the Managed MCP Server.
The schema

The whole idea, in one table definition.

Every memory table looks like this: a region-partitioned primary key, a 1024-dim vector column, and a distributed vector index prefixed by region.

ALTER DATABASE blackbox SURVIVE REGION FAILURE;

CREATE TABLE incidents (
    id           UUID NOT NULL DEFAULT gen_random_uuid(),
    service_id   UUID NOT NULL,
    title        STRING NOT NULL,
    summary      STRING NOT NULL,
    resolution   STRING,
    embedding    VECTOR(1024),              -- Titan v2
    crdb_region  crdb_internal_region NOT NULL
                 DEFAULT gateway_region()::crdb_internal_region,
    CONSTRAINT incidents_pkey PRIMARY KEY (crdb_region, id),
    -- distributed ANN index, co-located per region
    VECTOR INDEX incidents_embedding_idx (crdb_region, embedding)
) LOCALITY REGIONAL BY ROW;
Data residency

Proved live, not promised.

Residency is a per-row property here, not a separate deployment. This reads straight from the running cluster.

residency · EU

This memory is pinned to aws-eu-west-1. The query below runs from the aws-us-east-1 gateway, yet the row is served from — and only ever stored in — its home region.

SELECT crdb_region, title
  FROM incidents
 WHERE crdb_region = 'aws-eu-west-1'
 LIMIT 1;
-- crdb_region  | title
-- aws-eu-west-1 | …
memories per region
The comparison

Why not pgvector, DynamoDB, or Redis?

The survivability demo eliminates each usual choice: pgvector loses the agent’s memory when its region goes down; DynamoDB global tables are eventually consistent, so live state and recalled memory can disagree mid-crisis; Redis is fast but not a durable system of record; and a dedicated vector DB bolted to a separate state store is two systems to keep in sync, split-brain during the one outage you can least afford it.

A bolted-together stack
  • Vector DB + cache + relational state store to keep in sync
  • Single-region, or eventual consistency across regions
  • Data residency needs a separate database per region
  • An outage takes the agent’s memory with it
  • Live state and recalled memory can disagree mid-incident
BlackBox on CockroachDB
  • One system of record — vectors and transactional state together
  • Strongly consistent across three regions
  • Residency is a per-row property, not a deployment
  • Survives a full region loss with zero data loss
  • The agent can query its own memory to verify itself
Production readiness

Built like it will be operated.

Security

Guarded by default

Read-only, statement-checked MCP access; parameterised SQL throughout; TLS to the cluster; least-privilege SQL user; CSP and HSTS on every response; no error internals leak to clients.

Access control

Least privilege, twice over

The MCP service account holds only the read-only role it needs; the Bedrock IAM policy is scoped to InvokeModel on two models. Durable rate limiting rides the database itself — per-client windows survive serverless instance churn.

Reliability

Fails soft, never silently

Exponential backoff on embedding throttles; idempotent writes; the agent is stateless — all durable state lives in CockroachDB. If a memory write fails, the response says so: degraded memory is surfaced, not swallowed.

Observability

Every action is an event

Each tool call and result is inspectable; every memory is queryable; every memory-write decision lands in an auditable hygiene ledger; a real test suite covers recall, the gate, the loop, and rate limiting.

See the survivability test