Documentation embedded in this build.

Move labelled data only to the services cleared for it

What it is for

You have a feed — readings, events, records coming off a collector — and two places it has to go that are cleared to different levels. Point the producer at the front door and let it write records down one long-lived connection. Each record carries a small label, and the door decides on the label alone: it never buffers the record, never parses it, never stores it and never opens it.

One producer holds a single long-lived connection to a front door, which reads the label on each record and never its body; two arrows leave the door, each carrying a signed batch receipt, to two ingest services — one cleared for SECRET, one cleared for CONFIDENTIAL — and a record labelled SECRET reaches the first and never the second, where the refusal is recorded

Each destination is an identity of its own with its own certificate, and the level it is cleared to comes from the same directory everything else here comes from — so a record labelled above a destination's level is not sent there, the refusal is written down naming the destination, and the record still reaches the destinations that may have it. You do not have to run a flow engine to get that, and you do not have to trust one.

The door signs a receipt for each batch it delivers to each destination, and every record is chained to the one before it, so a destination can prove exactly what it was given, in order, with nothing missing — from one signature rather than one per record. If the contents should be hidden from the door as well, the producer can seal each record to exactly the destinations its label will reach, and the door never holds a key that opens one. And when the producer's certificate is revoked or its standing changes, the connection is closed and the reason is recorded: this is not a permission granted once and honoured for a year.

This is not a message broker or a data pipeline. Nothing is stored, replayed or transformed.

The topology

One producer, one connection, and a door that reads labels and never bodies.

                                             +--> INGEST, cleared higher
   PRODUCER  ==one long-lived connection==>  |      gets the record labelled higher
   labels each                  FRONT DOOR   |      + a signed batch receipt
   record                       reads the    |
                                label, never +--> INGEST, cleared lower
                                the body     |      does NOT get that record
                                             |      the refusal is written down

One producer, one connection, two destinations, one receipt per batch per destination.

Stand it up

$ make all-in-one-federated   # two ingest services, at two levels, behind the front door
$ make rehearsal              # a producer streams labelled records through them
$ make scenario-federation    # two instances, each its own authority

make rehearsal runs against a stack that is already up — bring the federated composition up first and leave it running. The scenario brings its own pair of instances up and takes them down.

What the smoke proves

The composed run stands both destinations up as real containers with real certificates, at two different levels, behind the real front door, and streams records through them. What only a running composition can show is that the label the producing library writes is the label the door reads — two implementations of one grammar — that the levels being compared are the staged deployment's own, and that the door's own entity is a link in the chain rather than a bystander. Each mechanism's own behaviour is proved in Go beside it.

make rehearsal drives that composition for a couple of minutes under load and renders its report, so the property is exercised at a cadence rather than once.

make scenario-federation runs the Go twin TestFederation: two instances, each its own authority, with a signed bundle adopted live by the second and a write relayed as the caller and decided upstream. It is the distribution half of the same story — what the door compares against had to get there somehow.

Read on

make:all-in-one-federated pins two destinations at two levels behind one door.
make:rehearsal pins the same path under a running load rather than in a single pass.
scenario:federation pins the signed distribution the comparison depends on.
The operator runbook is authorized stream operations, in the operator guide: a stream route inherits the door's bundle, its revocation lists, its receipt key and its reserved paths, and that page adds only what a stream does on top of them.
The sealed records scenario is the shape to read when the contents must be hidden from the door as well.