Encrypted workspaces

Incident runbooks

Five incidents, in the order they are likely to reach you. Each one states its answer before its procedure, because the first thing an operator needs is to know whether this is recoverable, and the second is what to say to the person asking.

Every step names a path or a command, what you should see, and what the refusal looks like. The error codes are the ones the API actually returns, and every one of them is opaque by design: the body carries a code, sometimes an envelope id and a workspace id, and never any content.

Five incidentsDiagnose before you act

Runbook

A device key is lost and tenant recovery is enabled

A laptop is gone, a browser profile was wiped, or a user opens a trace and sees "This browser cannot read this workspace".

The answer, before the procedure

Nothing is lost. The content is intact on the server and readable by every other key on the recipient list.

The work here is revocation and re-enrollment, not recovery. Recovery only enters if the user also needs to read the history that predates their new key.

Diagnose

Nothing in this section changes state. Establish the facts first.

DoOn any browser that still reads the workspace, open Settings, then Encryption, at /settings/encryption and read "Who can read this workspace".ExpectAt least one Active key besides the lost one, and a row labelled Tenant recovery key.If notIf no active key is left anywhere and there is no recovery row, this is not this runbook. Go to "Every key is lost and recovery is off" and follow its diagnostic before telling anyone anything.
DoRead the directory and find the identity id behind the lost key.ExpectAn entry whose recipient_key_id is the lost device's, with revoked false and eligible_recipient true.If notAn empty result means you are querying a different tenant. The directory is tenant-scoped and a session for another tenant sees none of it.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/identities?kind=user_device" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoAsk the user whether the device is merely inaccessible to them or actually in someone else's hands.ExpectA clear answer. Lost and possibly taken is a compromise, not a key loss.If notIf the answer is unclear, treat it as a compromise and follow "A device is suspected of being compromised" instead. That runbook is a superset of this one.

Respond

In this order.

DoRevoke the lost key: republish its identity record with revoked set to true. Then bump the epoch with the reason device_revoked and the full set of keys that remain.ExpectHTTP 200 on the revocation with eligible_recipient false, then HTTP 201 on the bump. The lost key is no longer wrapped into anything new.If note2ee/recipient-not-enrolled on the bump means the revoked key is still in the member_key_ids you sent. Remove it and send the set again.
shell
```
curl -sS -X POST "$CORE_API_ORIGIN/e2ee/epoch" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT" \
  -H "content-type: application/json" \
  -d '{
    "reason": "device_revoked",
    "actor_identity_id": "<your enrolled identity id>",
    "member_key_ids": ["<every remaining active key id>"]
  }'
```
DoHave the user enroll a key on their replacement browser and approve it, following the enrollment procedure in the operator guide.ExpectThe new key appears on the recipient list as Active, and the user can open a trace written after the approval.If notIf the user still sees an empty trace for new content, the approval did not include their key id. Compare the key id on their enrollment card against the last epoch's member set.
DoDecide whether the user needs the history. Re-approval is forward-only, so their new key reads content from its first epoch onward and nothing before it.ExpectA decision, recorded.If notIf they do need the history, the answer is a workspace rewrap run from an endpoint that can already read the old envelopes. The recovery custodian's endpoint qualifies, which is the concrete thing recovery buys you here. Approving the new key again does not backfill and never will.

Closeout

The recipient list shows the lost key as Revoked and the replacement as Active.
The epoch log carries the device_revoked entry with its actor.
The user can open a trace written after the bump, and understands why anything older is or is not visible to them.

Runbook

Every key is lost and recovery is off

No member can produce a working key for an encrypted workspace, and no tenant recovery key was ever enrolled.

The answer, before the procedure

If that is genuinely the situation, the content encrypted to those keys cannot be decrypted again. Not by the customer, not by support, not by Doug Fir Labs, and not later.

There is no master key, no support override, and no reset, because a system that had one would not be the system the customer bought. This is the same property that kept the platform from reading their traces.

The declaration is final, so run the diagnostic before you make it. It has found a way back more than once.

Diagnose

Nothing in this section changes state. Establish the facts first.

DoOpen /settings/encryption and read every Active row on "Who can read this workspace". For each one, ask who holds its private half and where.ExpectA named endpoint for each key, or a confirmed no. Check the places people forget: a second browser profile, a phone, another member who has not opened the workspace in weeks, an agent host, a CI runner.If notOne survivor is enough. If any endpoint still holds a private half, this is the lost-device runbook, not this one.
DoAsk whether any browser still has the workspace open, or has had it open since the keys were lost, without signing out.ExpectA browser whose local cache still holds decrypted records can still show what it decrypted earlier. Export what matters from it before anything else touches that browser.If notSigning out clears the cache, and so does the purge control on the encryption settings page. Tell the user not to sign out and not to select "Clear decrypted content from this browser" until the export is done.
DoCheck whether a tenant recovery key was ever enrolled, not just whether one is enrolled now.ExpectIf recovery was ever on, envelopes sealed during that window still list the recovery key as a recipient, because disabling recovery is not retroactive. The custodian of that key can still read those envelopes.If notAn empty list, and an epoch log with no recovery_enabled entry, closes this door. Read the epoch log before concluding: a revoked recovery identity is still a readable one for the envelopes it was a recipient of.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/identities?kind=tenant_recovery&include_revoked=true" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```

Respond

In this order.

DoIf the diagnostic found a key, a cache, or a recovery window, stop here and use the lost-device runbook. Otherwise continue.ExpectA clear determination, written down, naming what was checked.If notDo not continue on an assumption. "Probably nobody has it" is not the same finding as "we checked these six endpoints and nobody has it".
DoTell the customer plainly, in one message, without hedging: the content is encrypted to keys that no longer exist, Traceblox stores ciphertext and holds no key that opens it, and there is nothing to escalate to because there is nothing to escalate with.ExpectThe customer understands that this is the product working as designed and as disclosed, not an outage and not a bug.If notSoftening it is the worst option available. A customer who is left believing a recovery might arrive will wait instead of rebuilding, and will be angrier when the answer arrives late.
DoSay what is not lost. The workspace still works. New content encrypted to new keys is readable normally. The ciphertext and the server-visible metadata remain, and the account, members, and billing are unaffected.ExpectThe customer can plan around a bounded loss rather than an unbounded one.If notIf they ask whether the old envelopes can be deleted, that is their decision to make. They are unreadable ciphertext either way, and they continue to count as stored bytes for billing.
DoEnroll fresh keys and continue, following the enrollment procedure. Offer tenant recovery for the future and be explicit that enabling it now covers only what is written from that epoch onward.ExpectNew content readable, recovery decision made deliberately this time.If notDo not imply that turning recovery on now helps with the lost content. It does not, and saying otherwise sets up a second disappointment.

Closeout

A written record of which endpoints were checked and what each answered.
The customer has the plain answer in writing, without a pending escalation attached to it.
New keys are enrolled, the recovery decision is recorded, and the workspace is usable for new work.

Do not do these

Never open an escalation that implies a decryption path exists. There is nothing for engineering to look at.

Never suggest that a database restore, a backup, or a support tool might recover it. Every copy of that data is the same ciphertext.

Never ask the customer for a private key, a seed phrase, or a browser storage export, and never accept one if offered.

Runbook

A device is suspected of being compromised

A laptop was stolen, a machine is believed to have malware, a member reports their session was used by someone else, or a key id shows up from somewhere it should not.

The answer, before the procedure

Work in this order: revoke, bump the epoch, assess exposure, then decide on rewrap. Do not stop after the first two.

Revoking and bumping stop future reads. They do nothing about what that key could already read, and the assessment step is where you find out how much that is.

Diagnose

Nothing in this section changes state. Establish the facts first.

DoIdentify the exact recipient key id of the suspect device from the directory.ExpectOne recipient_key_id. Everything below is scoped to it.If notIf you cannot tell which of a user's keys is the suspect one, revoke all of theirs. An over-broad revocation costs a re-enrollment. An under-broad one costs the incident.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/identities" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoNote the epoch that was current while the device was trusted, and the epoch you are about to establish. That pair is the exposure window.ExpectTwo epoch numbers, from the epoch log.If notIf the epoch log cannot be read back cleanly, record that as part of the incident. Every entry carries the hash of the one before it, so a chain that does not verify is itself a finding.

Respond

In this order.

DoRevoke. Republish the suspect identity with revoked set to true.ExpectHTTP 200 with eligible_recipient false. From this point an envelope that wraps to that key is refused with e2ee/recipient-not-enrolled, and an envelope signed by it is refused with e2ee/sender-not-enrolled.If note2ee/forbidden (403) means the session lacks members:manage. Get an owner or admin now rather than working around it.
shell
```
curl -sS -X POST "$CORE_API_ORIGIN/e2ee/identities" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT" \
  -H "content-type: application/json" \
  -d '{ "identity": <the published identity record>, "revoked": true }'
```
DoBump the epoch with the reason device_revoked, naming every key that should still read the workspace.ExpectHTTP 201, and a new epoch whose recipient set excludes the suspect key. Envelopes written from now on are unreadable by it.If note2ee/no-recipients (400) means you sent an empty set. e2ee/recipient-not-enrolled (400) means the revoked key is still in the list.
shell
```
curl -sS -X POST "$CORE_API_ORIGIN/e2ee/epoch" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT" \
  -H "content-type: application/json" \
  -d '{
    "reason": "device_revoked",
    "actor_identity_id": "<your enrolled identity id>",
    "member_key_ids": ["<every remaining active key id>"]
  }'
```
DoAssess exposure. Page the envelope list and count the envelopes whose recipients include the suspect key id. That count and its sequence range are the exposure.ExpectA number, a sequence range, and a byte total. The server can tell you which envelopes and how many bytes, and it cannot tell you what is in them, because it cannot read them.If notDetermining what was actually in those envelopes needs a client with a key to decrypt them. That is a customer-side task, and it is the honest limit of what the platform can answer.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/envelopes?limit=500" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoExtend the assessment past the envelopes. Assume the endpoint held a decrypted local cache if it was signed in, and check whether it ever sent workspace content to a hosted model provider.ExpectA written exposure note covering the envelope range, the local cache, and any provider the agent on that endpoint was configured against.If notIf the endpoint sent content to a hosted model provider, no key operation reaches that. It is a separate disclosure with a separate scope.
DoDecide on rewrap. Rewrap when the compromise is credible and the content matters, so the suspect key stops being a recipient of anything stored on Traceblox.ExpectA decision, recorded. If yes, an authorized endpoint decrypts and re-encrypts every envelope under the current epoch and the current recipient set. It is safe to retry, because an envelope already at the target is skipped.If notA rewrap does not reach a copy the attacker already downloaded. If you believe content was exfiltrated, the rewrap is still worth doing and it is not the response. The response is content-level.
DoRotate the credentials that are not keys: the workspace API keys at /api-keys, and any session that endpoint held.ExpectThe compromised endpoint cannot authenticate to the workspace at all.If notSkipping this leaves a working credential in place. E2EE does not replace authorization, and a revoked key with a live API key can still enumerate metadata.

Closeout

The recipient list shows the suspect key as Revoked, and the epoch log carries the device_revoked entry with its actor.
A written exposure note names the envelope count and sequence range the key could read, plus the cache and provider assessment.
The rewrap decision is recorded either way, and API keys have been rotated.

Runbook

A support request needs data the platform cannot decrypt

A customer asks support to look at a trace, explain why one renders empty, recover content, or debug a decryption failure in an encrypted workspace.

The answer, before the procedure

Say the true thing first: we can see the envelope metadata and nothing else, and that is the product working rather than a limitation of our tooling.

Then help with what we can genuinely see, which is more than it sounds like. Most of these requests are diagnosable from metadata alone.

If the customer wants us to see content, they grant it explicitly, for that incident, for the envelopes they choose. There is no standing grant and no route on our side that creates one.

Diagnose

Nothing in this section changes state. Establish the facts first.

DoConfirm the workspace is actually encrypted before applying any of this. Read the migration status route, which reports encryption_mode.Expectencryption_mode is e2ee, with a current_epoch and a policy_version.If notIf it reports plaintext, the ordinary trace tools apply and this runbook does not. Do not tell a plaintext customer that we cannot read their data.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/migration" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoCheck whether the workspace holds any envelopes at all.ExpectA count above zero, with a next_cursor when there is more.If notA count of zero means nothing has been written in encrypted form. The problem is on the writing side, not the reading side, and the next question is whether the customer's agent negotiated the capability.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/envelopes?limit=1" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoCheck whether the reader's key is on the recipient set. Compare the key id shown on their /settings/encryption page against the recipients of a recent envelope and against the directory.ExpectEnrolled and a recipient means they should be able to read. Enrolled and not a recipient is the waiting-for-approval case. Revoked is the offboarding case.If notA key that is not in the directory at all has never been published. The customer's browser has not completed enrollment, and the fix is on their side, on the encryption settings page.
DoIf the customer reports a write failing, read the error code they received. e2ee/capability-required means their MCP client did not advertise com.dougfirlabs/scholia-e2ee during the handshake.ExpectA client configuration answer. The peer must negotiate the capability and write ciphertext through the envelope routes.If notThere is no plaintext fallback and no flag that enables one. A request to turn one on for a customer is a request to break the product's central promise, and the answer is no.
DoFor a write that reached the server and was refused, correlate on the envelope id in the error body. e2ee/signature-invalid means the sender's enrolled Ed25519 key does not match what signed. e2ee/sequence-not-monotonic means a client is replaying or two writers are racing. e2ee/aad-mismatch means the envelope names a different tenant or workspace than the credential.ExpectA specific cause, tied to a specific envelope id, which is server-visible and safe to quote back.If notIf none of the codes fit, escalate with the envelope id, the error code, the workspace id, and the timestamps. Never attach content, ciphertext, or a wrapped key, because none of those help and all of them are the thing we promised not to move around.

Respond

In this order.

DoWhen the diagnosis needs content, ask for a customer grant rather than access. Enroll a support identity named for this incident and give the customer only its recipient key id.ExpectA directory entry for a support identity, with a label that names the incident so it is recognizable on the recipient list.If notNever send a private key in either direction, and never accept one. The publish route refuses a body carrying anything shaped like a private half, with e2ee/private-key-material-rejected.
DoThe customer decides which envelopes, and their client wraps only those to the support key. Scope it to the incident and nothing wider.ExpectSupport can decrypt exactly the envelopes the customer chose, and nothing else in the workspace.If notIf support finds it can read content the customer did not wrap to it, that is an incident on our side and it gets reported as one, immediately.
DoClose the grant when the incident closes. The customer revokes the support identity and bumps the epoch with the reason member_removed.ExpectThe support key stops being a recipient of anything written afterwards.If notRevocation is forward-only, so it does not un-share what support already read. Support deleting its copy is a procedural promise and not a cryptographic one, and it should be described that way rather than dressed up.

Closeout

The customer has an answer that names what we could see and what we could not.
Any grant that was opened has been revoked, with an epoch bump recorded.
Nothing in the ticket, the logs, or the escalation carries ciphertext, a wrapped key, or decrypted content.

Do not do these

Never ask for a private key, a seed, a passphrase, or an export of browser storage.

Never suggest disabling encryption temporarily. There is no such operation, and offering it invents a capability the product does not have.

Never describe the platform as unable to see anything at all. It sees the published metadata list, and claiming otherwise is a claim we would have to retract.

Runbook

A migration failed or was interrupted

The browser was closed mid-migration, the tab crashed, the network dropped, the run reported a verification mismatch, or the phase text reads "Stopped on an error".

The answer, before the procedure

Nothing readable was removed. The readable copies are removed only after every planned record has been read back and decrypted successfully, and the database refuses the flip when that has not happened.

Whatever state the run stopped in, the workspace's plaintext is intact. Confirm that first, then resume.

Diagnose

Nothing in this section changes state. Establish the facts first.

DoRead the migration status.Expectencryption_mode plaintext, and a migration with a state of planned, uploading, or verifying, plus planned_count, uploaded_count, and verified_count.If notencryption_mode e2ee means the migration completed. There was no data loss, there is nothing to resume, and there is no route back.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/migration" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoConfirm the readable source is still there, rather than trusting the state field.ExpectA page of source records, with has_more true on a workspace of any size.If note2ee/encryption-mode-is-one-way (409) means the flip already happened and there is no source left to serve. That is the completed case, not a loss.
shell
```
curl -sS "$CORE_API_ORIGIN/e2ee/migration/source?limit=1" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT"
```
DoCompare uploaded_count against planned_count and verified_count against uploaded_count to see which phase it stopped in.ExpectA phase. Uploading and verifying resume differently in effort, not in method.If notCounts that do not move across two readings mean nothing is running. That is expected after an interruption, and it is the state the resume path is for.

Respond

In this order.

DoResume. On a browser with an approved key and an owner or admin session, open /settings/encryption and select "Encrypt this workspace" again, with the same recovery decision as before.ExpectThe client re-derives the plan, reopens the same run, pulls the envelope ids already stored, and uploads only what is missing. Progress resumes near where it stopped rather than at zero.If note2ee/migration-conflict (409) has two causes and they need different fixes. See the next two steps.
DoIf the conflict is because the workspace changed since the run opened, the plan hash no longer matches. Abort the stale run with the reason source-drift, then start a fresh one.ExpectThe abort returns source_intact true. An abort deletes nothing, ever, and the response says so explicitly.If note2ee/invalid-request on the abort means the reason is not one of verification-mismatch, source-drift, cancelled, or upload-failed.
shell
```
curl -sS -X POST "$CORE_API_ORIGIN/e2ee/migration/abort" \
  -H "authorization: Bearer $SCHOLIA_SESSION_JWT" \
  -H "content-type: application/json" \
  -d '{
    "migration_id": "<the migration_id from the status route>",
    "reason": "source-drift",
    "actor_identity_id": "<your enrolled identity id>"
  }'
```
DoIf the conflict is because a different run is already active, abort that one with the reason cancelled before starting yours. Only one migration is open per workspace at a time.ExpectThe active run moves to aborted, with the source untouched, and a new run can open.If notIf two operators are working the same workspace, stop and coordinate. Two runs racing produce conflicts, not corruption, but they waste an upload each.
DoIf the run stopped on a verification mismatch, it aborted itself with the reason verification-mismatch and removed nothing. Run it again.ExpectA second run usually completes. A transient upload or decode failure is the common cause.If notIf it mismatches again on the same records, stop and escalate with the item ids, the counts, and the migration id. Do not attach any record content: the ids are what identifies the problem and the content is what we promised not to move.
DoIf completing is refused with e2ee/migration-not-verified (409), verified_count is short of planned_count. Do not retry the completion. Re-run the verification pass.Expectverified_count reaches planned_count, and only then does the completion succeed.If notThis refusal is the safety property working. Both the route and a database trigger enforce it, so there is no sequence of requests that removes the readable copies early.
DoIf you decide not to continue at all, abort explicitly with the reason cancelled rather than walking away from an open run.ExpectState aborted, source_intact true, workspace unchanged and still plaintext.If notLeaving a run open blocks the next one with e2ee/migration-conflict, which is a confusing error to hand the next operator.

Closeout

After a successful completion, /settings/encryption shows the workspace encrypted and migrated_at is set.
The source route answers e2ee/encryption-mode-is-one-way, which is the confirmation that the readable copies are gone and the flip is durable.
After an abort, the workspace is still plaintext, every readable record is still there, and the run appears in the migration history with its reason.

Reference

What the platform can see

Support diagnoses from this list and nothing else. It is the same list published on the public security page, rendered from the same source, so what an operator works from and what a customer was told cannot drift apart.

Visible to Traceblox

Opaque tenant identifier. A random identifier that separates one customer's records from another's.
Opaque workspace identifier. A random identifier that groups envelopes into the workspace they belong to.
Opaque trace identifier. A random identifier that groups envelopes into a single reasoning trace.
Opaque envelope identifier. A random identifier for one stored envelope, used to fetch and to anchor it.
Ciphertext format version. The version string of the envelope format, so old and new envelopes can be told apart.
Recipient key identifiers. Which public keys an envelope was encrypted to, so a client can find the wrapped key it can open.
Key epoch. A counter that increases when workspace membership changes, so clients know which recipient set applies.
Sequence and replay fields. Ordering values the server checks to reject a replayed or reordered envelope.
Timestamps. When an envelope was created and stored.
Ciphertext byte size. How large the encrypted payload is, which storage and billing both need.
Coarse payload class. Whether an envelope holds a trace, an atom, an edge, a summary, or an attachment, and nothing finer.
Retention state. Whether an envelope is live, scheduled for deletion, or deleted.
Billing counters. The counts and byte totals required to produce an invoice.

Not visible to Traceblox

These are inside the ciphertext. No support tool, database query, or log reads them, and no escalation changes that.

Trace titles. Encrypted in the envelope payload. The platform stores the ciphertext and cannot read the title.
Trace objectives. Encrypted in the envelope payload, so the stated goal of a trace is not visible to the platform.
Tags. Encrypted in the envelope payload, so tag names and values are not available for server-side grouping.
Atom summaries. Encrypted in the envelope payload. Summaries are produced and read on the client.
Atom content. Encrypted in the envelope payload. This is the body of the reasoning and the platform never holds it in the clear.
Atom kinds. Encrypted rather than exposed, so the platform cannot infer reasoning structure from the distribution of kinds.
File paths and references. Encrypted in the envelope payload, including paths that would otherwise name a customer's repository layout.
Edge relations and labels. Edges are envelopes too, so graph topology is reconstructed on the client and is not visible to the platform.
Generated summaries. Encrypted in the envelope payload. Summarization happens on the client, not on the server.
Exports. Encrypted in the envelope payload, so an exported bundle stored by the platform stays unreadable to it.
Attachment contents. Encrypted in the envelope payload. The platform holds the bytes and the size, not the file.