Planning

A trusted owner agent (Daisy/Simon-style) can already initiate a local-collector connection through POST /v1/owner/connections/intents, but a browser-bound connector such as a second Amazon account returns unsupported. The reason is honest: the reference has no enrollment primitive that lets a local collector drive a real browser session and ingest through the device-exporter path. The enroll route hardcodes sourcekind: "localdevice" and does no binding-aware validation, so there is no way to record that a collected binding is browser-collected rather than filesystem-read.

Railway has a proven one-click Core deploy and Fly.io has a proven one-command launch, but the Docker path still demands a repository clone, a large env file, and the development/owner compose stack — the owner's words: "right now I am overwhelmed looking at the link for docker." The standalone Core image already contains everything needed; what is missing is two small image gaps (no localhost origin default, no first-boot owner credential) and a deliberately small user-facing surface (one docker run quickstart, one minimal production compose) per design-notes/deploy-surface-parity-2026-06-10.md and docs/research/deploy-button-parity-prior-art-2026-06-10.md.

Owners expect "Google Maps" in Add source to mean a live Google-account authorization flow when Google exposes one. The current Google Maps work only imports owner-provided Timeline files, and presenting that as a Gmail-like connection would be dishonest.

SUPERSEDED (2026-06-10) by converge-provider-rate-governance. This > change's rate-governance axes (per-provider pacing, ratio-based retry budget, > circuit breaker, run-budget envelope, detail-gap drain loop) landed and are > absorbed by the convergence change, which corrects the layer-ownership model: > a provider request path has exactly ONE pre-flight send governor (the AIMD > concurrency lane), and GCRA pacing is a signal folded into it, not a second > independent pre-flight gate. To avoid two active changes both adding the same > polyfill-runtime requirements, only converge-provider-rate-governance > carries the rate-governance deltas to archive; this change is parked. Its > still-independent work — commit-gated/opaque-cursor checkpoint durability > (§2.5) and catch-up vs. steady-state bookmark separation (§2.6) — is NOT > absorbed and should move to a dedicated cursor-durability change if pursued. > See converge-provider-rate-governance/design.md ("Disposition of > add-provider-budget-run-control").

The chase/statements and usaa/statements retained histories churn on every re-download even when the owner-visible statement is unchanged. The statement PDFs are content-addressed by pdfsha256 = sha256(raw bytes), but the raw bytes are not the content: Chase statement PDFs are RC4-encrypted and the source regenerates the per-download encryption key material and embedded generation timestamps on every fetch, so pdfsha256 (and the pdfpath/documenturl that embed it) moves with zero change to the decrypted text or page count. Read-only evidence (tmp/workstreams/ri-version-rationality-evidence-v1-report.md) proved the decrypted text sha and page count are invariant across this churn for every comparable Chase blob pair, and that USAA's own PDF-derived transactions are byte-identical content across the same pdf_sha256 churn.

The release train ran two channels: publishable work landed on main, but semantic-release published only when an owner advanced the beta branch and pushed it, cutting 0.1.0-beta.N prereleases to npm's beta dist-tag. The beta branch was a second moving part with no countervailing benefit:

Several first-party connector manifests expose owner-visible natural-language fields as readable schema fields without declaring the retrieval and presentation affordances clients need to use them well. The concrete trigger was WhatsApp messages.content: it was granted/readable but not advertised as lexical or semantic searchable. The same pattern existed in other supported connectors.

The current MCP closeout proves the ChatGPT Slack evidence path: search can show bounded evidence, readrecordfield can return an inline field window, and projected fetch avoids file materialization for ordinary messages.

The current hosted MCP surface exposes every read and event-subscription tool in one flat tools/list response. The first footprint tranche removes duplicated prose, but it intentionally preserves the 14-tool topology. That is still too broad for the normal read/query setup path.

Two confirmed vulnerabilities (program audit wave 2, S-1 + S-2) let an internet-facing reference deployment expose its owner control plane to anyone:

ship-adaptive-collection-rate-controller proved the SLVP-ideal adaptive rate controller LIVE on ChatGPT (19 → 32.7 conv/min): slow-start discovery, AIMD accelerate-under-success, a single owner-authored rate ceiling it never probes across, warm-start that compounds the learned rate across runs, and an operator- legible collection_rate readout.

GET /_ref/connectors is a core owner-console read path. Recent performance tranches made it fast by adding a short in-process cache and by making overview rows shallow, but that is not the final construction: the read path still has a large per-connection evidence fan-out underneath the cache, and record-ingest writes do not event-invalidate the summary projection.

The Postgres semantic-search path stores embeddings in semanticsearchblob.embedding as JSONB (384-dim float arrays, roughly 4.8 KB/row versus roughly 1.5 KB as a pgvector vector) and answers queries by SELECTing candidate rows and brute-force cosine-scoring them in JavaScript (postgres-search.js postgresSemanticSearch). The live deployment already runs the pgvector/pgvector:pg16 image, so the vector extension is available but unused. At the live table size (~1.85M rows / ~10 GB) the JSONB representation wastes roughly 3× the storage and the brute-force read path ships every candidate embedding over the wire to score it in JS — worse, the candidate SELECT carries a bare LIMIT with no ordering, so on scopes larger than the per-connector overscan the JS pass scores an arbitrary candidate subset rather than the true nearest neighbors.

The reference implementation maintains two storage engines (SQLite via records.js/search.js, Postgres via postgres-records.js/postgres-search.js) dispatched by isPostgresStorageBackend() branches scattered across the server. A full inventory (tmp/workstreams/storage-backend-pilot-design-2026-06-18.md) classified all 187 such branches: 153 are drifttoremove (the same conceptual storage operation implemented twice inline in shared orchestration), 21 are adapterselectionkeep (the legitimate backend-selection points, already the right pattern), and 14 are backendspecifickeep (honest dialect logic that is clearer kept backend-specific: SQLite WAL retry, Postgres FOR UPDATE locks, sqlite-vec vs pgvector).

The live site's fast merged cross-source timeline feed and the Explore over-time chart run on deploy-branch-only server code that was never merged to main: explore-timeline-substrate.ts (dual-dialect merged-timeline queries), the rs.explore.timeline operation, the GET /ref/explore/records route, and the semantictime column plus its idxrecordssemantictime / idxpgrecordssemantictime expression index. On fresh origin/main none of these exist: the records table carries only emittedat (ingest time), and the only timeline reader is GET /_ref/records/timeline (the ref-records-timeline operation), which orders by ingest time rather than authored/semantic time.

The PDPP grant model exposes a single_use access mode that is central to the protocol's safety story — it bounds the blast radius of a grant to one retrieval session and prevents silent reuse. The reference implementation has enforced this atomically since the initial grant implementation, but the enforcement had no HTTP-boundary proof:

@pdpp/mcp-server is the canonical MCP adapter for grant-scoped PDPP reads. It is advertised as an npx -y @pdpp/mcp-server command in:

The June-6 image-slimming change made the default reference image browser-free (the browsers Dockerfile stage was retained but its output was never wired into CI publication). Deployments that run browser-backed connectors (ChatGPT, USAA, ...) inside the reference container hit a silent hard failure at Patchright launch: "Executable doesn't exist at /opt/patchright-browsers/...". The fix took four days to diagnose because the image advertised no build-time signal that browsers were absent.

Explore is the owner's window into their own data, but its interaction model is the sum of independently-added parts, not one coherent system. The failures are systemic: a count can promise more records than the UI can reach ("188 upcoming" surfaces only 32); collapse/expand/load-more were not designed as one state machine; there are multiple search inputs and a confusing split between facet filters and typed operators with no way to invert a selection; and record cards pick which field to show by brittle field-name/stream-name heuristics, so arbitrary connectors render wrong. Owner feedback (captured in docs/research/explore-experience-feedback-2026-06-21.md) names all of these. A dual-owner strategy assessment (tmp/workstreams/codex-explore-strategy- assessment.md) concluded the diagnosis is right but two load-bearing abstractions must be made explicit before implementation: a canonical RecordSet / reachability contract, and a manifest-authored record presentation recipe. This change makes those explicit and threads the count == reachability invariant through every named set, grounded in prior art (docs/research/explore-query-filter-ia-prior-art-2026-06-21.md, docs/research/explore-feed-interaction-dynamics-prior-art-2026-06-21.md).

The owner console still fails the product promise even after several verified UI tranches. the owner's 2026-06-18 walkthrough shows the same root pattern across Dashboard, Sources, Add Data, Explore, Runs, Grants, Traces, Owner Tokens, and local-collector recovery: the console exposes implementation artifacts as primary UX objects, computes the same truth in multiple places, and asks the owner to infer the next action.

The hosted MCP surface currently repeats the same cross-cutting guidance across many tool descriptions, producing a ~49.6 KB tools/list payload for 14 tools. That cost is paid by every MCP client session and sits in an unpredictable zone for chat-hosted clients whose exact tool-description and tool-result budgets are host-defined.

Slack targeted channel repair can fetch a scoped source archive that contains messages older than the saved per-channel cursor. Those historical holes must be emitted during the repair; otherwise the run can succeed while retained records still miss source archive keys.

@pdpp/remote-surface is the extracted streaming and control substrate (geometry, pointer mapping, mobile IME, clipboard policy, n.eko/CDP adapters, diagnostics, leases, testing fixtures). It is reusable infrastructure for any remote-browser surface, and PDPP is one consumer among many we expect.

The reference Postgres lexical search path is now fast and honest, but broad queries can still rank only a bounded candidate window. That is an interim quality compromise: callers can tell recall is bounded, but the implementation can still miss a better lexical match outside the window.

The ChatGPT connector's adaptive collection rate controller is dressed in AIMD machinery that a hand-tuned floor disables. converge-provider-rate-governance correctly collapsed the two pre-flight waits into one (the AdaptiveLane is the sole send governor; GCRA pacing rides as a launchDelayHint), but three incidental constants and one cap-era policy still defeat the loop:

The first connector semantic-affordance pass made message-like text searchable and added presentation roles, but the live audit still found useful query semantics left implicit: time fields without range or bucket declarations, owner-recognizable fields without search/facet affordances, and ambiguity between presentation event-time roles and query-time date affordances.

The ChatGPT connector currently hydrates conversation detail with one GET /conversation/{id} request per conversation. Large accounts turn that into a provider request storm, increasing run time and 429 pressure even though ChatGPT exposes a batch detail endpoint that returns the same conversation-detail shape for up to 10 ids per request.

PDPP now exposes read paths through REST, CLI, MCP, and the owner console. The surfaces share the same underlying records and grants, but their evidence presentation, truncation, continuation handles, and display semantics can drift.

The main ruleset requires the typecheck + full test suite status check, but the workflow that emits that context is path-filtered. PRs that do not touch the filtered paths can become blocked because the required context never appears.

The owner console repeatedly reads the full connection-summary projection during one browser navigation. On the live Postgres instance, that projection is the shared hot path behind slow /dashboard/runs loads and can be fetched multiple times by RSC requests.

Large Postgres-backed record streams were not "blazing fast" on the public read surface. Live evidence on pdpp.vivid.fish showed a Slack messages page spending about 1.3s sorting recordjson->>cursorfield, and count:"exact" spending about 1.1s scanning the full records table.

The owner console overview and runs pages block on reference spine list routes. Live measurement after the connector-summary fix showed /ref/traces at about 365-900ms and /ref/runs?limit=100 at about 700-950ms. The traces route was doing a full aggregate over roughly 579k spine events to show six recent rows.

The live reference instance exposes /v1/search, /v1/search/semantic, and /v1/search/hybrid over a broad owner-visible package. API benchmarks show search as the slowest read-surface path, and broad lexical queries can fail with Postgres 53100 shared-memory errors under Docker.

Grouped aggregate responses (groupby and groupby_time) return the top-N groups ordered by count. Without a rollup field, callers cannot tell whether the top-N is a complete picture of the data or a truncated subset. A model or agent that trusts a limited facet list without knowing the tail size can draw incorrect conclusions (e.g. "these are all the senders" when 40% of records fell into groups beyond the limit). A second aggregation call to compute the tail is extra latency and an unnecessary round trip.

The connection lifecycle primitives revokeconnection (stop future collection, preserve records) and deleteconnection (erase exactly one connection's source-of-truth records/state) are shipped and audited — but only over the owner-agent bearer REST control plane (POST /v1/owner/connections/:id/revoke, DELETE /v1/owner/connections/:id). The operator console exposes neither. The records-list no-data copy directs the operator to "ask your owner agent to revoke it … or delete it."

The Explore canvas needs a single durable server endpoint that can serve a fully-paginated, time-ordered feed spanning ALL of the owner's sources. The previous approach (per-stream fan-out capped at a fixed total, assembled in the browser layer) has three compounding defects:

Explore's over-time chart currently fans out from the console to many per-stream aggregateRecordsByTime calls. That makes first load slow and keeps the chart sparse when the selected corpus spans years because granularity is derived from a fixed client window instead of the populated record extent.

Google Maps Timeline is a high-value owner data source, but the existing Google Takeout connector exposes location history only as one stream inside a broad archive connector. Current Google guidance also makes Timeline export a device/app file flow, so a browser scraper would be the wrong first implementation.

Headless or sandboxed MCP clients can fail the normal loopback OAuth callback flow by opening a browser the user cannot operate and then waiting indefinitely. Prior-art review shows that the SLVP path for browserless setup is an explicit, bounded device-authorization flow, but PDPP's existing RFC 8628 endpoint currently issues owner-agent credentials and /mcp must reject owner bearers.

The MCP adapter is token-efficient for list and search responses, but large record bodies can still become a dead end for clients that cannot reliably inspect structuredContent or expand custom transcript markers. PDPP needs a standard, grant-scoped path from compact MCP results to the full authorized record or field without dumping large bodies into every tool response.

listSpineCorrelations aggregates spine_events by correlation id and supports source filters. Postgres already carries a source/run summary index for that hot path; SQLite lacked equivalent coverage.

Retained-size dataset summary reads can report stale derived metadata even when the existing bounded reconcile path can repair the read model from durable reference state. The owner dashboard also must not turn internal maintenance details into hero copy.

The Sources page (/dashboard/records) loads in roughly seven seconds on the live corpus because it blocks on GET /ref/records/version-stats?limit=8. That route still runs an unbounded recordchanges aggregate whenever the retained-size global projection is dirty. On the live Postgres corpus this means sorting and grouping more than three million record_changes rows on each page load.

The deployed read/evidence substrate improves MCP and CLI structure, but hosted ChatGPT retesting still found dead ends: search handles without matched text, field-window resource URIs that the client could not read, and full fetches that materialized files for small text records.

Google Maps Timeline, WhatsApp exports, Apple Health exports, Takeout archives, device media folders, and similar sources expose a stale gap in the Collection Profile: collection can arrive through multiple acquisition methods, often partially and out of order, while still populating the same logical streams.

The 2026-06-09 incident: four connections were migrated env→store and verified green via MANUAL runs, but the scheduler's launch path (runtime/scheduler.ts::launchRun → runConnector) never consulted the encrypted per-connection credential store — only controller.runNow resolved staticSecretEnv. When the reference container was recreated without the old secret exports, compose ${VAR:-} mappings left the credential env vars as EMPTY STRINGS, and every scheduled static-secret run raised credentialsrequired ("github needs: GITHUBP..."), auto-cancelled, and reported connectorreportedfailed while valid store rows sat unread.

Slack retained historical records for an engineering channel while current connector runs no longer enumerated that channel from the slackdump archive. The run still completed successfully with no known gaps, so downstream clients could mistake stale partial coverage for current absence.

The Sources view can mislead an owner during recovery: row-local state can survive source switches, run-start toasts do not link to the run, owner-runnable verdict actions can be hidden from the list, and a recent successful collection with known terminal coverage gaps can read as a total collection failure.

The Codex and Claude Code local collectors now avoid whole-file and whole-result-set memory spikes, but the same source-size-to-heap hazard remains in peer filesystem and local-DB connectors. Heavy local exports such as Twitter archives, iMessage chat.db, and Slack dump databases can be hundreds of megabytes or larger. A connector-wide contract is needed so bounded memory is not a one-off property of two agent connectors.

Google Maps now has both a file/import path and an API-backed Data Portability path, and agent hosts still vary in how much MCP tool output they expose. The reference needs a durable rule for reusing stream definitions across acquisition paths without erasing source identity, and MCP search needs to put a usable fetch handle where clipped previews still show it.

PDPP's root protocol spec and public site should state the protocol's implementation-neutral posture directly. Current prose implies that posture through URI-based identifiers and grant-pinned manifests, but reviewers should not have to infer that PDPP conformance is independent of any specific hosted service, chain, token, registry, or vendor deployment.

A live ChatGPT retest of the 5-tool MCP surface (2026-06-09) showed the search→fetch journey failing on multi-source hosted packages. Search hits carried id = stream:recordid plus a SEPARATE connectionid field, so a model had to carry TWO values between tools. fetch(id) without connectionid returned a typed 409 ambiguousconnection; ChatGPT's rendered envelope buried the second field and its model never completed a fetch (retrying with both fields was verified to work). OpenAI's search/fetch contract treats result ids as single opaque handles; ours leaked a join requirement into the model loop.

Owner-agent REST callers can discover connectionid values from /v1/streams, but the polyfill owner read path still requires connectorid on stream record reads. This makes the public REST contract weaker than MCP and weaker than the advertised connection_id query shape.

The consent surface advertises a three-class trust model — protocol-enforced facts, manifest-authored descriptions, and client-authored claims — and the operator-ui consent card already renders each class distinctly (every element carries a data-authorship provenance hook). But the reference Authorization Server's hosted consent renderer (reference-implementation/server/routes/as-consent-ui-helpers.ts) did not honor that boundary:

The owner console can show multiple configured connections for the same connector type. The current dashboard still has routes and cards that fall back from a concrete connection to connector-wide run evidence. That makes real sources look broken, duplicated, or actionable based on sibling/orphan connector runs.

Explore rendered record identity independently in the feed row, stream table, mobile card, and detail header. That allowed id-only records to appear as confident detail-page titles even when no manifest-authored display role existed.

Live dashboard metadata can contain owner-runnable advisory source actions, such as retrying an Amazon detail gap or refreshing Reddit, while the home Overview remains in a calm state because it only treats channel: "attention" as owner-relevant.

The Standing dashboard answers "who can read parts of me?" and "what has been read?" by summarizing ref/grants and ref/traces. Those summaries can carry only clientid, so live rows can render as raw cli... identifiers even when the reference server has registered client metadata. That is honest but below the owner-comprehension bar.

The vanished-run diagnosis (tmp/workstreams/vanished-run-diagnosis-2026-06-10.md, run run1781118340000) showed that a run which started, persisted both lifecycle events, and failed terminally 452 ms after launch still looked vanished to its observer. The contract break: the runid returned by the run-now 202 ack stops being resolvable the moment the run settles.