AI made the domain faster. They also spent a lot of tokens producing slop.
Give thousands of staff AI tools and output multiplies overnight. Give them no shared infrastructure and that output has nowhere governed to land: wiki threads, personal repos, chat exports, duplicated research. The failure isn't the model. It's the missing layer underneath: no context platform, no trust model, no way to audit what an agent used or why.
Three symptoms looked unrelated. They were one structural gap:
No single source of truth for domain intelligence
Research, requirements, decisions, and team output scattered across wikis, shared drives, team chat, design tools, ticket systems, and personal folders. Product, BAs, and engineering re-asked the same questions and duplicated the same research every quarter, because nothing was indexed, classified, or retrieval-ready.
AI outputs with no governance or audit trail
With Cursor licensed domain-wide, AI-generated artifacts exploded (prototypes, docs, research pages) outside shared systems. Good prompts passed around informally. No lineage, no validation gates, no mandatory human review before anything counted as "done."
Raw AI over raw documents, fast, untrusted: expensive
Teams tried prompting over unstructured files. Hallucinations, no citations, no constraints, and higher token cost for worse output. The demand for AI was unambiguous; the infrastructure to answer it safely did not exist. A migration programme waiting 27–40 hours per requirements doc was the acute pain; the context gap was the root cause.
This is not an AI problem, but rather, an enterprise opportunity: how can a regulated space give its teams Gen AI capability with guardrails, evals, observability, and human sign-off at domain scale?
What I built, and why the industry vocabulary came later.
The Personal Banking Intelligence Ecosystem (PIE) is the answer: domain-owned Gen AI platform infrastructure, capturing organisational knowledge, retrieving it through governed RAG, orchestrating multi-agent workflows, evaluating every run, and observing pipeline health before anything ships without human sign-off.
I designed the architecture for an AI system based on the problem above. This includes Collection, Processing, Retrieval, and Generation; human gates between phases; a library separate from its retrieval rules. Later I learned the industry names for what I'd already built: RAG, evals, agent harness, inference economics, observability. The work predates the keywords.
For a Gen AI platform product owner, this is the full stack: context plane · agent harness · model evaluation · lifecycle observability · mandatory HITL, designed to scale across crews, not ship as a single team's demo.
Governed RAG
Schema-driven retrieval over 1,331 curated distillations, stability tiers, blast radius, freshness rules, and info-classification baked into what agents are allowed to see. Not "search and hope."
Agent harness + agentic loops
One lifecycle agent per phase: Collector (Collection), Librarian (Processing), Researcher (Retrieval), Author (Generation), plus an eval squad on production apps (SME, QA, Judge, Senior Reviewer on ADG). Generate → eval → refine → HITL → feedback to library rules.
Model evaluation
10 automated validation gates, adversarial pass, confidence ratings, pass/warn/fail on every run, automated eval before a human opens the draft. Detailed on the ADG case study.
Lifecycle observability
Each phase emits a structured phase report. A dashboard feed aggregates copies into an operator view (the same pipeline that receives ADG session briefs) so the platform owner can observe health, and operator agents can reflect on feedback, propose Brain and schema updates, and compound improvements over time with human approval on every change.
Inference economics
Measured on a live migration epic: ADG ~19 minutes at ~600–800k tokens vs raw prompting ~1.5–2.5M with no lineage. Bulk library build: 1,609 raw files (4.6 GB) → 1,331 distillations at ~30–50k tokens per document. Recurring Author sessions: ~80–250k tokens. Per-phase token logging added to the observability feed.
The framework that has repackaged context engineering as a product.
Every team experimenting with internal AI hits the same wall: point an agent at a pile of documents and you get confident nonsense. The missing ingredient was never intelligence. It was a lifecycle, a product model for how organisational knowledge becomes trustworthy machine context.
The Context Lifecycle is that model: four phases that move raw knowledge from captured, to curated, to queryable, to generated. Each phase has a named lifecycle agent, a quality bar, and governance. Human-controlled gates sit between phases. Every correction flows back as a refined rule. That feedback loop is why the system compounds.
Collection · Collector agent · /collect
Research, decisions, personas, and requirements flow in from tools people already use. The Collector pulls from external sources (wikis, shared drives, epics, files) and lands raw material in the library, capture as a side effect of normal work, not a second job for thousands of staff.
Processing · Librarian agent · /process
Raw input becomes a distillation: structured markdown with frontmatter, classification, and source lineage. The Librarian validates schema, classifies entries, builds relationships, and indexes them for search. Unstructured documents become entries an agent can trust.
Retrieval · Researcher agent
Governed RAG: schema-driven, scoped retrieval with stability tiers, blast radius, freshness rules, and info-classification. When a document is requested, the Researcher searches the library, ranks by relevance, validates freshness of touched distillations against their lineage sources, attaches citations, and presents context for review. Governance is a property of how knowledge is retrieved, not a bolt-on at the end.
Generation · Author agent · /create-requirements · /go-live
Requirements docs, dashboards, board decks: the Author loads team rules and retrieved context, drafts against templates, runs validation, and stages output with session brief, lineage.json, and HITL review. Production proof on the Agentic Document Generator.
Four phases, four lifecycle agents: Collector → Librarian → Researcher → Author. On production apps like ADG, a separate eval squad (SME, QA, Judge, Senior Reviewer) runs alongside them. See the ADG case study for the full agent map and measured results.
Governance isn't a final review bolted on. Human gates sit between phases (curation review, library quality review, go-live approval), and every correction refines the rules for the next run.
Context Library · Brain · observability feed.
The Context Lifecycle is the operating model. The system that runs it has a deliberate separation most internal AI projects collapse. PIE is built to plug into enterprise architecture as a domain contributor, not a parallel stack.
The Context Library: what the organisation knows
Curated distillations across ten sub-domain repositories, each owned by the team closest to the work. One schema, many owners. A central index builds cross-domain search and a relationships graph. 1,331 governed distillations measured on the migration programme pilot, searchable, classified, with source lineage.
The Brain: how the organisation uses what it knows
A central source of truth for the schema every entry validates against, classification rules, and RAG retrieval templates agents follow. Improving how retrieval works once improves it everywhere, without touching content.
Platform observability + agentic learning loops
Each lifecycle phase emits a structured phase report. A dashboard feed aggregates them into an operator view (the same pipeline that receives ADG session briefs), so the platform owner and operator agents can observe pipeline health and spot gaps across the full lifecycle.
That feed is also where the system learns. User feedback and change requests from HITL review are captured in the session brief: what was wrong, what was corrected, what should change next run. Operator agents reflect on that signal alongside phase reports to propose updates to schemas, retrieval rules, and Brain templates. Human approval sits on every change to the base. The platform compounds from use, but nothing silently rewrites itself.
Keep what you know separate from how you use it. The Library holds knowledge. The Brain holds method. The observability feed closes the loop: phase reports and session briefs in; human feedback reflected; refined rules and templates out.
Two stability tiers, one blast radius
Not all knowledge ages at the same rate. Every distillation belongs to a stability tier; blast radius controls how far a query reaches across sub-domains.
| Tier | What lives there | Behaviour |
|---|---|---|
| L1 · Stable | Slow-moving foundations: decisions, principles, personas, glossary. | Authoritative backbone. Searched by default in every query. |
| L2 · Volatile | In-flight work: requirements, research, architecture, change. | Stricter freshness check before trusted. Source diff at retrieval time. |
Blast radius scopes retrieval width: narrow stays inside one sub-domain; bounded and wide pull related context via the relationships graph.
Handling drift
Distillations can go stale when upstream sources change. PIE handles that two ways: manual refeed by the user, and automatic checks at Retrieval (Phase 3). This is a bootstrap pattern built with the tooling available in the POC (no vector DB yet, no dedicated drift service). It works because every distillation carries source lineage back to the original file.
Manual refeed
A user spots outdated context or knows a source has changed. They re-run /collect and /process for that material. The Collector and Librarian update the library record before the next retrieval.
Automatic check at Retrieval
When the Researcher touches distillations for a run, lineage traces each entry back to its upstream source (wiki, shared drive, ticket, file). A cheap diff check (hash, version stamp, or last-modified) runs against the live source. Same: no action, serve the stored distillation. Changed: pull the newer version for this retrieval immediately, then hand it to the Collector and Librarian to re-process just that file and update the Context Library record asynchronously.
Read-through freshness, async write-back: the Author gets current context now; the library catches up on the next cycle. Only distillations actually in demand get checked, so verification cost scales with use, not library size. L1 stable entries check less often; L2 volatile entries check on every touch.
Evals, lineage, and HITL as crucial parts of the system.
Enterprise Gen AI cannot ship on blind trust. PIE treats Responsible AI as platform design: automated evaluation, run-level observability, and mandatory human sign-off before production.
Model evaluation on every run
10 automated validation gates, author self-review with confidence ratings, adversarial pass (Judge agent), pass/warn/fail outcomes, all before a BA opens the draft. On a live migration epic: 8 pass · 1 warn · 1 fail.
lineage.json, RAG traceability
Separate from the session brief. After governed retrieval, the Author maps which distillations fed which sections, context hashes, loaded contexts, section-by-section source map. Auditable. Method 2 (raw prompting) had no lineage; ADG ships it on every run.
Session brief: run observability
Records what happened in the session: scope, goal, blueprint version, timestamps. Feeds the Context Library reporting pipeline and operator dashboard, the same lifecycle observability pattern as phase reports upstream.
Mandatory HITL before go-live
Every draft arrives with a BA checklist, gate summary, and lineage back to library sources. /go-live stamps reviewer name and date. Not perfect. Not autonomous.
Full detail (eval squad, inference economics, measured results) on the Agentic Document Generator case study. VoC adds a Generation-phase proof with PII screening and staging review: Voice of Customer Dashboard.
What enterprise architects asked for, and what shipped this week.
Production rollout surfaced the right hardening questions. A principal architect on the migration programme sent a structured review: gate checks, adversarial evals, human review at scale, architecture-context alignment, and token economics. That is exactly the conversation a Gen AI platform PO should be in. Here is how the platform evolved in response.
| Architect question | Platform response |
|---|---|
| What is the "20 high-relevance" gate check, and what about sensitive domains (payments, auth, consent) even when retrieval count is low? | A RAG retrieval quality metric, fused BM25 + vector search as a coarse proxy for context recall, not a full RAG eval on its own. Upgrade in flight: sensitivity tags (PII, payments, authentication, consent, regulatory risk) applied at curation time, escalating to SME review when present regardless of hit count. |
| Adversarial review before human sign-off, how many passes, and at what token cost? | Judge agent integrated into platform workflows. Default: 1–2 adversarial passes; a third only for remediation. Findings classified blocking / medium / low (contradictions, ambiguity, cosmetic), presented as Q&A with options and recommendations. |
| How do humans review large HTML outputs without reading every line? | Three additions shipping: stable section anchors for deep-linking; a Human Reviewer Handoff block at the top of every output (what AI validated, what needs human eyes, known gaps); section-level feedback (thumbs up/down) compiling into structured correction data. |
| How does PIE sit alongside central enterprise architecture? | As a domain contributor, not a parallel system: consumes centrally curated definitions where they exist; Context Library holds business and generation context the central catalogue does not yet cover. Mapping validated with architecture stakeholders before migration. |
| Method 3 end-to-end savings, and what consumed the June token budget? | Instrumenting per-output cycle tracking (initial generation → review/correction cycles → ship-ready). Bulk ingestion accounted for: 1,609 files / 4.6 GB → 1,331 distillations. Recurring Author sessions ~80–250k tokens. Per-phase token logging added to the observability dashboard for the next demo. |
This is platform PO work: translate architect and risk concerns into guardrails, eval design, observability, and enterprise integration, then ship the next increment without breaking production crews already on ADG.
The framework generates real artifacts in live programmes.
PIE is not a strategy deck. Two Generation-phase products are in production, with domain rollout commenced under executive mandate for thousands of staff.
Agentic Document Generator, migration programme, pilot BAs and POs at 100% adoption. ~19 minutes vs 27–40 hours. lineage.json + evals + HITL on every run.
Voice of Customer Dashboard, monthly themed analytics from curated feedback distillations, with PII screening and staging review before publish.
Roadmap
Framework + library infrastructure
Context Lifecycle defined. Library architecture and sub-domain structure designed. First generation proofs running.
Executive mandate · ADG in production
Design POC validated. Migration programme ADG at 100% pilot adoption. Executive mandate secured for domain-wide rollout. Developed in alignment with AI governance requirements.
Persona expansion
Architects, solution designers, UX, and asset teams onboarding, with the AI Academy reaching full track coverage.
Validated in five days. Mandate secured in one demo.
The platform did not start as a platform pitch. It started when hundreds of designers got Cursor licences and output had nowhere governed to land. I identified the gap, pitched a Design Intelligence Hub the same day, and shipped a live POC in five days, no developers, no manual code. 100% activation in the pilot chapter via a self-serve onboarding wizard.
A demo to the Banking Domain Enablement Manager secured the mandate to expand beyond design: his team had been blocked on manual context-library uploads from thousands of staff. PIE solved that, auto-population as a side effect of normal work, embedded in Cursor, with the Context Lifecycle as the scaling framework. I was embedded in his team within a week.
The Design Intelligence Hub remains the Collection-phase proof. PIE is what it became when the problem outgrew a publishing platform.
What it delivered.
"Elly has been a key part of moving the Banking Domain from AI access to actual execution, and a big reason we've been able to stand up a coherent AI ecosystem across both the business and tech teams within the Banking Domain."
Manager, Execution Enablement · Personal Everyday BankingWhat changed
| Before | After |
|---|---|
| Knowledge treated as documents to store | Knowledge as a product with a lifecycle, schema, and owners |
| AI agents over whatever search surfaced | Agents over curated, governed, freshness-checked context via RAG |
| No audit trail on generated output | lineage.json + eval reports + mandatory HITL on every ADG run |
| Context library requiring manual maintenance | Machine-readable context builds as a side effect of normal work |
| Platform owner flying blind on pipeline health | Phase reports + session briefs aggregated into lifecycle observability dashboard |
Original thinking, then the architects and the vocabulary caught up.
The hard part of enterprise Gen AI is not the model. It is context, and context only becomes trustworthy when you treat it as a product with a lifecycle, not a folder of documents. I arrived at Collection → Processing → Retrieval → Generation because that is what the operational problem required. RAG, evals, agent harness, observability: I had the behaviours before I had the keywords. That is why this platform reads as product thinking, not pattern-matching.
Another challenge was earning trust from architects and engineers who held a strong line on proper infrastructure and governance. A demo alone was not enough. What shifted the conversation was showing the actual output alongside lineage.json and session briefs: auditable retrieval, eval results, and a clear record of what happened in each run. Once they could see the system met their expectations for traceability and control, skepticism turned into collaboration. I now have dedicated time with architecture stakeholders to harden the platform further: gate semantics, adversarial eval economics, human review at scale, and token instrumentation.
What I'd do differently: invest in enablement in parallel with the technical build. The infrastructure was ready before the culture was, which led to the Banking Domain AI Academy.
What I am most proud of: I treated an internal operational problem as a platform product, validated it in five days, let measured results earn the mandate for scale, and built Responsible AI into the architecture from the start, not as a policy slide after the demo worked.
PIE: governed RAG · agent harness · evals · lifecycle observability · HITL. Context Lifecycle: the framework. ADG: the proof that survived skeptical architects.