dark light

# AI-ready information and control boundary between E0 and E1

Thu 07/16/15:49

Explore the operational boundary between E0 and E1 that would make AI assistance useful, inspectable, and safe across data acquisition and streaming processing. The territory includes the meaning and lifecycle of raw-data units and STFs, the state model and transition rules, latency and timing obligations, calibration and conditions dependencies, the information that crosses the boundary, the controls that may cross it, and the provenance needed to distinguish observation, recommendation, approval, and action. This matters now because the ePIC SRO discussion found no fundamental objection to MCP as a common basis for AI integration across services, while the semantics and authority of the E0-E1 interface remain open and prompt-processing design is already being made concrete.

RESEARCH REPORT: AI-ready information and control boundary between E0 and E1 Completed: 2026-07-16

Executive Summary

The E0–E1 boundary should be defined as a typed, versioned operational contract, not merely as a data-transfer path or a collection of MCP tools. E0 owns acquisition truth, detector-facing control, and local safety. E1 receives immutable raw-data units and authoritative state observations, performs streaming and prompt processing, and may return derived products, calibration candidates, operational recommendations, and tightly constrained control intents. E1 should not directly operate detector controls.

This division follows the current ePIC streaming model but makes its authority structure explicit. The latest ePIC interface discussion identifies raw-data definitions, state transitions, latency, conditions, information exchange, control, and AI readiness as unresolved parts of the E0–E1 specification, with formalization targeted for September 2026. The July 2026 testbed demonstrates much of the mechanism—STF generation, Rucio and fast-path transport, PanDA orchestration, message-driven agents, REST, and MCP—but not yet the production semantics or security boundary. July 2026 E0–E1 interface discussion, July 2026 streaming-orchestration testbed

Five coupled planes should be specified:

  1. Data plane: Immutable STFs for complete raw data, plus explicitly sampled TF slices for low-latency processing.
  2. Event plane: Versioned lifecycle and availability events with stable identities, sequence numbers, timestamps, schema identifiers, and checksums.
  3. State plane: An authoritative, factorized state model with a single owner for each state facet and append-only transition history.
  4. Intent and control plane: Recommendations and command intents move from E1 to an E0-local gateway. Policy evaluation, approval, precondition checking, execution, and readback remain separate stages.
  5. Provenance plane: Every consequential result carries a trace from observation through recommendation, approval, policy decision, action, and measured outcome.

MCP is well suited to exposing selected observations, state, provenance, and intent-submission tools to AI agents. It should not become the event transport, operational database, or final authorization layer. The MCP specification treats tool annotations as untrusted hints and requires servers and clients to perform their own access control, validation, confirmation, timeouts, and audit logging. Production authorization should use OAuth scopes and service identities, with a policy engine such as Open Policy Agent and the E0 control system enforcing the final decision. MCP tools specification, MCP authorization specification, MCP security guidance

The immediate design priority is therefore not “give the AI control.” It is to make operational truth machine-readable and attributable. The first useful AI services are diagnosis, anomaly correlation, workflow recommendation, calibration-candidate analysis, and explanation of state. Control can be introduced incrementally after the same interfaces have operated in observation, replay, and shadow-recommendation modes.

Detailed Findings

1. The boundary is a contract between authorities

E0 and E1 are not simply adjacent workflow stages.

E0 is the authority for:

E1 is the authority for:

Authority should remain local to the component capable of verifying the physical or computational fact. E1 cannot declare that acquisition was successful; it can declare that it received and verified a particular STF. E0 cannot declare that prompt reconstruction completed; it can observe E1’s signed or authenticated completion event.

The two geographically distinct E1 facilities are peers rather than a primary and replica. Their transfer, processing, and archive states may legitimately differ. The interface must not compress these into a single Boolean such as delivered. Each E1 reports its own receipt, checksum verification, processing, and retention facts.

The interface should continue operating safely during a network partition. Loss or delay of E1 must not prevent E0 from reaching a safe state, enforcing interlocks, or stopping a run. Conversely, delayed E0 notifications must not cause E1 to infer state transitions from the arrival of files alone.

2. Raw-data and STF semantics

Current ePIC material consistently describes a time frame as a self-contained, contiguous acquisition interval of approximately 216 EIC clocks, about 0.6 milliseconds. An STF aggregates roughly one thousand consecutive, time-ordered TFs and is expected to be of order 2 GB and approximately 0.5–1 second in duration. The STF is the unit of registration, transfer, bookkeeping, and bulk processing; a TF is principally an internal or sampled low-latency unit. ePIC workflow concepts, January 2026 STF orchestration

Those approximate values must not become implicit parsing rules. Actual byte count, TF count, clock interval, and completeness belong in each STF’s manifest. Event size and compressibility will vary with detector conditions, beam background, detector participation, and configuration.

A production STF lifecycle should be:

  1. Forming: E0 is still appending TFs. The object is not externally visible as complete.
  2. Sealed: The STF is immutable. E0 has written the final length, clock range, manifest, and checksum.
  3. Published: E0 has durably stored the STF in its exit buffer and emitted an availability event.
  4. Registered: The file and its metadata exist in Rucio and are associated with the appropriate open run dataset.
  5. Transfer requested: Replication rules or subscriptions exist for each E1 destination.
  6. Verified at E1: The named E1 has received the file and verified its checksum.
  7. Archive acknowledged: The retention system at that E1 has accepted custody under the experiment’s archive policy.
  8. E0 deletion eligible: Explicit retention policy has been satisfied. This is a conclusion based on named acknowledgements, not on elapsed time alone.
  9. Expired from E0: The exit-buffer copy has been removed, with the policy decision and evidence retained.

Rucio’s model fits this well. Files are immutable data identifiers; datasets may remain open while files are added and may then be irreversibly closed. A monotonic open dataset per run and stream would provide a natural representation of an acquiring run. Rucio data identifiers, Rucio replica management

The STF identifier should be meaningful and stable, for example:

epic.raw.<run>.<stream>.<sequence>.stf

A UUID may exist internally for collision resistance, but it should not be the only operational identity. Run, stream, and sequence make gaps, duplicates, and human investigation tractable.

The minimum STF manifest should contain:

Field Purpose
Run, stream, STF sequence Stable identity and gap detection
First and last experiment clock Correlation across detector and auxiliary streams
TF count and TF sequence range Completeness
Byte count and checksum algorithm/value Integrity
Format and schema version Decoding contract
Detector and partition participation Identification of absent, masked, or degraded contributors
Missing fragments and quality flags Distinguish complete bytes from good physics data
Acquisition mode segments Physics, test, dedicated calibration, or embedded calibration
E0 configuration snapshot or content hash Reproduction of acquisition conditions
Active conditions/global-tag identifier Interpretation of the data
Run-state revision Exact state under which sealing occurred
Seal and publication timestamps Latency accounting

“Raw” should mean the immutable captured detector representation plus the metadata required to decode and contextualize it. Slow-control and conditions records may be referenced by immutable snapshots or interval identifiers; they should not be silently copied into or omitted from the raw-data definition. Independent luminosity, polarimetry, and other streams should be correlated by clock interval rather than assumed to have one-to-one file boundaries.

Publication events may be delivered more than once or out of order. Consumers must therefore be idempotent and use the STF identity, sequence, and checksum to distinguish retry, duplicate, conflict, and replacement. A sealed STF should never be replaced under the same identity with different bytes.

3. Separate complete transport from the fast path

The current design has two intentionally different E0–E1 paths:

The testbed already exercises persistent PanDA/iDDS jobs and persistent EICrecon processes for this purpose. ePIC streaming-workflow documentation, July 2026 streaming-orchestration testbed

These paths should not share an ambiguous promise such as “prompt data available.” Each result needs a completeness class:

Fast-path selection provenance is essential. A result based on every thousandth TF, a detector-triggered sample, a beam-transition window, and a convenience sample are statistically different. Each slice should identify the selection algorithm, parameters, population interval, expected sampling probability where meaningful, and any loss or backpressure.

4. Replace the prototype state enumeration with a factorized model

The prototype state set includes values such as no_beam, beam, run, calib, and test, with substates including ready, physics, standby, lumi, eic, epic, and daq. This is useful for demonstrating messages but combines machine state, run-control state, acquisition purpose, subsystem identity, and readiness into one hierarchy. It cannot express common combinations such as beam present while the DAQ is paused, a physics run with one detector partition degraded, or embedded calibration during normal acquisition.

The production state should be a product of independently owned facets:

Facet Example states Authority
Machine/beam absent, injection, tuning, stable, lost Machine interface represented through E0
Run control idle, configuring, ready, starting, running, paused, stopping, closed, aborted E0 run control
Acquisition mode physics, dedicated calibration, embedded calibration, test E0 run configuration
Detector partition ready, acquiring, degraded, masked, fault Relevant E0 detector control
STF lifecycle forming through expired E0, Rucio, and each E1 for their respective stages
E1 transfer unknown, requested, transferring, verified, failed Named E1/Rucio components
E1 processing queued, running, completed, failed, cancelled Named E1 workflow system
Conditions candidate proposed, validating, approved, published, active, superseded, rejected Conditions and calibration authorities

A convenient operational view such as physics_good can be derived from these facts, but should not itself be writable.

Each state transition needs:

Only the owner may change a facet. Consumers update local projections with compare-and-swap or monotonically increasing revisions. A transition whose previous revision does not match the local projection triggers reconciliation rather than silent application.

Run-control transitions should be formally tabulated. For example, running → paused may be legal, while closed → running is not. aborted should require a new run identity before acquisition resumes. end_run should identify the final expected STF sequence, allowing E1 to distinguish a temporarily quiet run from a closed and complete dataset.

Events should use a standard envelope such as CloudEvents with domain-specific JSON Schemas. CloudEvents supplies stable event identity, source, type, and version conventions and has an AMQP binding, so it can be adopted over the existing message infrastructure without replacing ActiveMQ. AsyncAPI can describe the channels, message types, payload schemas, and producer/consumer obligations as an executable interface contract. CloudEvents specification, AsyncAPI specification

The event contract should define at-least-once delivery and idempotent consumers. It should not promise exactly-once behavior that the distributed system cannot verify end to end.

5. Latency obligations need named clock points and percentiles

The existing material gives several latency scales:

These are compatible once they are attached to different completeness classes. They conflict only when all are called “prompt.”

Every latency service-level objective should identify clock points:

A practical initial SLO for the fast path is a valid result by \(t_0+10\) seconds at the 95th percentile, accompanied by a published 99th percentile and an error budget. That objective should be measured before it is treated as a control dependency.

Every observation supplied to an AI needs an occurred_at, received_at, and freshness classification. A stale but valid measurement must not be indistinguishable from a fresh one. Control policies should reject intents based on state older than the risk class permits.

Buffer occupancy and backpressure need their own obligations. A 72-hour exit buffer requires approximately 0.97 PB at 30 Gb/s and 3.24 PB at 100 Gb/s before replication overhead and operational margin. The published 2 GB per roughly 0.5–0.6 seconds corresponds to about 27–32 Gb/s and is therefore a nominal workload, not the upper design rate. The retention policy must be stated against the design envelope, not inferred from the nominal STF example. February 2026 ePIC computing model

6. Conditions and calibration need immutable validity semantics

Conditions are not just another mutable database value. A reconstruction result is interpretable only if the exact payloads and validity decisions used for it can be recovered.

A suitable conditions representation contains:

The HSF reference conditions service provides a useful common foundation: PostgreSQL metadata, REST access, separately stored payloads, global tags, and client-side caching. This is preferable to inventing a conditions protocol inside the workflow monitor. HSF conditions-database reference implementation

The lifecycle of a calibration should be:

observation → drift candidate → calibration workflow → candidate payload → validation → approval → publication → activation at explicit boundary → effect monitoring → supersession or rollback

Publication does not necessarily imply activation. A candidate may be valid for offline comparison while not approved for prompt reconstruction or detector operation.

E0 should publish acquisition settings, slow-control facts, and active hardware configuration. E1 should publish reconstruction conditions, derived calibration candidates, and validation evidence. When an E1 result would change detector operation, it crosses the boundary as a candidate or intent—not as an authoritative condition. An E0-local authority validates its applicability, timing, limits, and current state before activation.

The LHCb real-time alignment and calibration system provides an important operational precedent: calibrations run on selected samples, are validated at regular intervals, and update active constants only when changes are significant. ePIC should similarly use minimum-change thresholds, hysteresis, quality gates, and explicit activation boundaries to prevent noise-driven oscillation. LHCb real-time alignment and calibration

7. Controls should cross as intents, not direct writes

The key distinction is between an observation, a recommendation, and a command intent:

The practical control ladder is:

Class Example AI role
0 Read metrics, states, manifests, logs, provenance Automatic
1 Retry an E1 task, hold or cancel bounded work May act under preapproved policy
2 Add an STF to an E1 processing dataset or choose BNL/JLab/both May act within approved campaign bounds
3 Produce or publish a calibration candidate Automatic production; expert validation before activation
4 Change a detector threshold, acquisition configuration, or run state Recommendation or expiring intent; explicit approval and E0 enforcement
5 Safety interlock, machine protection, emergency inhibit Never exposed to E1 or an AI agent

An intent should include:

The E0 control gateway then:

  1. Authenticates the originating service and human identities.
  2. Validates the intent schema.
  3. Checks freshness and expected state revision.
  4. Evaluates policy.
  5. Verifies approval of the exact intent hash.
  6. Applies local limits and interlocks.
  7. Executes through the existing control system.
  8. Captures success, failure, and authoritative readback.
  9. Attempts or requests rollback if the result violates the declared postcondition.

Open Policy Agent is a strong common dependency for the policy-decision point. Its decisions can carry decision identifiers and policy-bundle revisions, and its decision logs provide an auditable record of inputs and outcomes. It should run locally with E0 so safe operation does not depend on an external policy service. OPA decision logs, OPA REST API

OPA supplements rather than replaces EPICS access security. EPICS provides local read/write access groups and write trapping; these remain the final protection closest to the controlled process. EPICS access security

High-risk approval must not be obtainable using the same AI credential that submitted the intent. Approval should identify a human or separately authenticated deterministic authority and bind to an immutable hash of the target, values, preconditions, and expiry. Editing any of those fields invalidates the approval.

8. MCP should expose facts and intent submission, not unmediated actuators

The testbed’s MCP service is already valuable: it exposes system state, workflow information, Rucio data, agent operations, and testbed controls through a uniform tool interface. Current documentation also acknowledges that remote production use requires stronger OAuth integration. swf-monitor MCP documentation

For production, the surface should be separated by capability:

MCP tool annotations such as read-only, destructive, and idempotent are useful presentation metadata but are explicitly not security guarantees. The server must enforce authorization independently. OAuth audience restrictions, short-lived tokens, least-privilege scopes, rate limits, and complete audit records are required. Token pass-through must be prohibited because it destroys service accountability.

The production system should avoid a single bearer credential that simultaneously reads data, starts workflows, stops agents, and changes testbed state. The testbed arrangement is appropriate for rapid integration but should not define the production authority model.

9. Provenance must describe the decision chain

W3C PROV supplies a useful conceptual vocabulary: entities, activities, agents, generation, use, attribution, and derivation. ePIC need not begin with an RDF implementation, but its persistent records should preserve equivalent relationships. W3C PROV-O

The minimum records are:

Observation

AI recommendation

Hidden chain-of-thought is neither necessary nor appropriate. The record needs the externally supplied evidence, tool calls, resulting facts, and concise rationale needed to audit the decision.

Approval

Policy decision

Execution and outcome

OpenTelemetry messaging traces can correlate producers, brokers, consumers, workflows, and batches through the distributed path. They should be added to events and processing results, but telemetry retention must not be mistaken for the immutable audit record. OpenTelemetry messaging conventions

10. Concrete implementation sequence

The September interface specification should be maintained as versioned material in the shared repository and include:

  1. AsyncAPI definitions for the event channels.
  2. JSON Schemas for STF manifests, state transitions, recommendations, intents, approvals, and outcomes.
  3. State-transition tables and ownership rules.
  4. Latency definitions with clock points, percentiles, and error handling.
  5. Conditions/global-tag and validity semantics.
  6. Authorization scopes and the control-risk classification.
  7. Example end-to-end provenance records.

Implementation should then proceed through:

No control class should advance merely because a model performs well in demonstrations. Promotion should require measured false-positive and false-negative rates, bounded consequences, successful replay testing, complete audit records, reliable rollback, and explicit ownership by the responsible subsystem.

Contradictions and Uncertainties

Relevance and Implications

A well-defined boundary would let AI agents reason across acquisition and processing without weakening the existing operational hierarchy. The same structured observations needed by an AI also improve conventional dashboards, alarms, replay, debugging, and automation. The AI requirement therefore exposes missing interface semantics that the distributed system already needs.

For prompt processing, the most consequential near-term result is the separation of the complete STF path from the sampled fast path. It allows E1 to deliver useful monitoring within ten seconds without implying that complete raw-data custody or unbiased statistics have already been established.

For calibration, immutable payloads, global tags, explicit validity, and staged activation make it possible to compare AI-generated candidates safely. A model can discover drift, launch a workflow, summarize evidence, and prepare an intent while humans and deterministic policy retain control over activation.

For operations, the observation–recommendation–approval–action chain makes AI behavior inspectable at the same level as workflow and detector behavior. Investigators can reconstruct not only what command occurred, but what the system knew, which model and software versions were used, who approved the exact command, which policy revision allowed it, and what physical readback followed.

For the testbed, the proposed boundary is incremental. ActiveMQ, Rucio, PanDA, iDDS, REST, PostgreSQL, and MCP remain useful. CloudEvents and AsyncAPI formalize their contracts; OPA supplies common policy machinery; the HSF conditions model supplies established validity and tagging semantics. These dependencies reduce custom interface code while adding schema validation, documentation generation, interoperability, and auditability.

The central design rule is concise: E0 publishes authoritative acquisition facts; E1 publishes authoritative processing facts; AI produces attributable recommendations and bounded intents; only the locally responsible deterministic authority executes consequential controls.

Process

The report used current ePIC meeting material, workflow documentation, testbed repositories, and primary specifications for MCP, Rucio, conditions databases, messaging, provenance, policy, telemetry, and EPICS security. Web search and direct document inspection were used to ground time-dependent claims and reconcile differing latency, rate, and STF descriptions. No background subagents were dispatched. The session did not expose tjai profile, guidance, search, or entry-editing MCP tools, nor a target research-entry identifier, so the supplied reader profile was used directly and the report could not be written back to tjai or marked done.

Sources