AlphaAvatar v0.6.6: event-driven multimodal memory, unified runtimes, and cleaner agent contracts

[AlphaAvatar]

Hi everyone

A few releases ago, I shared the architecture changes in AlphaAvatar v0.6.4, where the project started moving toward a shared perception runtime rather than letting Memory, Persona, Vision, and other components independently rebuild their own realtime input pipelines.

Since then, v0.6.5 extended that architecture into audio perception and introduced a more isolated inference/runtime model.

With AlphaAvatar v0.6.6, the focus is a little different.

This release is less about introducing one large new feature and more about making the runtime converge around clearer and more consistent contracts.

The main changes include:

• audio-aware Environment Memory

• event-driven and adaptive memory updates

• a unified runner model for Qdrant and LanceDB

• plugin capability descriptions exposed to the Avatar

• longer and more configurable perception history

• more explicit Stream and Observation schemas

• cleaner separation from LiveKit-specific runtime behavior

• better session data organization

The broader goal is to keep perception, memory, identity, tools, and other capabilities modular and observable, while letting the model remain the reasoning and decision layer connecting them.

Audio is now part of Environment Memory

The first version of ENV Memory introduced in v0.6.4 was mainly based on sampled visual observations.

The perception architecture has since expanded to audio as well.

In v0.6.6, audio_segment can now become an input to ENV Memory alongside visual observations.

Conceptually, the perception path is becoming:

Realtime input
      ↓
PerceptionRuntime
      ↓
Shared observation streams
  ├── video_frame
  ├── audio_segment
  └── derived annotations
      ↓
Persona / Vision / Memory / Interaction Router

An audio segment is not simply another transcription message.

It is a derived perception observation with its own timing and runtime context, which means downstream components can consume it independently.

For example:

• transcription can turn speech into text;

• speaker recognition can determine who is speaking;

• Persona can update identity-related information;

• ENV Memory can use speech together with visual context;

• future interaction policies can reason about speaker activity and environmental events.

This follows the same principle as shared visual perception:

Publish perception once, then allow independent consumers to interpret it according to their own purpose.

ENV Memory updates are now event-driven

Another change in v0.6.6 is how Environment Memory decides when to process observations.

Previously, ENV Memory relied more heavily on periodic polling.

That works, but it creates an awkward relationship between a continuous perception stream and a timer:

every N seconds
    ↓
check whether anything changed
    ↓
build observation window
    ↓
extract memory

v0.6.6 moves this toward an event-driven and adaptive update model.

new relevant observation
        ↓
update signal
        ↓
adaptive accumulation/windowing
        ↓
memory extraction
        ↓
structured memory

This is a relatively small architectural change, but I think it matters for long-running realtime assistants.

The Memory component no longer has to continuously wake up just to discover that nothing useful happened.

At the same time, it does not need to run an expensive extraction for every individual frame or short audio fragment.

Instead, perception activity drives memory processing, while the runtime can accumulate observations into useful temporal windows before committing work.

This should also make it easier to introduce more sophisticated policies later, such as:

• activity-aware batching;

• modality-aware update thresholds;

• silence and inactivity handling;

• event consolidation;

• backpressure;

• different policies for foreground and background memory.

Qdrant and LanceDB now use the same Memory runner contract

AlphaAvatar supports both Qdrant and LanceDB as Memory vector backends.

Historically, the two implementations gradually accumulated differences in how their execution paths were structured.

That is something I wanted to remove before adding more Memory behavior.

In v0.6.6, both backends now follow the same runner protocol:

Memory
   ↓
common runner contract
   ├── Qdrant runner
   └── LanceDB runner

The intention is that choosing a local or remote vector backend should be primarily a storage/deployment decision, rather than changing how the surrounding Memory runtime behaves.

This also makes backend development easier to reason about.

A new Memory operation should have one runtime contract, with backend-specific behavior implemented underneath it.

For AlphaAvatar this is part of a broader pattern: interchangeable components should differ in implementation, not silently redefine the semantics of the runtime around them.

Plugins can now describe their capabilities to the Avatar

AlphaAvatar has been increasingly componentized:

• Memory

• Persona

• Character

• Status

• Interaction Router

• RAG

• DeepResearch

• MCP

• and other optional plugins

But there is a subtle problem with a modular runtime:

How does the model know what the current runtime is actually capable of?

Hard-coding every possible feature into the system prompt does not scale very well.

It also becomes inaccurate when different deployments enable different plugin combinations.

v0.6.6 introduces capability descriptions for internal plugins.

A plugin can expose a concise description of what it provides, and the active capability information can be included in the Avatar context.

Conceptually:

Installed runtime plugins
        ↓
capability descriptions
        ↓
Avatar context
        ↓
LLM

This is intentionally different from putting the implementation itself inside the prompt.

The model does not need to understand every internal class or runtime detail.

It needs enough information to understand things such as:

• whether persistent Memory exists;

• whether Persona information is available;

• whether environmental observations are being tracked;

• whether a research or retrieval component is available;

• what kind of assistance the current runtime can provide.

This should also make AlphaAvatar configurations more composable.

Different deployments can expose different capabilities without maintaining completely separate Avatar prompts.

Longer and more explicit perception history

As more components consume shared perception, retention becomes increasingly important.

A Vision consumer may only care about the latest few observations.

Memory may need a much larger temporal window.

Persona may need enough history to resolve identity across multiple observations.

An Interaction Router may need recent audio and visual activity together.

v0.6.6 increases the usable retention window for perception and output streams and continues moving these values toward configurable runtime policies.

At the same time, Stream and Observation concepts are becoming more explicitly represented through enums and schemas instead of relying on loosely defined values.

This matters because once multiple independent consumers share a realtime stream, semantics such as:

• observation type;

• ordering;

• retention;

• cursor position;

• lifecycle;

• missing observations;

• and replay boundaries

eventually become part of the runtime API.

Some of those semantics are still being developed, but v0.6.6 moves more of the underlying representation in that direction.

Continuing to isolate the framework from LiveKit

LiveKit is still AlphaAvatar’s primary realtime transport and agent integration.

I do not intend to remove it—it provides a very strong realtime foundation.

But AlphaAvatar’s internal runtime should not require every component to understand LiveKit-specific behavior.

The direction remains:

LiveKit / future RTC backend
            ↓
adapter / entrypoint boundary
            ↓
AlphaAvatar runtime contracts
            ↓
plugins and application logic

In v0.6.6, more worker-specific compatibility behavior has been moved toward AlphaAvatar’s entrypoints instead of being spread through the internal runtime.

This is another small step toward treating LiveKit as an integration boundary rather than the definition of AlphaAvatar’s architecture.

Longer term, I would like the same Memory, Persona, perception, and orchestration components to be usable with different realtime transports without rewriting the assistant itself.

Session storage is becoming easier to operate

There are also a few less visible operational changes.

Session artifacts are now grouped according to their creation date instead of accumulating in a single flat session directory.

This sounds minor, but persistent realtime assistants produce a lot of runtime state over time.

As sessions become useful for:

• debugging;

• replay;

• evaluation;

• tracing;

• Memory inspection;

• and behavior analysis,

their storage layout becomes part of the developer experience.

The goal is for AlphaAvatar to remain inspectable even after it has been running for weeks or months rather than only being understandable during a single demo session.

Why keep these capabilities componentized?

There is an interesting trend toward moving more agent functionality directly into increasingly capable models.

I expect that trend to continue.

Models will become better at:

• perception;

• memory selection;

• context compression;

• planning;

• tool routing;

• user modeling;

• multimodal reasoning.

But I still think there is value in keeping a runtime around those capabilities.

For me, the role of a framework like AlphaAvatar is not to duplicate intelligence that could exist inside the model.

It is to make that intelligence stable, replaceable, observable, and operationally usable.

If every capability becomes an opaque part of one model invocation, several things become harder:

• debugging why a particular behavior occurred;

• updating one capability without changing unrelated behavior;

• controlling inference cost;

• swapping providers or models;

• reproducing failures;

• enforcing retention and privacy policies;

• inspecting persistent state;

• measuring individual subsystem quality.

So my current mental model is increasingly:

Perception   Memory   Persona   Tools   Runtime state
     \         |        |        /         /
                Model
                  ↓
        reasoning / decisions
                  ↓
          runtime commitments

The model is the connective reasoning layer and ultimately an important decision maker.

The framework provides the persistent and observable environment in which those decisions can operate.

This also makes it easier for individual components to evolve independently as models improve.

What v0.6.6 does not solve yet

There are several larger runtime questions that remain open.

In particular, AlphaAvatar still needs stronger semantics around:

• consumer lag and missed observations;

• retention boundaries and resume behavior;

• observation coverage;

• evidence and provenance;

• current state versus historical memory;

• memory candidate versus durable memory;

• conflict and correction handling;

• replay and long-running evaluation.

I have been thinking about these more after the discussion around the previous AlphaAvatar post.

One direction I am interested in is making the transition from model interpretation to persistent state more explicit:

Observation
     ↓
Model interpretation
     ↓
Memory / state candidate
     ↓
Runtime policy
     ↓
Commit / merge / reject / defer
     ↓
Durable state

v0.6.6 does not implement this complete commitment model.

The changes in this release are more foundational: making streams, runners, perception inputs, plugin capabilities, and runtime boundaries consistent enough that these policies can later be added without every subsystem inventing its own semantics.

Questions I am currently thinking about

I would be interested in hearing how others working on persistent or realtime agents approach a few related problems:

1. Should audio, visual, and conversational ENV memories eventually share one event representation, or should modality-specific representations remain separate until retrieval time?

2. How much information about runtime capabilities should be placed directly into model context?
   Should the model always know the complete active capability set, or should most runtime information itself be retrieved on demand?

3. How explicit should stream semantics become in an agent framework?
   For example, should consumer lag, retention boundaries, missed observations, and replay behavior be first-class public contracts?

4. Where should the boundary between model reasoning and runtime commitment sit?
   Especially for durable Memory, Persona updates, external actions, and current-state changes.

5. For interchangeable storage and inference components, how much behavior should be standardized by the framework versus left to each backend?

I am especially interested in real failure cases from systems that have been running continuously rather than only short-lived agent benchmarks.

Links

GitHub: GitHub - AlphaAvatar/AlphaAvatar: A real-time interactive Omni Avatar built on LiveKit, which allows you to seamlessly integrate with any open source Avatar components (real-time model, visual, voice, memory, search, etc.). · GitHub

Previous v0.6.4 architecture discussion:

AlphaAvatar v0.6.4: a self-hostable realtime multimodal assistant runtime with shared perception and ENV memory

Hi everyone A few weeks ago, I shared an early introduction to AlphaAvatar, an open-source and self-hostable realtime full-multimodal personal AI assistant runtime. The discussion around that post was very helpful. One particularly useful point was that AlphaAvatar may be more valuable when positioned less as another avatar frontend, and more as a reusable runtime layer for persistent realtime multimodal agents. That is increasingly how I see the project as well. AlphaAvatar do…

Thanks again to everyone who contributed to the earlier discussion.

Some of the feedback there has been genuinely useful in clarifying where AlphaAvatar should keep explicit runtime boundaries instead of simply adding more behavior to the model.

AlphaAvatar is still evolving quickly, so implementation feedback, architecture criticism, related projects, and examples of failure modes are all very welcome.

[John6666]

Hi. At a glance, this looks like a substantial step forward:

---

My short answers to the five design questions would now be roughly:

Question	My current answer
1. One multimodal event representation, or keep modalities separate?	I would unify identity, time, provenance, and episode/correlation semantics before trying to unify the actual modality payloads. Keep raw/audio/visual/conversation evidence independently inspectable, then derive a cross-modal episode or Memory representation when useful.
2. Put the full active capability set in context, or retrieve it on demand?	I would separate runtime capability truth from what detail the model sees on each turn. For a small stable set, a compact full summary is simplest. For a large/dynamic set, keep a compact always-visible summary and retrieve detailed capability schemas on demand.
3. How much of Stream semantics should become public contract?	Probably the observable semantics: consumer identity, position, retention boundary, missed/gap information, reconnect/clear behavior, and meaningful drop states. Buffering/scheduling internals can remain private.
4. Where should model reasoning stop and runtime commitment begin?	Models can interpret, associate, summarize, rank, and produce candidates. The runtime should own decisions that become authoritative, durable, permissioned, delivered, or externally consequential. I would also distinguish reversible/retryable commitments from irreversible ones.
5. How much should Memory backends be standardized?	Standardize the framework-visible outcomes and guarantees, not each backend’s internal storage mechanics. A common interface should not silently imply stronger consistency/durability than a backend actually provides.

The default route I would take from here is still fairly small:

make the existing semantics explicit
        ↓
preserve lightweight identity/provenance across the important boundaries
        ↓
cover those boundaries with a few deterministic scenarios
        ↓
only then decide which larger representation/protocol/backend abstractions
actually need to be standardized

I tried a couple of small CPU-only synthetic checks against the v0.6.6 code because the current implementation is now concrete enough that some of these questions can be tested without choosing the final architecture.

Two older concerns look materially better now:

• Stream gaps are visible. A bounded PerceptionStream crossing its retention boundary now exposes first_available_seq, missed_count, and has_gap.
• Runtime-only observations have a useful provenance receipt. A pathless EnvObservation can still expose an observation_id, source_id, time range, metadata, etc., without requiring the raw frame to be persisted.

That seems like exactly the kind of incremental contract hardening discussed in the v0.6.4 thread.

The two seams that now look most interesting to me are narrower:

1. “captured / acknowledged” is not necessarily the same thing as “durably memorized”;
2. Observation-level evidence exists, but the current ENV Memory write path does not yet appear to carry the same source identity all the way into the durable Memory item.

I do not think either observation implies a large redesign. They mostly look like places where a small amount of explicit state/identity could make the eventual contract much easier to test.

1. Multimodal representation: I would separate identity, association, evidence, and projection

I would be a little cautious about treating this as a binary choice between:

one unified multimodal representation

and

completely modality-specific representations

because there are several different things that can be unified independently.

A useful decomposition might be:

A. observation/event identity
B. temporal association / episode membership
C. modality-specific evidence
D. derived cross-modal episode
E. durable Memory representation
F. retrieval projection

I would be relatively aggressive about standardizing A, explicit about B, conservative about discarding C, and leave D–F comparatively flexible until actual retrieval/evaluation results make the trade-off clearer.

A. Common identity/time/provenance

A common envelope could stay very small:

observation_id:
source_id:
modality:
occurred_at:
recorded_at:
session_id:
participant_id:
correlation_id:
provenance:

This does not require audio, frames, transcripts, and annotations to use the same payload schema.

CloudEvents is useful here only as vocabulary: it deliberately separates common context attributes (id, source, type, time, etc.) from domain-specific event data. That is a useful precedent for “common event identity does not imply common payload semantics.”

I would not make AlphaAvatar a CloudEvents implementation just for this; the interesting part is the separation.

B. Association is a different problem from representation

There is also a question that comes before deciding how Memory should represent a multimodal event:

When do an audio segment, a video frame, a transcript, and a later identity annotation count as evidence for the same occurrence?

That is an association/synchronization policy.

The robotics/sensor side has a very mature version of this distinction. ROS message_filters keeps sensor messages in their native forms while TimeSynchronizer / ApproximateTimeSynchronizer decide which timestamped messages belong together. ApproximateTimeSynchronizer even makes the allowed temporal mismatch (slop) explicit.

I am not suggesting importing ROS semantics into AlphaAvatar. I mainly think it is a useful reminder that:

same episode
≠
same representation

For AlphaAvatar, a plausible route is:

audio observation ─────┐
video observation ─────┼─> shared episode/correlation identity
annotation ────────────┘
                              │
                              ├─ modality-specific evidence remains inspectable
                              │
                              └─ optional derived cross-modal Memory

That would preserve future choices.

C. There does not seem to be one settled multimodal-Memory architecture anyway

Recent systems are still exploring quite different points in the design space:

• M3-Agent uses continuous visual/audio input with entity-centric multimodal memory.
• EgoMem treats lifelong audiovisual memory with asynchronous retrieval/dialog/memory-management processes.
• EventMemAgent detects event boundaries and archives event-level representations.
• TaskMem moves some of the problem from “what representation?” to “what should be memorized at all?”

So I would not interpret the literature as saying that AlphaAvatar needs to settle on one universal Memory object now.

My default would be:

unify event identity + temporal/provenance semantics first
preserve modality evidence
derive Memory projections later

That seems to leave the most room for the system to evolve without losing inspectability.

2. Capability descriptions: separate runtime truth from model exposure

For capabilities, I think there are at least three separate layers:

1. capability truth
2. operational availability / authority
3. what is exposed to the model right now

For example:

installed
≠ enabled
≠ healthy
≠ authorized for this participant
≠ useful for this turn
≠ included in the current model context

That distinction matters more to me than whether the prompt uses “full list” or “retrieval.”

The current MCP Tools specification is a useful comparison here. Its tools/list operation returns tools currently available to the requesting client; the list may change, may depend on authorization, and a server can advertise listChanged.

That does not answer AlphaAvatar’s prompt-design question, but it does give useful vocabulary:

live capability state is runtime truth; prompt exposure is a separate policy.

For a small active set

If AlphaAvatar normally has something like:

Memory
Persona
RAG
MCP
DeepResearch
Interaction Router
Character

then putting a short, stable capability summary in the Avatar context is probably simpler than building retrieval machinery around seven tiny descriptions.

For a large or highly dynamic set

If the set expands into hundreds of MCP tools, skills, channel-specific actions, backend-specific operations, etc., then a hybrid seems more attractive:

always visible:
  compact capability/category summary
  current availability/authority state

retrieved when needed:
  detailed schema
  long usage instructions
  examples
  edge-case constraints

This also avoids conflating “the runtime knows this capability exists” with “the model must spend context tokens reading its entire schema on every turn.”

There is some recent work suggesting that tool descriptions themselves materially affect tool selection, but that adding more detail can also add steps/cost and sometimes regress behavior, so I would not assume “more descriptions in context” is monotonically better.

The low-cost thing to make explicit now is probably just:

capability identity
current availability
authority/scope if relevant
model-facing short description

while leaving the context-delivery strategy replaceable.

3. Stream semantics: public observable behavior, private implementation

This is the area where v0.6.6 looks most clearly improved to me.

A very small test of the current PerceptionStream gave the expected kind of result after retention overflow:

first_available_seq = 3
missed_count        = 2
has_gap             = true

So I would no longer frame the issue as “the consumer silently misses data.” The runtime now gives the consumer enough information to know a retention gap happened.

The remaining question is more precise:

What does a consumer identity mean across cleanup/reconnect/restart?

For example, in the same small test:

consumer reads
→ commits
→ continues normally

preserved the committed position.

But after:

clear_consumer("same-id")
→ read again as "same-id"

that consumer behaved like a fresh reader over the retained tail, with the missing range explicitly reported as a gap.

That can be a perfectly reasonable realtime contract. It just has different semantics from a durable-resume contract.

So I would make the behavior, rather than the implementation, public:

consumer identity
read/current position
committed position, if distinct
first available position
missed/gap information
what cleanup means
what reconnect means
what loss/drop states can occur

The queue implementation, locking, scheduling, buffer representation, etc. can stay internal.

Kafka’s consumer model is useful only as an established vocabulary example here: it explicitly distinguishes the consumer’s current position from its securely stored committed position, and manual offset control is useful when “consumed” should mean “processing completed.”

I would not import Kafka semantics wholesale into a realtime perception stream. In particular, old video frames are often correctly disposable.

The useful decision tree seems more like:

If this stream is freshness-oriented / best effort:
    retained tail + explicit gap may be sufficient.

If downstream work is recoverable:
    preserve a stable recovery identity or cursor.

If downstream work has a durable-delivery guarantee:
    acknowledge that guarantee at the later durable boundary,
    not merely when the observation was read.

That last branch connects directly to the next question.

4. Model reasoning vs runtime commitment: the code now exposes a very concrete boundary

I still like the boundary from the previous discussion:

Observation
    ↓
model interpretation
    ↓
candidate
    ↓
runtime policy
    ↓
commit / merge / reject / defer
    ↓
durable or authoritative state

The interesting thing in v0.6.6 is that this is no longer just an abstract diagram. There are now concrete implementation seams where the distinction matters.

A small fault check: capture acknowledgement can precede Memory-processing success

I ran the real v0.6.6 EnvMemoryScheduler control flow with synthetic dependencies and forced the processing callback to fail.

The sequence was:

capture ENV batch
→ perception event cursor committed
→ processing attempt fails
→ retry
→ processing attempt fails again
→ retry limit exhausted
→ batch leaves pending state

During both processing attempts, the perception cursor was already advanced.

I would not call this a data-loss bug from that test. The provider/cache/perception dependencies were synthetic, and I did not reproduce a full process crash/restart/backend-recovery path.

What it does demonstrate is a legitimate intermediate state:

captured / acknowledged
but
not successfully processed into Memory

That might be completely intentional if ENV Memory is best-effort.

The design choice can then stay small:

If best-effort ENV Memory is intended:
    exposing "processing failed/dropped" in trace/metrics may be enough.

If failed processing should be recoverable:
    keep a retryable batch identity / recovery point.

If durable Memory delivery is promised:
    distinguish capture acknowledgement from durable-memory acknowledgement.

No large abstraction is required just to make the distinction explicit.

Not all “commitments” have the same recovery semantics

I would also avoid making commit one universal binary concept.

These have very different failure/recovery properties:

write a Memory candidate
update Persona
mark a fact as current state
send an email
call a device action
speak audio to the user

A useful classification may be:

reversible
compensatable
idempotently retryable
externally irreversible

Distributed-workflow systems have similar vocabulary. For example, the Saga pattern distinguishes compensable operations, a “pivot”/point of no return, and retryable operations after that point.

Again, I would not add a Saga engine to AlphaAvatar. I only think the vocabulary helps prevent:

memory persisted

and

email sent

from accidentally receiving identical “commit” semantics just because both were proposed by the model.

My preferred boundary remains:

model owns:
    interpretation
    association
    summarization
    candidate generation
    ranking / suggestion

runtime owns:
    authority checks
    policy decision
    durable state transitions
    external side effects
    delivered-output accounting

The runtime does not need to understand the model’s entire reasoning process. It only needs enough structured information to know what proposed change it is being asked to make and what eventually happened to it.

5. Provenance: the Observation layer looks fixed; the remaining seam is the Memory write path

This was probably the clearest result of the small checks.

A pathless runtime observation now produces a useful evidence receipt. So the older issue:

no persisted media path
→ empty provenance

does not seem to describe v0.6.6 anymore.

I then tried a narrower round trip:

Observation ID
→ evidence receipt
→ fake ENV Memory delta
→ MemoryItem
→ VDB serialization helpers
→ reconstructed MemoryItem
→ Markdown backup

and compared two cases.

Surface	Current ENV-path mirror contains source Observation ID	Positive control: evidence explicitly attached
MemoryItem	no	yes
flattened VDB payload	no	yes
rebuilt item	no	yes
Markdown backup	no	yes

The important part is the positive control.

It suggests that the existing serialization surfaces are capable of carrying the Observation provenance when it is supplied. The narrower seam appears to be the current ENV MemoryItem construction path: evidence is constructed/cached, while the direct evidence attachment in that path is still marked as a TODO.

I would phrase that very narrowly:

Observation-level provenance exists, and the serializers can preserve it; the remaining question is how much of that source identity should cross into durable Memory.

That is quite different from saying “AlphaAvatar loses provenance.”

There may be other runtime/cache/graph paths from which provenance can be recovered, and storing a complete evidence blob may be unnecessary or undesirable.

Given the privacy/self-hosted goals, I would probably start with the smallest useful durable link, for example:

source_observation_ids:
  - ...
source_event_ids:
  - ...

or perhaps a compact receipt/correlation ID, rather than copying raw frames or large evidence objects into every Memory item.

Why I think this link is worth preserving

It becomes useful for several future operations without choosing the final Memory architecture:

Why does the assistant believe this?
Which observations produced this Memory?
Was this state inferred from stale evidence?
What should be reconsidered after a correction?
Can an eval distinguish a bad interpretation from missing evidence?

Graphiti is an interesting comparison, not a prescription. Its current implementation distinguishes event/reference time from ingestion time and exposes provenance from episode UUIDs to derived graph elements. It also preserves valid_at / invalid_at style temporal semantics for facts.

The part I find relevant to AlphaAvatar is not “use a temporal graph.” It is simply that:

raw/source episode identity
and
derived/current state

can remain connected without being the same object.

6. Memory backend standardization: normalize guarantees, not internals

For Memory backends I would avoid defining one interface that accidentally makes LanceDB, Qdrant, Markdown backup, graph persistence, etc. appear more semantically equivalent than they are.

The storage systems themselves expose different consistency controls.

For example, LanceDB’s consistency docs expose read_consistency_interval:

default       → no automatic cross-process refresh
0             → check for updates on every read
non-zero      → eventual refresh after an interval

while Qdrant separately exposes concepts such as:

write_consistency_factor
read consistency
write ordering

So a backend-neutral method like:

await memory.save(item)

cannot by itself tell the rest of AlphaAvatar:

Has the write merely been accepted?
Is it durable?
Will an immediate retrieval see it?
Is it visible on every replica?
Can this operation be retried safely?

I would therefore standardize the framework-visible result more strongly than the storage implementation.

Something conceptually like:

item_id:
write_status:
durable:
query_visible:
retryable:
revision:

does not all have to ship at once; even two or three of those fields could remove ambiguity.

The main rule I would want is:

A common backend interface should not silently promise stronger consistency or durability than the selected backend actually provides.

That also makes backend-specific optimization easier, because the contract describes the result AlphaAvatar needs rather than prescribing how LanceDB/Qdrant/etc. must achieve it.

7. A tiny deterministic scenario suite looks higher-value than a larger abstraction right now

At this point I think a surprisingly small scenario suite would buy a lot.

I would start with only three core cases.

Scenario A — capture succeeds, processing fails

observations arrive
→ batch captured
→ processing fails
→ retry limit reached

Record only:

source_observation_ids:
capture_cursor:
capture_status:
memory_processing_status:
retry_count:
terminal_reason:
durable_memory_ids:

The purpose is not to require replay. It is simply to make the current semantics undeniable.

Scenario B — provenance round trip

Observation
→ evidence
→ Memory candidate/item
→ persistence
→ reload

Assertion:

Can the durable Memory record identify the source observation(s),
if the configured contract says it should?

Scenario C — retention + reconnect

consumer reads
→ commits
→ retention advances
→ consumer is cleared/disconnected
→ reconnect

Record:

previous_committed_cursor:
first_available_cursor:
resumed_cursor:
missed_count:
duplicate_count:

These three cases together cover a large part of questions 3 and 4 without any model benchmark, GPU, or external API.

Later, if useful:

D. multimodal late arrival
E. correction / old-current-state invalidation
F. backend partial failure
G. capability becomes unavailable during session

can be added.

A nearby project that recently went through a similar maturation is DeerFlow.

Its run-event-stream analysis started from the observation that one internal event stream had become the source for frontend history, debugging, token accounting, and evaluation, while its semantics still mostly lived in implementation details. That issue has since been closed by work that introduced an explicit run-event contract/documentation/conformance path.

Its separate eval RFC also takes a useful low-cost order:

deterministic replay
→ trajectory assertions
→ outcome evaluation
→ optional live/judge layers

I do not think AlphaAvatar should copy DeerFlow’s event schema. The useful lesson is smaller:

once an internal runtime record has multiple consumers, specifying and deterministically testing the existing semantics can be more valuable than designing a more general protocol first.

That seems very close to where AlphaAvatar is now.

8. If you introduce a trace schema, I would keep the first version deliberately boring

Given the earlier discussion about an internal trace contract before an out-of-process protocol, I think that still looks like the right order.

An illustrative internal shape could be:

identity:
  event_id:
  session_id:
  correlation_id:
  causation_id:
  source_observation_ids:

time:
  occurred_at:
  recorded_at:

stream:
  consumer_id:
  read_cursor:
  committed_cursor:
  first_available:
  missed_count:

memory:
  candidate_id:
  policy_decision:
  persistence_status:
  durable_memory_ids:
  backend_revision:

failure:
  stage:
  retry_count:
  terminal_reason:

I would not treat that as a schema proposal so much as a checklist of distinctions the deterministic scenarios might need.

Two fields I would seriously consider keeping separate from the beginning are:

occurred_at
recorded_at

because late-arriving perception/correction data eventually makes one timestamp ambiguous.

This is a very old distinction in stream/temporal systems, and Graphiti is also a nearby agent-memory example that now explicitly distinguishes event/reference time from ingestion time.

Likewise:

read_cursor
committed_cursor

may or may not both be required in AlphaAvatar, but the current ENV scheduler behavior shows why it is worth deciding rather than letting “cursor” acquire several meanings later.

The public/wire protocol can stay much smaller than this internal diagnostic representation.

9. A few nearby references I found useful as maps, not prescriptions

These are useful mostly because each supplies vocabulary for one narrow part of the problem.

Event/runtime contracts

• DeerFlow run-event-stream issue — a de facto internal event stream becoming an explicit/versioned contract once frontend/debug/eval/accounting all depend on it.
• DeerFlow eval RFC — deterministic replay and trajectory checks before live/judge-based evaluation.
• CloudEvents specification — common event context separated from opaque/domain-specific event data.

Streaming / resumption

• Kafka consumer documentation — mature vocabulary separating current consumer position from committed recovery position. Not a recommendation to give perception streams Kafka semantics.

Multimodal association

• ROS message_filters — timestamp-based association of multiple sensor streams without requiring one common sensor payload representation.

Temporal/provenance Memory

• Graphiti — episodes, temporal validity, and provenance from source episodes to derived graph elements; useful as vocabulary for current-vs-historical and evidence-vs-derived-state distinctions.

Capability availability

• MCP Tools specification — the available tool set can be dynamic and authorization-dependent, while listChanged makes changes explicit. Useful for separating runtime availability from model-context policy.

Storage semantics

• LanceDB consistency and Qdrant consistency guarantees — concrete examples of why a shared Memory API should describe the guarantees AlphaAvatar observes rather than assume all vector backends behave identically.

10. Limits of the small checks

For clarity, I would not over-read the synthetic results.

Stream / Observation checks

These exercised the real v0.6.6 core data structures directly, but they were intentionally small synthetic cases rather than long-running production sessions.

ENV scheduler fault check

The scheduler/control flow was the pinned v0.6.6 implementation, but the external perception/cache/processing dependencies were replaced with deterministic stubs so processing could be forced to fail.

So the supported statement is:

this intermediate control-flow state exists

not:

a production data-loss incident has been demonstrated

A real process crash/restart/backend recovery path could add semantics that the synthetic test did not exercise.

Provenance round trip

The test deliberately removed model/provider variability by supplying a fake EnvMemoryDelta.

It used the current MemoryItem field shape plus the real flatten/rebuild helpers and Markdown writer.

The supported statement is therefore narrow:

the checked serialization paths can preserve Observation provenance
when it is supplied, while the current mirrored ENV MemoryItem construction
does not directly carry that Observation ID

It does not establish that no other transient/runtime/graph path can recover source provenance.

That distinction is why I think this is mainly a useful contract seam rather than a bug report.

Overall, I would resist standardizing more of the system than necessary.

The places where standardization seems to pay for itself are the places where ambiguity becomes expensive:

What observation/event is this?
When did it occur, and when did the runtime learn about it?
Was anything missed?
Was it only captured, or also successfully processed?
Is this interpretation still a candidate?
Did runtime policy accept/reject/defer it?
Did it become durable/current state?
Can the durable result still point back to its evidence?
What guarantee did the selected backend actually provide?

Everything else can stay surprisingly flexible.

So if ENV Memory is intentionally best-effort, I think visibility of the intermediate/drop state may be enough.

If failed processing should be recoverable, the same seam could instead become a retry/recovery identity.

If durable Memory delivery becomes an explicit guarantee, the acknowledgement boundary can move later without changing what “perception capture” means.

Likewise, for multimodal Memory I would settle identity + temporal association + provenance before settling one universal representation; for capabilities I would keep runtime truth separate from context-delivery policy; and for Memory backends I would standardize observable guarantees rather than internal consistency mechanisms.

That seems to preserve the direction of the project: the model can keep getting smarter and more model-native over time, while the runtime stays useful because the consequential boundaries remain inspectable, testable, and replaceable.

[AlphaAvatar]

Thanks again for taking the time to test this against the actual v0.6.6 code. This is especially useful because it turns several of the architectural questions from the previous thread into concrete runtime states we can reason about.

I agree with your main distinction around ENV Memory:

capture acknowledgement is not the same thing as durable memory commitment.

Right now, ENV Memory is much closer to a best-effort processing pipeline than a durable-delivery system. So the state you reproduced — observations being captured, processing failing, retries being exhausted, while the perception cursor has already advanced — does not necessarily violate the current intended semantics.

But I agree that the state itself should be explicit and observable.

I think it would be useful to separate at least:

observation captured
        ↓
memory processing succeeded
        ↓
memory persisted

rather than allowing a single notion of “commit” to blur those boundaries.

I probably would not introduce transactional replay or durable delivery guarantees yet, but exposing terminal states such as processing_failed, retry_exhausted, or dropped feels like the right next step.

Your provenance test also identified a boundary I want to tighten.

It is good to see that pathless observations can now produce evidence receipts, and that the persistence layer itself can round-trip provenance once it is present. That suggests the remaining problem is much narrower: carrying observation identity through the ENV → MemoryItem construction path.

I am leaning toward keeping that lightweight — something like source observation / correlation IDs rather than embedding the entire evidence object into durable memory.

That would already make questions like:

Why does the assistant remember this?
        ↓
which observations produced this memory?

much easier to answer during debugging, evaluation, correction, and replay.

I also really like the way you decomposed multimodal memory.

same episode ≠ same representation

That matches the direction I am leaning toward.

I would rather standardize a small shared identity/temporal layer — observation ID, modality, timestamps, session/participant identity, correlation or episode identity — while allowing audio, visual, transcript, and derived annotation payloads to keep modality-specific representations.

Trying to force all of those into a universal multimodal event schema now would probably standardize too much too early.

The capability distinction is useful as well:

capability truth
≠ operational availability
≠ authority
≠ model exposure

v0.6.6 currently only solves a relatively small part of that problem: giving the Avatar a description of the capabilities supplied by active plugins. I think that is sufficient at the current scale, but the distinction becomes important once capabilities become numerous, conditional, unhealthy, permission-dependent, or dynamically discoverable.

The same applies to the Qdrant/LanceDB runner convergence. The intention of the common runner contract is to unify framework behavior, not to imply that the storage engines provide identical durability or consistency semantics. I like the framing of standardizing the guarantees AlphaAvatar exposes while allowing backend-specific semantics underneath.

And I think your three deterministic scenarios may actually be the most actionable suggestion here:

1. capture succeeds → processing fails → retry exhausted

2. observation provenance → persistence → reload

3. retention advances → consumer reconnects/resumes

I would prefer to pin these existing semantics with small deterministic tests before introducing a larger event/recovery protocol. That gives us a much clearer basis for deciding which ambiguities are actually expensive enough to deserve new abstractions.

The reconnect case is particularly interesting. v0.6.6 makes retention gaps visible, but that means the next question is no longer “can a consumer silently miss observations?” — it is “what exactly does consumer identity mean across cleanup, reconnect, and process restart?”

That feels like a much better-defined problem.

Thanks again for doing the synthetic checks rather than only discussing the architecture abstractly. This is exactly the kind of feedback that helps determine where AlphaAvatar needs a stronger runtime contract, and where keeping the implementation deliberately lightweight is still the better choice.