0
Kai Detmers
← Project index
Generative interaction engine · streaming · multimodalAdvanced prototype

Walkaround— typed explainers with progressive playback.

Walkaround turns a question, concept or code change into a streamed interactive artifact: structured scenes, narration, generated media, pointer choreography and learning interactions. The model authors a constrained presentation format rather than unrestricted markup or a long chat response.

Stack

RuntimeNode.js · ESM
Agent interfaceModel Context Protocol SDK
NarrationAzure / Microsoft Speech · alternate TTS paths
ContractsZod · typed content + visual recipes
Player QAPlaywright · accessibility checks
Mediaimage search · coherent sets · attribution

Engineering evolution

Reconstructed from commit history, current source and design specifications. The sequence is evidence-led; explanatory rationale is condensed rather than presented as a literal design diary.

01
Start with a structured document model.

The first architectural constraint was a scene/document format the player could validate and render. That made partial validity, progressive playback and later visual QA possible.

02
Treat narration as a timing subsystem.

Silence trimming, playback speed, cadence, native timing and pooled synthesis were added because audio latency directly changes the perceived responsiveness of the entire experience.

03
Constrain visual entropy.

As the recipe catalog grew, visual-mix auditing was added to limit layout, animation and image density. Variety without constraints produced inconsistent explanatory structure.

04
Make pointer motion semantically meaningful.

Pointer movement became target-aware and narration-aware. Technical content required precise anchors so motion communicates current focus rather than adding generic animation.

05
Use default-first controls with explicit overrides.

Format, energy, depth, visual density and delivery became separable dimensions while automated selection remains the default. The system stays controllable without turning every request into a configuration form.

06
Harden the renderer for broader workflows.

Reduced motion, high contrast, semantic dialogs, image attribution and pull-request walkthrough modes moved the renderer toward a reusable presentation runtime rather than a single demo flow.

Interaction architecture

One focal sceneA scene answers one visual question. Limiting simultaneous concepts keeps narration and visual focus synchronized.
Typed recipe grammarComparisons, timelines, diagrams, cards and image compositions are bounded components. The model selects and fills them; it does not invent arbitrary DOM on every generation.
Pointer as narration cursorMotion is justified when it narrows attention. The pointer ties spoken content to visible targets in code, diagrams and equations.
Interaction interrupts autoplayKnowledge checks intentionally stop passive playback. User action becomes part of presentation state instead of an optional decorative layer.
Progress before completenessThe player renders coherent chunks as soon as they are valid. Narration and media can attach later, reducing blank waiting states in multimodal generation.
Controlled visual energyVisual-mix rules cap animation, image and layout variety. The renderer can be expressive without becoming visually stochastic.

System shape

Author→Agent emits typed explainer chunks and presentation intent.
Document→Stable semantic anchors support partial validity and later media/timing attachment.
Media→TTS and image work execute independently with concurrency and latency controls.
Player→Browser renders the latest coherent state, synchronizes narration, pointer and interaction, and updates incrementally.

Current constraint

The engine can continue expanding its feature surface, but further complexity is no longer the primary objective. The next constraint is product validation: identify a repeatable explanation workflow with measurable return usage and let that evidence determine which system capabilities deserve further investment.

Asyncdocument, narration and media remain independently schedulable
Typedgenerated UI is constrained into testable presentation recipes
Validationrepeat-use workflow before additional surface-area expansion

Related