Wan-Streamer v0.2: Higher Resolution, Same Latency
Abstract: We present Wan-Streamer v0.2, a latency-preserving upgrade of the native-streaming, end-to-end audio-visual interaction model. v0.2 keeps the v0.1 modeling formulation, but raises the interactive output stream from 192x336 to 640x368 while preserving approximately 200 ms model-side signal-to-signal latency at 25 FPS. The higher-resolution stream supports scene-grounded mid-shot agents whose posture, gaze, hands, nearby objects, and local scene layout remain legible during real-time conversation. To support the larger visual stream without adding user-visible delay, v0.2 keeps the thinker as a single-GPU low-latency path for streaming perception, the short language/state Transformer pass that builds the generation cache, and final decoding. The performer becomes a multi-GPU Ulysses-style context-parallel group for the expensive next-unit latent generation. Each performer rank writes incoming K/V into a pre-sharded local cache. The long high-resolution latent video sequence is split across ranks for denoising and gathered through Ulysses communication, while the much shorter audio latent sequence is generated without sequence sharding. In this split, the thinker's language/state computation reaches the performer only as K/V conditioning, so no separate language sequence has to be communicated inside the performer group. This concentrates additional hardware on visual generation while preserving the compact thinker-performer boundary, keeping total remote interaction latency at approximately 550 ms when a 350 ms bidirectional network budget is included.
First 10 authors:
Summary
- The paper demonstrates that upgrading video resolution to 640×368 achieves higher visual fidelity while preserving ~200ms model latency in a native-streaming system.
- It introduces a bifurcated Thinker–Performer model where the single-GPU Thinker manages lag-sensitive tasks and the multi-GPU Performer efficiently denoises high-resolution video latents.
- Experimental outcomes show improved legibility of facial expressions and environmental details, enabling more immersive, scene-grounded digital agent interactions.
Wan-Streamer v0.2: Preserving Latency in High-Resolution, Native-Streaming Audio-Visual Dialogue
Model Formulation and Upgrade
Wan-Streamer v0.2 advances the paradigm of end-to-end native-streaming multimodal interaction established by v0.1. The architecture frames spoken dialogue, language, and rich visual signals on a shared, block-causal timeline coordinated by a single Transformer. This causal modeling allows audio, video, and linguistic histories to jointly condition agent generation, maintaining tight temporal coherence between modalities during real-time exchange.
In v0.2, the system's output video resolution is elevated from 192×336 to 640×368 at a fixed 25 FPS, targeting a significantly broader and higher-fidelity visual composition. This enhancement preserves posture, hand motion, gaze, physical context, and scene layout, expanding the setting from face-centric, close-up interaction to mid-shot, scene-grounded digital agents. Critically, this upgrade is achieved while maintaining approximately 200 ms model-side signal-to-signal latency.
Figure 1: Wan-Streamer architecture; block-causal audio, language, and video states evolve on a unified timeline and are processed via block-causal attention.
Latency-Preserving Infrastructure
Meeting the increased computational cost of high-resolution video generation without degrading responsiveness necessitated a re-architecting of model serving. Wan-Streamer v0.2 bifurcates the model into two optimized paths:
- Thinker: Resides on a single GPU and manages lag-sensitive perception, causal language/state update, KV-cache construction, and final audio-visual decoding. The Thinker's output is a compact K/V cache slice that incorporates all language/state information necessary for visual and acoustic generation.
- Performer: Implements a Ulysses-style context-parallel group distributed across multiple GPUs. Each Performer rank manages a sharded segment of the total long-sequence K/V cache and is responsible for denoising a disjoint subsequence of the high-resolution video latent. Ulysses collective operations all-to-all/gather efficiently synchronize denoising results across ranks.
Figure 2: v0.2 system decomposition, showing the single-GPU Thinker for low-latency tasks and Ulysses-style, multi-GPU Performer for context-parallel video latent generation.
Audio latents, being much shorter, bypass sequence sharding to avoid unnecessary parallelism overhead. Only the video sequence is partitioned for parallel denoising, isolating the computational load and preserving the Thinker–Performer interface as a low-latency boundary. Typical remote interaction latency remains at ≈550 ms when including the same 350 ms bidirectional network budget as previously established in v0.1.
Experimental Outcomes
Evaluations retain the same response boundary definition used in v0.1, isolating model-side processing for fair comparison against latency baselines. Despite the much larger visual output, v0.2 achieves model-side “signal-to-signal” delay near 200 ms at 25 FPS, demonstrating that the serving decomposition and context-parallel denoising approach scale efficiently.
Qualitative inspection of generated 640×368 interaction sequences confirms legibility improvements: facial expression, gaze shifts, manual gestures, and environmental details are all preserved for both listening and speaking states, across typical video-call and mid-shot contexts. The temporal stability of latent generation and decoding avoids artifacts commonly manifest in runtime-constrained, high-resolution synthesis. The richer context enables new forms of situated interaction and embodies a shift towards immersive, conversational AI agents.
Implications and Directions
Wan-Streamer v0.2’s design demonstrates that native-streaming causal modeling and real-time multimodal response can scale to substantially higher visual fidelity without recourse to offline postprocessing or increased perceptual delay. The modular Thinker–Performer split—particularly the isolated, context-parallel Performer path—presents a template for efficiently scaling interactive multimodal agents in both research and deployment.
This work also provides an experimental foundation for direct, fine-grained synchrony of audio, language, and visual modalities over long horizons, relevant for downstream settings such as persistent embodied assistants, digitally mediated telepresence, and interactive human-model world modeling. Future directions will likely include: extension to 3D or world-centric representations, finer control of agent-environment interaction, exploration of more granular agent body/scene state modeling, and throughput scaling for large-population deployment.
Conclusion
Wan-Streamer v0.2 achieves high-resolution (640×368) real-time audio-visual streaming while strictly preserving model-side latency. Through a principled, latency-aware Thinker–Performer decomposition using a Ulysses-style context-parallel group for visual denoising, the system supports scene-grounded, mid-shot digital agents with legible expression, posture, and environmental context. This upgrade establishes a new operational ceiling for low-latency, end-to-end, multimodal interactive models (2607.04443).
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper introduces Wan-Streamer v0.2, an upgrade to a system that can talk and show video at the same time in real time. Think of it like a super smart video call partner that listens, thinks, speaks, and shows expressive video of a digital person while you’re talking. The big news: it makes the video much clearer (from about 192p to 640×368) without making you wait longer for responses.
Key Objectives
The paper aims to:
- Make live, generated video look clearer and show more of the person and their surroundings.
- Keep the system just as fast and responsive as before (about 200 ms on the model side and about 550 ms total including internet delay).
- Do this by reorganizing how the system runs on computer chips (GPUs) without changing how the model “thinks.”
How It Works (in simple terms)
Wan-Streamer treats conversation as one continuous stream of events: your voice and video come in, the system understands them, and it produces matching speech and video back, all in sync and in real time.
To hit higher video quality without slowing down, the system splits its work into two cooperating parts:
The “Thinker” and the “Performer” (an analogy)
- The Thinker is like a quick decision-maker on one GPU. It:
- Listens and watches what you do.
- Updates the “conversation state” (what’s been said and shown so far).
- Prepares compact notes for the next video/audio generation step.
- Turns the generated “sketches” (latents) into actual sound and video you can see and hear.
- The Performer is like a team of artists on multiple GPUs. It:
- Takes the Thinker’s compact notes and uses them to create the detailed next chunk of video (and audio) behind the scenes.
- Splits the long, high-resolution video work across multiple team members so they can work in parallel and finish in time.
Key ideas explained with everyday language
- Causal timeline: The system processes time in order, like a conveyor belt—each 160 ms “chunk” of the conversation is handled in sequence.
- Latents: Think of these as rough blueprints of the next bit of video/audio before final polishing.
- K/V cache (Key/Value cache): Like a notebook of recent important details. The Thinker writes brief, useful notes that help the Performer generate the next frames quickly.
- Sharding/parallelism: Splitting a big task among several people so it finishes faster. They each handle a slice of the video sequence and then combine their parts.
- Ulysses-style context parallelism: A specific way of splitting and sharing the workload among GPUs so they can coordinate effectively.
Timing goals
- The system runs at 25 frames per second.
- The model-side delay from “we got your latest 160 ms of audio/video” to “we generated the matching reply” stays about 200 ms.
- Including normal internet time, total delay feels like roughly half a second, which is short enough for natural conversation.
Main Findings
Here’s what the upgrade achieves:
- Higher video resolution: from 192×336 to 640×368, still at 25 FPS.
- Same responsiveness: model-side latency stays about 200 ms; total interactive delay stays around 550 ms when you include a typical network budget.
- Better on-screen presence: the digital person can now be shown in a slightly wider “mid-shot,” so you can see not just their face but also posture, hands, gaze, nearby objects, and a bit of the room—making the interaction feel more grounded and real.
- Smart use of hardware: the Thinker stays fast on one GPU, while the Performer spreads the heavy video work across multiple GPUs. Audio is lighter and doesn’t need to be split.
Why this matters: You get clearer, more informative video during live conversation without losing the “instant feel” that makes talking natural.
Implications and Impact
With v0.2, real-time talking digital agents can:
- Look clearer and more lifelike while listening and speaking.
- Show more of their body and surroundings so gestures and context make sense.
- Keep the conversation snappy and natural, even with higher-quality visuals.
This approach shows how to improve visual quality in live AI interactions by reorganizing how the system runs behind the scenes, rather than slowing things down. It can help future virtual assistants, tutors, customer support avatars, and game characters feel more present and engaging—without making you wait.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what the paper leaves missing, uncertain, or unexplored.
- Quantitative evaluation is absent: no objective audio, visual, and audiovisual synchronization metrics (e.g., lip-sync error, A/V skew, temporal consistency, LPIPS/FVD, hand/gaze tracking accuracy) or human evaluations comparing v0.1 vs v0.2 and against baselines.
- No ablations on latency–quality trade-offs: it is unclear how resolution, denoising steps, latent sizes, cache length, or number of performer ranks impact model-side latency and visual/audio quality.
- Lack of throughput and scalability analysis: no measurements for multi-session concurrency, per-session GPU-hours, tokens/s, or how performance degrades under load or with more concurrent users.
- Unspecified resource and cost footprint: GPU memory per rank, total VRAM, compute utilization, interconnect bandwidth, and energy consumption are not reported, hindering reproducibility and deployment sizing.
- Intra-performer communication requirements are under-specified: Ulysses all-to-all/gather bandwidth/latency needs, sensitivity to NVLink vs PCIe vs Ethernet, and straggler effects are not quantified.
- Thinker–performer communication overhead is unmeasured: size of K/V slices per 160 ms unit, serialization format, compression, and end-to-end overlap with compute are not analyzed.
- KV-cache growth and memory policy are unclear: no details on cache truncation/compaction, long-horizon context limits, or the effect of cache pruning on temporal coherence and responsiveness.
- Robustness to network variability is not evaluated: the fixed 350 ms bidirectional budget assumes stable networks; impact of jitter, packet loss, bandwidth throttling, and adaptive buffering remains unknown.
- Synchronization guarantees are not validated: the paper claims preserved ~200 ms model-side latency but does not report jitter distributions (p50/p95/p99) or audio–video desynchronization under stress.
- Audio path assumptions may not hold at higher fidelity: audio latents are “short enough” to avoid sharding, but the bottleneck risk at higher sample rates, stereo/multichannel audio, emotional prosody, or multilingual speech is not tested.
- Decoder and codec pipeline is unspecified: the algorithms/codecs used for latent decoding, their compute/bandwidth cost, and how decoding scales with resolution/FPS are not described.
- Upper bounds of resolution scaling are unknown: maximum achievable resolution/FPS at ~200 ms model-side latency, and the inflection points where context or spatial parallelism fails to maintain interactivity, are not studied.
- Limited evaluation of “scene-grounded mid-shot” claims: no task definitions, datasets, or metrics for posture/hand/gaze legibility, object grounding, or local layout understanding during real-time interaction.
- Generalization and robustness gaps: performance under occlusions, fast motions, camera viewpoint changes, non-stationary lighting, cluttered backgrounds, or multi-person scenes is not assessed.
- Long-horizon stability is untested: drift, identity consistency, motion stability, and prosody coherence across multi-minute conversations (or longer) are not measured.
- Failure mode characterization is missing: no analysis of typical breakdowns (e.g., hand artifacts, lip–audio mismatch, gaze drift, cache staleness, race conditions across ranks) or automated detection/mitigation strategies.
- Safety, bias, and social acceptability are not addressed: no evaluation of demographic bias, inappropriate gestural/affective behaviors, content moderation in video, or safety red-teaming for interactive settings.
- Personalization and adaptation are unexplored: methods for identity preservation across sessions, user-specific prosody/gesture adaptation, or on-the-fly few-shot adaptation are not described.
- Data and training specifics are omitted: dataset composition for mid-shot scenes, annotation strategy for hands/gaze/objects, training schedules, optimization choices, and any distillation/forcing procedures for streaming are not detailed.
- Reproducibility is limited: the paper does not provide model sizes, layer counts, latent dimensionalities, tokenization schemes, hyperparameters, or open-sourced checkpoints/inference stacks to replicate results.
- Control interfaces are not defined: APIs for explicit control of gaze, gestures, camera framing, or responsiveness, and their latency/quality implications, are not presented.
- Multi-user/multi-agent scenarios are untested: how the system handles crosstalk, barge-in, overlapping speakers, or multiple agents in a shared scene remains unclear.
- Scheduling and QoS under resource contention are unstudied: prioritization across sessions, backpressure strategies, and latency SLO adherence when GPUs are saturated are not evaluated.
- Security and privacy aspects are unaddressed: handling of K/V cache slices across processes/machines, memory isolation in multi-tenant settings, and on-device vs cloud privacy trade-offs are not discussed.
- Transport-layer integration is outside scope: no guidance on bitrate adaptation, error correction, or end-to-end streaming protocols to preserve audiovisual alignment over variable networks.
- Extensibility to 3D/embodied interaction is unclear: whether the current architecture can integrate world models, camera motion, or 3D scene control without breaking the latency budget is not explored.
- Internationalization and accessibility are not covered: multilingual speech quality, code-switching, accents, and accessibility features (e.g., captions, visual simplification) are not evaluated.
Practical Applications
Immediate Applications
The paper enables deployable upgrades and workflows wherever real-time audio-visual agents are used, now with higher visual fidelity at the same interaction latency. Below are concrete use cases, each tied to sectors and practical tools/workflows, with key dependencies noted.
- Scene-grounded customer support avatars that can gesture and reference on-screen objects
- Sectors: software, retail, finance, telecom
- Tools/workflows: drop-in upgrade from 192p to 640×368 in existing contact-center avatar stacks; WebRTC/WebTransport streaming; “Edge Thinker Gateway” near the user; multi-GPU “Performer Pool” in the cloud; fallback to close-up framing on resource pressure
- Assumptions/dependencies: 350 ms round-trip network budget; multi-GPU nodes with Ulysses/NCCL; content safety/brand compliance; codec bandwidth for 25 FPS video
- Virtual sales associates and live-shopping co-hosts with readable hands and product handling
- Sectors: e-commerce, retail media
- Tools/workflows: product cam + agent cam switching; scripted hand-offs from human host; inventory CMS hooks for dynamic product mentions; latency instrumentation to keep each 160 ms unit within budget
- Assumptions/dependencies: reliable studio lighting; product imagery capture; moderation for claims/compliance
- Multimodal in-app assistants for complex UIs (demonstrating gestures, pointing, and gaze)
- Sectors: software/SaaS, developer tools
- Tools/workflows: on-canvas agent window; synchronized voice + mid-shot gestures for onboarding/tutorials; SDK exposing thinker-performer microservices; KV-slice broadcast API; observability for per-unit throughput
- Assumptions/dependencies: app instrumentation for focus/element highlights; edge node placement to meet network budget
- Higher-fidelity classroom tutors and language-learning coaches with lip-synced articulation
- Sectors: education
- Tools/workflows: curriculum LMS integration; prosody-controlled TTS feeding the model; gesture templates for emphasis; multilingual captioning
- Assumptions/dependencies: age-appropriate safeguards; data privacy for minors; adequate compute during school peaks
- Telemedicine receptionists and pre-visit triage with clearer nonverbal cues
- Sectors: healthcare
- Tools/workflows: kiosk/portal agents for intake forms; insurance eligibility dialogs; escalation to clinician; EHR scheduling APIs
- Assumptions/dependencies: HIPAA/GDPR compliance; clinician oversight; risk triage policy; qualitative fidelity acceptable for front desk—not diagnosis
- Concierge kiosks and branch greeters that can indicate directions with torso/hand motion
- Sectors: hospitality, transportation, banking, government services
- Tools/workflows: digital signage integration; wake-word + mid-shot framing; multilingual support; local floor-plan references
- Assumptions/dependencies: acoustically treated spaces; accessibility policies (visual disclosure of synthetic agent)
- VTuber/live-streamer pipelines with lower latency and mid-shot expressiveness
- Sectors: media/entertainment
- Tools/workflows: mixer/OBS plugin; controllable motion presets; stream-safe music ducking; GStreamer-based decode path from performer latents
- Assumptions/dependencies: platform TOS for synthetic content; stream moderation; GPU rental cost
- In-vehicle and smart-TV companions that maintain eye contact and indicate controls
- Sectors: automotive, consumer electronics
- Tools/workflows: on-device thinker (single GPU/accelerator) + cloud performer; hand-off when connectivity drops; offline audio-only fallback
- Assumptions/dependencies: vehicular connectivity constraints; E2E latency budget in motion; energy/thermal limits
- Enterprise meeting co-pilots that present summaries with synchronized gaze and gestures
- Sectors: enterprise productivity
- Tools/workflows: calendar + documents grounding; slide-control hooks; real-time captioning; secure tenant isolation for KV/state caches
- Assumptions/dependencies: data governance; opt-in for synthetic presenter; bandwidth in conference rooms
- Serving-pattern adoption: thinker–performer split and Ulysses context parallelism for long-sequence generation
- Sectors: AI infrastructure, MLOps
- Tools/workflows: reusable microservice pattern; pre-sharded performer KV caches; sequence-sharded video path and non-sharded audio path; autoscaler keyed to unit-time SLOs; NCCL all-to-all/gather templates
- Assumptions/dependencies: cluster networking with high bisection bandwidth; robust KV-slice schema versioning; backpressure handling
- Research platforms for full-duplex human–AI interaction studies with controlled latency
- Sectors: academia (HCI, speech, vision, dialogue)
- Tools/workflows: reproducible 200 ms model-side S2S latency; instrumentation at unit boundaries; controlled manipulation of gaze, timing, and posture
- Assumptions/dependencies: IRB/ethics approval; participant consent for synthetic media
- Policy experiments on disclosure and watermarking of synthetic video dialogue
- Sectors: policy/regulation
- Tools/workflows: evaluate watermark visibility vs. compression; user studies on disclosure UIs; standardized latency reporting (first-frame vs. S2S)
- Assumptions/dependencies: access to deployment environment; cooperation with platforms for watermark detection
Long-Term Applications
These require additional research, task-specific data, or broader deployment infrastructure, but are directly suggested by the model’s native-streaming, low-latency, higher-resolution design.
- Real-time sign-language generation and interpretation by mid-shot agents
- Sectors: accessibility, education, public services
- Tools/products: sign-language TTS (speech-to-sign) with handshape fidelity; dual-mode caption + avatar
- Dependencies: domain datasets (gloss-to-video), rigorous evaluation with Deaf communities, hand-tracking accuracy, compliance with accessibility standards
- On-device or edge-only deployment for privacy-critical settings
- Sectors: healthcare, finance, defense
- Tools/products: quantized thinker + compact performer variants; tensor-parallel performers on edge clusters; adaptive resolution/framerate
- Dependencies: hardware accelerators with sufficient HBM; optimized collectives on edge fabrics; power/thermal envelopes
- Embodied robotics telepresence with a co-located avatar guiding tasks
- Sectors: robotics, manufacturing, field service, energy/utilities
- Tools/products: robot HUD with agent overlay; synchronized pointing/gesture to indicate parts/tools; safety-aware timing
- Dependencies: robust perception-to-language-to-action grounding; safety certification; low-jitter industrial networks
- Cross-lingual live dubbing with lip-synced avatars preserving speaker identity
- Sectors: media localization, conferencing, education
- Tools/products: voice conversion + prosody transfer; identity-preserving video avatar; latency-aware caption alignment
- Dependencies: robust voice cloning consent frameworks; bias/fairness evaluation; international privacy laws
- Multi-party conversational scenes with turn-taking and body layout awareness
- Sectors: social platforms, virtual events
- Tools/products: multi-agent coordination of gaze and speaking order; scene memory retrieval for continuity
- Dependencies: multi-person training data, dialogue management for crosstalk, scalable KV memory
- Safety-critical coaching (e.g., clinical rehabilitation, lab safety) with verified motion cues
- Sectors: healthcare, industrial training
- Tools/products: validated gesture libraries; progress assessment via vision feedback; audit logs
- Dependencies: clinical trials or occupational safety validation; liability frameworks; fail-safe fallbacks
- Interactive broadcast news/presenting with on-the-fly graphics and deictic gestures
- Sectors: broadcast, sports, finance
- Tools/products: NLE/control-room integration; real-time chart overlays referenced by the avatar; compliance subtitle pipelines
- Dependencies: studio-grade reliability; standards for synthetic anchors; watermarking
- Privacy-preserving “synthetic face” mode for human callers
- Sectors: telework, telehealth, education
- Tools/products: real-time face redaction replaced by user-approved avatar; preserved prosody and timing
- Dependencies: calibrated identity transfer; consent UI; adversarial testing against re-identification
- Task-grounded agents that manipulate nearby physical/digital objects in AR
- Sectors: enterprise AR, field service
- Tools/products: AR glasses integration; hand/object annotations; scene-consistent memory retrieval
- Dependencies: SLAM-quality tracking; robust object recognition; safety rules for instructions
- Federated serving with KV-slice privacy guarantees
- Sectors: AI infrastructure, regulated industries
- Tools/products: secure KV-slice serialization; differential privacy at the thinker–performer boundary; encrypted collectives
- Dependencies: cryptographic protocols for low-latency settings; regulatory acceptance
- Standardization of latency metrics and disclosure for conversational AV systems
- Sectors: policy, industry consortia
- Tools/products: open benchmarks for “model-side S2S” vs. “first-frame” vs. “product response time”; certification labels
- Dependencies: multi-stakeholder agreement; measurement tooling; compliance audits
- Generalization of the thinker–performer pattern to other long-sequence generators
- Sectors: music generation, code copilots with video context, simulation
- Tools/products: reusable serving blueprints; per-modality sharding strategies (shard long sequences, avoid sharding short ones)
- Dependencies: modality-specific training/serving research; cost models; scheduler co-design
Notes on feasibility across applications
- Core strengths to leverage: native full-duplex audio-visual timeline; ~200 ms model-side latency at 25 FPS; mid-shot legibility (hands, gaze, posture); thinker–performer serving that isolates heavy video denoising.
- Common dependencies: stable 350 ms bidirectional network budget; sufficient multi-GPU capacity with high-bandwidth interconnects (NCCL all-to-all/gather); streaming codecs and WebRTC/WebTransport; safety, disclosure, and watermarking for synthetic media; domain-specific data for regulated/skill-specific tasks.
- Risk/assumption flags: qualitative visual evaluation (not clinical-grade); compute and bandwidth costs at scale; need for robust fallbacks (audio-only, close-up mode) under load or weak networks.
Glossary
- all-to-all/gather: A collective communication pattern where every device exchanges tensor shards with every other device, often used to parallelize attention over sequence context. "and communicate through Ulysses all-to-all/gather collectives around attention."
- bidirectional network budget: The assumed round-trip time allocation for client-server networking when reporting end-to-end interaction latency. "With a 350 ms bidirectional network budget, the resulting remote interaction latency remains approximately 550 ms."
- block-causal attention: An attention scheme that enforces causality in blocks so each output depends only on appropriate past inputs across modalities. "Wan-Streamer remains one native-streaming model: language, audio, and video inputs and outputs are represented on a shared causal timeline and coordinated by block-causal attention."
- causal decoders: Decoding modules that generate outputs sequentially, conditioned only on past context, to maintain causality during streaming. "and the causal decoders that turn returned latents into output audio and video."
- causal timeline: A unified temporal representation where all inputs and outputs are aligned so that each step depends only on prior history. "user and agent text, audio, and video are represented on one causal timeline and modeled by a single Transformer."
- context-parallel: A parallelization strategy that splits a long sequence’s context across multiple devices to accelerate attention and generation. "A Ulysses-style context-parallel GPU group hosts the expensive flow-matching latent generation path."
- denoising: The iterative process (from diffusion/flow-based generation) of removing noise from latent variables to synthesize clean outputs. "The long high-resolution latent video sequence is split across ranks for denoising and gathered through Ulysses communication"
- flow-matching: A generative modeling approach that trains a continuous-time flow to transform noise into data, enabling efficient sampling. "A Ulysses-style context-parallel GPU group hosts the expensive flow-matching latent generation path."
- full-duplex: Real-time, two-way communication where speaking and listening occur simultaneously without enforced turn-taking. "Real-time audio-visual interaction sits at the intersection of full-duplex spoken dialogue, multimodal perception, streaming video generation, and interactive digital humans."
- K/V cache: Stored transformer key/value tensors that allow fast incremental attention computation during streaming generation. "The language/state path is reflected in the K/V cache that conditions generation."
- latent video sequence: A compressed, learned representation of video frames used as the target space for generative denoising. "The long high-resolution latent video sequence is split across ranks for denoising and gathered through Ulysses communication"
- native-streaming: A training and serving paradigm where the model processes and produces outputs continuously on a single causal stream rather than in disjoint stages. "Wan-Streamer v0.1 established a native-streaming formulation for this setting:"
- performer: The multi-GPU component dedicated to heavy latent generation, parallelized across sequence context. "Performer. A Ulysses-style context-parallel GPU group hosts the expensive flow-matching latent generation path."
- pre-sharded: Pre-partitioned across devices so that incoming data (e.g., K/V tensors) can be placed directly into local shards without additional splitting. "Each performer rank writes incoming K/V into a pre-sharded local cache."
- sequence parallelism: Distributing different segments of a long sequence across devices to accelerate attention and generation workloads. "through pre-sharded K/V caches and sequence parallelism for the high-resolution latent video denoising path."
- sequence sharding: Dividing a sequence into shards across devices; useful for long sequences but sometimes avoided for short ones to reduce overhead. "so they are generated without sequence sharding."
- signal-to-signal latency: The time from when an input signal becomes available to when the corresponding output signal is ready for emission. "while preserving approximately 200 ms model-side signal-to-signal latency at 25 FPS."
- streaming unit: A fixed-duration chunk of the continuous stream processed per iteration in the real-time pipeline. "At streaming unit , the thinker consumes the current user observations and produces a new performer-compatible K/V slice."
- thinker: The single-GPU, low-latency path that handles perception, language/state updates, K/V construction, and final decoding. "Thinker. A single GPU hosts the causal audio/video encoders, the token-causal Transformer path for language and state update, KV-cache construction, and the causal decoders that turn returned latents into output audio and video."
- thinker-performer boundary: The interface separating the low-latency control path from the heavy-generation path, minimizing data exchanged between them. "while preserving the compact thinker-performer boundary"
- token-causal Transformer: A Transformer configured so each token attends only to earlier tokens, enabling incremental streaming updates. "the token-causal Transformer path for language and state update"
- Ulysses-style context parallelism: A DeepSpeed-Ulysses technique that parallelizes attention by splitting context across GPUs with collective communication. "The performer uses Ulysses-style context parallelism for the 640368 latent generation path"
Collections
Sign up for free to add this paper to one or more collections.