Physical Object Understanding with a Physically Controllable World Model

Sign up for free to create and run prompts on this paper using GPT-5.

What is this paper about?

This paper is about teaching a computer to “understand” the physical world in videos: where objects are, how they move, how their parts are connected, and how they affect each other. The authors build a single, unified “world model” that can answer questions like “if I poke this part, what else moves?” and “what will this scene look like a moment later?”—without needing human-labeled training data.

What questions were the researchers asking?

• Can we build one model that can predict many things about a scene—appearance, motion, and camera changes—and let us condition on any of them? (Example: predict future motion from a single image, or predict how a future image will look if we know the motion.)
• Can such a model discover object-like regions on its own, just by noticing which pixels tend to move together?
• Can it find parts that move independently (like a laptop lid vs. base)?
• Can it use that knowledge to edit images in realistic 3D ways (move an object around) and reason about how objects support each other (like a stack of blocks)?

How did they build their model? (Methods in simple terms)

Think of a video frame as a grid of small patches. For each patch, you can store:

• its appearance (color/texture),
• how it moves between frames (a small arrow called optical flow),
• and how the camera moved.

The model turns each patch into simple “tokens,” a bit like letters in a sentence:

• A pointer token says “where” (which patch in space and time).
• A content token says “what” (color token, motion token, or camera token).

Then they train a big GPT-like “autocompleter” on millions of short video clips:

• It sees mixed sequences of these tokens and learns to predict the next token.
• Because the sequence can include any mix of “where” and “what,” the same model learns to answer many kinds of questions. For example:
  • Give it an image and ask for motion tokens: it predicts how things might move.
  • Give it motion tokens and ask for the next image’s appearance: it predicts what the scene will look like after moving.
  • Give it a “camera stop” token: it focuses on object motion instead of camera motion.
  • Give it a “poke” (a few motion tokens at one spot): it completes a full, physically plausible motion field.

Key ideas explained simply:

• Probabilistic: Instead of one answer, the model gives several plausible futures, each with a likelihood. That’s good because the real world is uncertain.
• Optical flow: Tiny arrows that say how each pixel moves between two frames.
• Sequential vs. parallel generation:
  • Sequential is like writing a story one word at a time, using what you just wrote to decide the next word—higher quality, slower.
  • Parallel is like filling many blanks at once—faster, but it can miss some dependencies.

What did they find?

• The model predicts realistic future motions and appearances
  • From a single image, it generates multiple believable ways things could move and look a moment later.
  • If you “poke” one spot, it predicts how the rest reacts in a physically consistent way.
• It discovers objects by how they move together
  • By sampling several futures and checking which pixels tend to move in sync, the model groups pixels into object-like segments.
  • On a benchmark called SpelkeBench (which tests finding “movable objects”), it reaches state-of-the-art performance, beating both self-supervised and some supervised baselines.
• It finds articulated parts (subparts that move independently)
  • If you poke a subpart (like a laptop lid), the model can separate that moving part from the rest.
  • It performs best on DragAMove, a test for discovering such articulated parts.
• It enables better 3D object manipulation in images
  • After finding the object, you can specify a 3D move (like rotate/translate), and the model generates a realistic edited image showing the result.
  • On 3DEditBench (which measures edit quality and accuracy), the model achieves state-of-the-art results.
  • Using these physically grounded segments beats using off-the-shelf segmentation masks, leading to more believable edits.
• It reasons about object relationships (like support)
  • When you “poke” a block in a stack, the model predicts movement not only of that block but also of blocks it supports.
  • It builds a “support graph” that helps with tasks like Visual Jenga: choosing pieces that can be removed with minimal disturbance.

Why these results matter:

• One model, many abilities: predicting motion, generating future frames, discovering objects and parts, 3D editing, and physical reasoning—without task-specific training.
• Physical coherence: It’s not just about what looks like an object; it’s about what moves together and how forces travel through a scene.

Why does this matter?

• For robotics: A robot could “imagine” the results of poking or moving something before acting, helping it plan safer, smarter moves.
• For video editing and AR: More realistic object edits and animations, because the model understands what belongs together and how it should move in 3D.
• For general AI: Shows how a GPT-style model of videos can learn physical structure and cause-and-effect by predicting possibilities, not just single outcomes.
• Beyond everyday scenes: The same idea—discover structure by predicting and probing—could help in areas like medical imaging or materials science, where “objects” and their interactions are less obvious.

In short, the paper introduces a unified, controllable world model that learns from raw video to predict what might happen next and to reveal the hidden structure of objects and their relationships—laying a foundation for more physically aware intelligent systems.

Below is a consolidated list of concrete knowledge gaps, limitations, and open questions that the paper leaves unresolved. Each point is phrased to suggest actionable next steps for future work.

• Missing explicit 3D geometry: The model operates over RGB and 2D flow tokens without explicit depth, normals, or 3D scene geometry; evaluate adding geometry tokens (e.g., depth, surface normals, NeRF/tri-plane features) and test on occlusion-heavy, multi-view consistency benchmarks.
• Limited temporal horizon: Training sequences appear to use two frames (r0, flow, r1), leaving long-horizon dynamics unaddressed; extend to multi-frame rollouts and quantify stability, error accumulation, and physical consistency over longer horizons.
• Flow as proxy for action: “Virtual pokes” via sparse flow tokens are only correlative proxies for forces/actions; collect and evaluate on action-conditioned datasets with measured contacts/forces to test causal fidelity and close the loop to robotics.
• Quantizer discretization effects: RGB/flow quantization and binned camera tokens may cap fidelity for small motions and fine camera changes; systematically study codebook size, multi-scale quantization, and continuous camera parameterizations, as well as their impact on accuracy and calibration.
• Probability calibration: The paper uses token probabilities to compute motion statistics but does not assess calibration; evaluate ECE/Brier scores, apply temperature scaling or calibration methods, and test robustness under distribution shift.
• Consistency across serializations: Theoretical equivalence between PGMs and autoregressive modeling is asserted but not validated empirically; test whether conditional predictions are consistent across different pointer orderings and prove or diagnose order-invariance violations.
• Parallel inference independence assumption: Parallel predictions assume conditional independence across undecoded locations; quantify the quality-efficiency trade-off, identify failure modes (e.g., articulated bodies), and develop structured parallelization (e.g., blockwise/graph-aware decoding).
• Camera motion handling: Camera tokens are binned 6-DoF and “camera stop” conditioning is central, but sensitivity to camera discretization and errors in camera estimation is not studied; evaluate continuous camera models and performance with non-zero and time-varying camera motion.
• Occlusions and disocclusions: 3D edits require hallucinating newly revealed surfaces; explicitly measure occlusion handling, disocclusion inpainting quality, and temporal coherence under large edits or object motion.
• Physical parameter inference: The model does not infer mass, friction, or stiffness; investigate whether such latent physical properties can be learned or estimated from distributions over futures and used to improve interventional predictions.
• Structural causal validity: Claims of physically controllable inference are not matched with causal identifiability tests; design counterfactual benchmarks and randomized interventions to evaluate whether learned dependencies are causal rather than correlational.
• Kinematic structure discovery: Articulated part discovery is demonstrated but hinge axes, joint limits, and kinematic graphs are not extracted; develop procedures to recover explicit part-joint graphs and evaluate against ground-truth kinematic annotations.
• Multi-object interaction fidelity: Collisions, stacking stability, and contact dynamics are not quantitatively validated; introduce benchmarks with ground-truth support/collision graphs and dynamics (e.g., block towers with labeled supports) and measure prediction fidelity.
• Visual Jenga evaluation: The “support graph” and motion influence score are illustrated but only qualitatively; provide quantitative metrics and datasets with annotated support relationships to validate correctness and reliability.
• Domain generalization: The model is trained on natural videos but not tested in domains like medical imaging or astrophysics as suggested; perform cross-domain evaluations and study transferability, fine-tuning strategies, and failure modes.
• Data provenance and bias: The 3M video corpus and its distribution are not described; document sources, licensing, and demographic/content biases, and study their effects on learned physical priors and object discovery.
• Dependence on external flow quality: Flow tokens likely originate from an external estimator (details relegated to supplement); measure sensitivity to flow noise, compare to self-supervised motion learning, and train with robustness objectives.
• Runtime and scalability: Sequential sampling with multiple pokes (e.g., N=8) and 64+ autoregressive steps per sample is compute-heavy; report end-to-end latency, profile bottlenecks, and explore accelerations (e.g., speculative decoding, distillation, masked parallel decoding).
• Scaling laws and model size: Scaling benefits are reported up to 7B parameters without systematic data/compute scaling analysis; derive scaling laws (data, parameters, tokens) and identify diminishing returns and optimal regimes.
• Pointer discretization and resolution: Spatial pointers discretize coordinates and may limit scale/rotation invariance and boundary precision; evaluate multi-resolution grids, learned continuous coordinate encodings, and super-resolution decoders for sharper boundaries.
• Static-object discovery: Motion-correlation segments prioritize movable entities and may miss static but semantically coherent objects; explore integrating appearance/shape priors, affordance cues, or language to discover non-moving objects.
• Robustness to complex materials: Performance on transparent, reflective, thin, deformable, and fluid objects is not analyzed; design targeted benchmarks and augment token vocabularies or training curricula to handle such materials.
• Hyperparameter sensitivity: Segment extraction depends on thresholds (e.g., flow magnitude for “motion tokens,” correlation thresholds) and number of pokes/seeds; provide sensitivity analyses beyond the shown ablations and automated selection strategies.
• Order of modalities and conditioning: Although multiple inference pathways are possible, the coverage of cross-modal conditionals is not systematically tested (e.g., predicting depth from flow or vice versa); enumerate and evaluate a comprehensive set of conditional queries.
• Edit adherence vs. realism trade-offs: 3D editing is evaluated with LPIPS/SSIM/EA but not user studies or perceptual realism under large edits; conduct human evaluations and analyze failure cases (ghosting, texture stretch, identity drift).
• Persistent object identity: The framework does not enforce or evaluate consistent object IDs across rollouts; add identity-tracking objectives and assess temporal consistency in discovered objects and parts.
• Learning with continuous actions: Current “poor man’s actions” rely on discrete tokens (flow/camera); investigate continuous control tokens aligned with physical action spaces and study how this affects interventional fidelity.
• Long-range dependencies in clutter: Articulated and multi-object scenes with long-range constraints (e.g., ropes, chains) are challenging; construct benchmarks and incorporate structured attention or graph priors to capture non-local dependencies.
• Formal uncertainty decomposition: Predicted distributions blend aleatoric and epistemic uncertainty; quantify and disentangle them (ensembles, MC dropout) and study how uncertainty quality affects downstream selection (e.g., Visual Jenga choices).
• Reproducibility of training details: Critical choices (flow estimator, quantizer architecture, token vocab sizes, binning schemes) are deferred to the supplement; provide exhaustive specifications and release training/preprocessing code to ensure reproducibility.
• Integration with language and task goals: The model is not conditioned on text or high-level goals; explore language-conditioned interventions and evaluate instruction-following for physically grounded editing and manipulation.

Immediate Applications

Below are concrete ways the paper’s findings and methods can be applied today, along with likely sectors, product/workflow ideas, and key dependencies to consider.

• Sector: software/media (VFX, image/video editing), creator tools
  • Application: physically coherent object masks for editing
  • What it enables: extract “movable object” masks that align with what moves together (better than appearance-only masks), improving cutouts, warps, and 3D-like edits in single images
  • Potential tools/products/workflows: Photoshop/After Effects/DaVinci plug‑ins for “Movable Mask,” “Articulation Mask,” and “Flow‑conditioned Warp”; batch processing pipeline for studios
  • Assumptions/dependencies: GPU inference for a 7B AR transformer; good generalization to target content; access to flow/RGB quantizers; handling of camera motion token (or assume camera-stop)
• Sector: software/media and AR
  • Application: single-image object manipulation with edit adherence
  • What it enables: 6DOF transformation of selected objects via dense flow synthesis and re-rendering, with improved edit adherence (EA) and SSIM/LPIPS
  • Potential tools/products/workflows: “3D Edit from 1 Image” SDK; mobile photo editor feature “Move/Rotate this object”; AR preview of rearrangements (e.g., furniture)
  • Assumptions/dependencies: model’s 2D flow-based edits approximate 3D; occlusion/lighting not fully physically simulated; OOD scenes may degrade
• Sector: robotics (R&D, lab prototyping), automation
  • Application: movability heatmaps and expected-motion fields for poke/grasps
  • What it enables: choose where to interact; identify movable vs static regions; predict motion field from a virtual poke; reduce trial-and-error in manipulation
  • Potential tools/products/workflows: “Poke Planner” module; grasp point selection from motion probability; non-prehensile manipulation planning; active-data collection with virtual pokes
  • Assumptions/dependencies: camera-motion compensation; domain shift from internet videos to robot’s camera; not a dynamics model of robot forces; latency on embedded hardware
• Sector: warehouse/logistics, manufacturing
  • Application: support graph extraction and “Visual Jenga” unstacking planning
  • What it enables: identify which items support others; pick items least likely to destabilize a stack; plan safe unstacking/re-stacking sequences
  • Potential tools/products/workflows: “Support Graph Extractor” and “Low-Influence Picker” in bin-picking/palletizing software
  • Assumptions/dependencies: uses probabilistic correlations from visual world model, not true mass/friction estimation; requires reliable segmentation of items; camera viewpoint effects
• Sector: e-commerce imagery, marketing
  • Application: physically grounded product cutouts and pose variations from a single photo
  • What it enables: more plausible repositioning/re-posing of products; background edits with fewer artifacts; cohesive masks for multi-instance categories
  • Potential tools/products/workflows: “Product Re-poser” service; ad creative generation pipelines; catalog retouch tooling
  • Assumptions/dependencies: generalization to product categories; artifact control on high-frequency textures; brand safety approval for edits
• Sector: academia (computer vision, robotics, cognition), benchmarking
  • Application: unified probing for objecthood and articulation without labels
  • What it enables: evaluate zero-shot object discovery; measure causal/structural dependencies via motion correlation; create new benchmarks and datasets
  • Potential tools/products/workflows: open-source “Movability Heatmap” and “Articulated Part Finder” scripts; dataset bootstrap for segmentation with point prompts
  • Assumptions/dependencies: reproducible inference settings; access to code and quantizers; compute budgets for sequential sampling
• Sector: safety engineering (labs, testbeds)
  • Application: virtual poke risk pre-checks
  • What it enables: anticipate cascades (what else moves if I move this?); pre-emptively identify fragile configurations
  • Potential tools/products/workflows: “Motion Influence Score” dashboard for lab staging; pre-task checks for human-robot collaboration
  • Assumptions/dependencies: correlations are not guarantees of real-world dynamics; must be validated on in-domain data
• Sector: developer tooling, cloud AI
  • Application: world-model inference endpoints
  • What it enables: APIs for motion probability maps, expected motion, sequential motion sampling, support graph estimation, flow-conditioned RGB rendering
  • Potential tools/products/workflows: “World Model API” with endpoints: /movability, /expected_flow, /sequential_rollout, /support_graph, /render_from_flow
  • Assumptions/dependencies: serving costs and latency for 7B model; license/rights for training data; monitoring for misuse (edits that could mislead)

Long-Term Applications

These use cases are plausible but require additional research, domain adaptation, scaling, or systems integration before dependable deployment.

• Sector: robotics (field and household)
  • Application: real-time, on-robot physically controllable world model for manipulation
  • What it enables: closed-loop planning with uncertainty; selecting actions from model’s “what-if” rollouts; safe multi-object manipulation and tidying
  • Potential tools/products/workflows: “Uncertainty-Aware Manipulation Stack” integrating virtual pokes with grasp planners and tactile feedback
  • Assumptions/dependencies: substantial model compression/acceleration; integration with control and state estimation; camera intrinsics/extrinsics calibration; robustness to clutter and lighting
• Sector: autonomous driving and mobile perception
  • Application: local-variable probabilistic prediction of agent dynamics and scene affordances
  • What it enables: uncertainty-aware forecasting in crowded scenes; identifying movable hazards vs static structures
  • Potential tools/products/workflows: “Affordance Heatmaps” for planning; “Probabilistic Object Dependency Maps”
  • Assumptions/dependencies: domain-specific training (dashcam, LiDAR/depth tokens); safety certification; tight latency constraints
• Sector: digital twins, simulation, and planning
  • Application: sim-less “what-if” reasoning layer for virtual environments
  • What it enables: predicting plausible outcomes of interventions without constructing full physics simulators; stress-testing stacks/shelves/layouts visually
  • Potential tools/products/workflows: “Visual What-If Engine” embedded in CAD/BIM/digital-twin platforms; rapid design iteration
  • Assumptions/dependencies: groundedness to real domain; hybridization with physics engines for guarantees; support for multi-view geometry and depth tokens
• Sector: industrial automation and palletizing
  • Application: robust stacking/unstacking optimization with learned support/influence graphs
  • What it enables: safer, faster pallet operations; fewer collapses; automated reconfiguration planning
  • Potential tools/products/workflows: “Stack Stability Planner” co-optimizing order and robot kinematics
  • Assumptions/dependencies: incorporation of weights, friction, compliance; multi-view sensing; regulatory safety standards
• Sector: healthcare and scientific imaging
  • Application: data-driven structure discovery in modalities beyond natural images (e.g., medical imaging, microscopy, astrophysics)
  • What it enables: unsupervised grouping of functional substructures; probing causal/structural relations from observations
  • Potential tools/products/workflows: “Probabilistic Structure Integrator” variants with modality-specific tokens (e.g., MR sequences, spectral bands); research tools for hypothesis generation
  • Assumptions/dependencies: domain shift is large; clinical/safety validation; privacy and data governance; token design for non-RGB modalities
• Sector: AR glasses and on-device perception
  • Application: real-time movability and articulation overlays in the user’s view
  • What it enables: assistive guidance (“this hinge moves”), home re-arrangement previews, educational overlays
  • Potential tools/products/workflows: “Physical Insight HUD”
  • Assumptions/dependencies: edge inference acceleration; energy constraints; privacy-preserving on-device models
• Sector: policy, standards, and governance
  • Application: guidelines for physically plausible image/video edits and disclosure
  • What it enables: labeling/metadata standards for flow-conditioned edits; audit trails for professional/advertising use; safety pre-checks in robotics
  • Potential tools/products/workflows: watermarking and provenance for edits; conformance tests for uncertainty quantification and OOD detection
  • Assumptions/dependencies: multi-stakeholder consensus; regulatory alignment (consumer protection, workplace safety); robust misuse detection
• Sector: education and outreach
  • Application: interactive physics and perception sandbox
  • What it enables: students explore “poke-and-see” consequences and articulation; understand uncertainty and objecthood
  • Potential tools/products/workflows: browser-based simulators powered by a distilled model; curriculum modules
  • Assumptions/dependencies: cost-effective serving; age-appropriate safety constraints
• Sector: enterprise AI platforms
  • Application: turnkey “world-model as a service” for verticals
  • What it enables: standardized endpoints for object discovery, articulation, motion prediction, and support reasoning integrated into enterprise workflows
  • Potential tools/products/workflows: sector-specific adapters (retail, manufacturing, construction) and deployment recipes
  • Assumptions/dependencies: SLAs on latency/accuracy; security/compliance; cost optimization

Cross-cutting assumptions and dependencies

• Data and domain fit: performance depends on similarity between training videos and deployment scenes; domain adaptation or fine-tuning may be required.
• Camera modeling: “camera stop” conditioning assumes correct handling of viewpoint changes; calibration or reliable camera-motion estimates improve results.
• Computational costs: 7B-parameter autoregressive inference can be slow; parallel querying trades quality for speed; quantization/distillation/ONNX/TensorRT may be needed.
• Physical fidelity limits: the model infers correlations and plausible motions via flow; it is not a full physics engine (no mass/friction/force guarantees).
• Safety/ethics: flow-conditioned edits can convincingly alter reality; provenance/watermarking and usage policies should be in place.
• Licensing and governance: ensure rights for training data; comply with privacy and IP standards in deployment.

• 3DEditBench: A benchmark for evaluating 3D object editing/manipulation quality in images. Example: "achieving state-of-the-art results on 3DEditBench~\cite{lee20253d}"
• 6DOF transforms: Six degrees-of-freedom transformations (3D rotations and translations) used to describe camera/object motion. Example: "the camera tokens are obtained by binning 6DOF transforms."
• ablation studies: Controlled experiments that systematically vary components or hyperparameters to assess their impact. Example: "Ablation studies"
• action-conditioned predictors: World models that predict future states conditioned on agent actions. Example: "classical world models are typically defined as action-conditioned predictors~\cite{ha2018world}"
• autoregressive framework: A modeling approach that predicts each variable/token conditioned on previously generated ones. Example: "the standard autoregressive framework"
• autoregressive sequence modeling: Training sequence models by predicting the next token given the previous ones. Example: "trained efficiently with autoregressive sequence modeling"
• Average Precision (AP): A detection/segmentation metric that summarizes precision across thresholds. Example: "Average Precision (AP) measures the fraction of predicted segments that end up being matched and detected"
• Average Recall (AR): A detection/segmentation metric that summarizes recall across thresholds. Example: "We also report Average Recall (AR) which measures detection accuracy as the fraction of segments with $\text{IoU} \geq \tau \in [0.5, 0.99]$."
• binning: Discretizing continuous values into discrete bins for tokenization. Example: "the camera tokens are obtained by binning 6DOF transforms."
• camera stop conditioning: Conditioning the model on zero camera motion to isolate object dynamics. Example: "with camera stop conditioning."
• camera tokens: Discrete tokens that represent camera/viewpoint changes between frames. Example: "camera tokens that encode viewpoint changes between frames."
• conditional independence: An assumption that variables are independent given some conditions; used to justify parallel decoding. Example: "Parallel generation assumes conditional independence between undecoded locations"
• cross entropy loss: A standard loss function for training classifiers and next-token predictors. Example: "train with next-token prediction cross entropy loss"
• directed graph: A graph with directed edges used to model influence/causal relations among objects. Example: "We construct a directed graph $G$"
• Edit Adherence (EA): A metric that measures how accurately the applied edit matches the intended geometric transformation. Example: "include the Edit Adherence (EA) metric introduced in prior work"
• expected motion maps: Maps of probability-weighted average motion vectors across the image. Example: "Expected motion maps."
• flow tokens: Discrete tokens representing optical flow/dynamics between frames. Example: "flow tokens that encode dynamics"
• global scene embeddings: Holistic learned representations of entire scenes used for conditioning/generation. Example: "global scene embeddings~\cite{rombach2022high, agarwal2025cosmos}"
• GPT-style next token prediction: A modeling paradigm that predicts the next token in a sequence as in GPT. Example: "a GPT style next token prediction sequence model"
• instruction-conditioned generations: Generative outputs conditioned on textual or high-level instructions. Example: "performing instruction-conditioned generations~\cite{lin2024video,agarwal2025cosmos}"
• LPIPS: Learned Perceptual Image Patch Similarity, a perceptual metric for image similarity. Example: "PSNR, SSIM, and LPIPS"
• mean Intersection over Union (mIoU): A segmentation metric averaging IoU across classes or instances. Example: "measure segment boundary precision using mean intersection over union (mIoU)."
• motion-correlation analysis: Grouping pixels into objects by correlating their motion responses to interventions. Example: "using motion-correlation analysis to group pixels that move together"
• motion influence score: A score quantifying how much a perturbed object causes other objects to move. Example: "we compute its motion influence score, $$\mathbb{I} [O_i] = \frac{1}{N-1} \sum_{j \neq i} w_{ij}$$"
• movability score: A score estimating how freely an object can be moved without disturbing others. Example: "movability score, quantifying how freely an object can move without disturbing others."
• non-maximum suppression: A procedure to remove redundant detections/segments by suppressing similar ones. Example: "via non-maximum suppression"
• optical flow: The apparent pixel-level motion between frames in a video. Example: "optical flow can be generated as an intermediate representation"
• parallel generation: Decoding multiple tokens independently in parallel, trading quality for speed. Example: "Parallel generation assumes conditional independence between undecoded locations, trading off efficiency for quality."
• pointer tokens: Tokens that index specific spatiotemporal locations to be predicted/decoded. Example: "The use of pointer tokens enables sequences to be constructed in arbitrary spatiotemporal order."
• pointer–content representation: A sequence format interleaving location (pointer) and value (content) tokens. Example: "pointer–content representation of visual data."
• Probabilistic Graphical Models (PGMs): Structured probabilistic models encoding conditional dependencies among variables. Example: "Probabilistic Graphical Models (PGMs)"
• probabilistic world model: A model that represents and predicts distributions over scene variables and their dependencies. Example: "we construct a probabilistic world model"
• quantizers: Modules that discretize continuous signals (e.g., patches/flows) into token vocabularies. Example: "training shallow convolutional quantizers"
• RGB tokens: Discrete tokens representing appearance/texture content in image patches. Example: "RGB tokens that encode appearance"
• sequential inference: Inference by sampling tokens one-by-one so each prediction conditions on previous ones. Example: "through sequential inference."
• sequential sampling: Token-by-token generation in sequence models for higher fidelity and dependency capture. Example: "uses the standard sequential sampling used in sequence models like LLMs."
• spatiotemporal pointer locations: Discrete indices specifying spatial and temporal positions for token decoding. Example: "spatiotemporal pointer locations $\mathcal{P}$"
• support graph: A graph capturing which objects physically support others. Example: "providing a direct estimate of the underlying support graph."
• SSIM: Structural Similarity Index, a perceptual metric for image quality. Example: "PSNR, SSIM, and LPIPS"
• Warmup-Stable-Decay schedule: A learning-rate schedule with warmup, plateau, and decay phases. Example: "Warmup-Stable-Decay schedule~\cite{wen2024understanding}"
• zero-shot: Performing tasks without task-specific training or fine-tuning. Example: "in a zero-shot way"

Sign up for free to add this paper to one or more collections.