Frontier black-box Large Language Model (LLM) agents are proficient at long-horizon reasoning and precise action generation. However, when it comes to a specific agentic task, they have no internalized knowledge of how that particular environment unfolds. A world model offers a natural remedy, providing the agent with environment specific knowledge by foreseeing how the environment will respond to the action it is about to take. Prior world models for LLM agents mainly predict next observations at each step, which is costly and often uninformative because individual observations provide limited task-relevant guidance and not every step requires call for foresight. This raises a key question: what a world model should predict and when should it be invoked for agentic tasks? Our pilot study suggests that world model guidance is most useful when it anticipates pitfalls within a bounded multi-step future. Motivated by this, we introduce Pitfall Anticipation World Model (PAWM), which selectively intervenes when a potential pitfall arises and predicts the impending failure to steer the agent away from it. PAWM is trained in two stages: Supervised Fine-Tuning (SFT) on synthetic data labeled with agent pitfalls, followed by GRPO co-training with a black-box LLM to further optimize it in the target environment. PAWM improves Claude Sonnet 4.5 performance by 24.3% on ScienceWorld and 7.7% on WebArena-Lite, outperforming a range of baselines. These results demonstrate that PAWM provides an effective form of world model guidance for black-box LLM agents.
Neural Radiance Fields (NeRF) synthesize photorealistic novel views from dense image collections, but degrade sharply when few views are available. We attribute a specific, measurable defect to the objective itself: the rendering loss is a sum over independent rays and contains no term comparing densities that different rays assign to the same region of space, so nearby 3D points can settle on contradictory densities while every training pixel is reproduced correctly. We make this quantity explicit by measuring cross-ray density disagreement directly during training, and show that it stays high in vanilla NeRF, falls only incidentally under frequency-based regularization, and moves together with held-out quality in the sparse regime. We then close the gap with DriftNeRF, a drop-in regularizer that instantiates the drifting-field template on density samples: a masked exponential kernel aggregates densities from nearby points on other rays into a stop-gradient consensus target, and the resulting drifting field penalizes unsupported disagreement while leaving agreed-upon boundaries intact. The regularizer reads only sampled positions, ray identifiers, and predicted densities, so it requires no pretrained model, external geometry, or additional images, and transfers across field parameterizations. On NeRF-Synthetic it reduces residual disagreement by an order of magnitude, and across NeRF-Synthetic, DTU, and LLFF it improves 3-view PSNR over the strongest MLP-NeRF baseline by 0.42-0.65 dB and remains competitive with sparse-view Gaussian pipelines under a matched protocol without their depth, flow, or covisibility preprocessing.
Abdominal organ segmentation is essential in modern medical applications. While existing deep learning approaches demonstrate promising performance, their heavy reliance on large-scale image data raises concerns over generalization capability and privacy risks. This motivates the development of generalizable segmentation training paradigms based on domain-randomized image synthesis, facilitating generalization across unseen real image domains. However, existing paradigms either suffer from limited segmentation performance when applied to abdominal organs, or require dataset-specific finetuning. To address these limitations, we propose ReSeg, a novel two-stage retraining-free framework for domain-agnostic zero-shot abdominal organ segmentation. At the first stage, our organ-agnostic synthetic data-driven pretraining approach exploits semantically and visually diverse synthetic images generated by a dedicated data engine to optimize a Generalizable Semantic Representation Network (GSRNet). The network is thereby capable of learning semantic features robust against variations in anatomical structure, imaging contrast, resolution and spatial context, which provide universal representation capabilities that serve as a foundation for segmentation learning. Subsequently, in the second stage, we continue to train an Organ-specific Encoder-decoder Segmentation Network (OESNet), whose encoder is initialized with the pretrained weights. This training process is facilitated by another dedicated data engine, which generates diversified visual images while strictly retaining the inherent spatial layout of abdominal organs. Comprehensive evaluation on seven abdominal organ segmentation datasets across three different modalities including CT, MRI, and PET demonstrates that ReSeg consistently outperforms existing state-of-the-art (SOTA) methods while eliminating the need for real training images.
Amodal video segmentation asks for the complete mask of an object in every frame, including the parts hidden behind other objects. The best current methods generate these masks with fine-tuned video diffusion models, which need many denoising steps and an extra depth network, and give a different answer for every sample. Self-supervised video world models of the JEPA family are trained on a closely related task: predicting the representation of masked space-time regions from the visible ones. We therefore treat occlusion as masking. The tokens that the object's visible mask does not cover are dropped, the pre-trained V-JEPA 2.1 predictor fills them in, and a light head decodes the observed and predicted features into amodal masks. The resulting model, OccJEPA (Occlusion-as-masking JEPA), is deterministic and 33.5x faster than the diffusion-based state of the art, and more accurate on SAIL-VOS, on TAO-Amodal, including a held-out test split (+4.00 AP50) and visible masks tracked by SAM2, and on Kubric. Controlled experiments trace the gain to the predictor's pre-trained weights rather than to its capacity, for world models from 87M to 1.85B parameters. Code is available at https://github.com/zgp110/occjepa.
We introduce HumanCentric-Scenes-AI, a human-curated dataset of 296 AI-generated stimuli that depict people across four contexts in which people encounter AI-generated media. The Surveillance subset contains 85 AI-generated CCTV-style images. The Occupations subset contains 128 AI-generated occupational portraits in a balanced paired design covering 64 occupations with one female and one male version of each. The AI-edited subset contains 27 real photographs in which a region was altered or replaced by a generative model. The Video subset contains 56 AI-generated short videos. We curated the images and videos manually over multiple rounds to make them as photorealistic as possible. For each stimulus we release the file, the generation prompt where recorded, and per-stimulus metadata. We report depicted-gender distributions from an initial set of 134 generations without a specified gender and compare them with U.S. Bureau of Labor Statistics occupational gender shares. We report edit-region size and its association with zero-shot detection difficulty for the AI-edited subset. We report image-quality distributions for the Surveillance subset and subject-category coverage for the Video subset. HumanCentric-Scenes-AI is intended first for human-perception research, and also for calibration of AI-detection classifiers and bias analysis of generative models.
Added Projects. One FableCut keeps many projects; ./project.json stays the open one, so tools and scripts are unchanged. The Projects dashboard (top bar) lists them with a picture, size, frame rate and length, and opens, creates (from Landscape, Vertical, Square, Portrait, 4K and 720p canvases or a saved project template), duplicates, renames, tags, takes notes on, archives and deletes them, and imports a project file from elsewhere. Media, exports and the library are shared. Agents use the new fablecut_projects tool; REST GET / POST /api/projects . Versions. Saved copies of a project, never overwritten: Ctrl/Cmd+S or Save version with a name and note, automatically every 10 minutes while the project changes, and before an agent's first edit or a whole-project rewrite. Compare a version with now (clips added / removed / changed, length, media, settings), restore one (what was there is saved as a version first), mark the final cut. Download backup carries every version and imports back; an unreadable project.json at startup is kept aside and the newest good copy restored. Agents use the new fablecut_versions tool; REST /api/versions . Sequences. Several timelines in one project — a 16:9 and a 9:16 cut, cut-downs, alternatives — sharing its media and compositions, each with its own canvas, frame rate, IN / OUT, markers, mix and delivery settings. The menu at the left of the timeline toolbar opens, adds (empty or as a copy, at any canvas), duplicates, renames and deletes them; each keeps its own undo history for the session. Media and comps used in any sequence are counted and protected. Agents use the new patch ops addSequence , inSequence (edit a sequence without opening it), openSequence , updateSequence and removeSequence . NVIDIA GPU export profiles nvenc-h264 and nvenc-hevc . They need an NVIDIA card and an ffmpeg built with NVENC. (thanks @delirious6423 , #118 ) Changed Export decodes video with WebCodecs instead of seeking <video> : about 5× faster (95.7 s → 18.8 s on a 28 s 720p project) and frame-exact. Files it can't decode, and background-removal clips, keep the <video> path. (thanks @delirious6423 , #118 ) Fixed Ctrl/Cmd+S no longer also splits at the playhead and opens the browser's Save dialog (it saves a version now). A relinked or replaced file mixes its new audio in preview and export (decoded audio was kept per media entry, not per file). Hardware-encoder profiles pass the export dry run. It probed a 64×64 frame, which NVENC refuses; it now uses 256×144 and reports ffmpeg's reason. (thanks @delirious6423 , #118 ) Contributors Thanks to @delirious6423 for the faster, frame-exact export: WebCodecs decoding, the NVIDIA profiles and the dry-run fix ( #118 ). 官方 Release：v1.16.0。请查看 Release notes、依赖和许可证变化。
Synthesised data is a common strategy to address the lack of annotated handwritten music scores in Optical Music Recognition (OMR). Smashcima is a recent synthesis framework that assembles pages of handwritten scores by sampling glyph images from MUSCIMA++ and positioning them with hard-coded rules. In this work we ask whether these rules are grounded empirically and whether a learned model for context-dependent positioning can further improve synthesis quality. We extract inter-glyph distributions from around 285 thousand annotated relationships across MUSCIMA++ and OmniOMR, finding that several key parameters deviate systematically from real handwriting. Replacing these constants with empirical means reduces positioning RMSE by up to 60%. We then train a Graph Neural Network (GNN) on the Music Notation Graph structure to predict context-dependent offsets, achieving an additional 50 to 85% RMSE reduction on stems and beams by modelling how neighbouring glyphs constrain placement. Cross-dataset analysis shows that spacing differences between the two corpora are explained by note density (3.3), not style, and that duration-proportional spacing does not hold in handwriting (R2 = 0.001). In downstream evaluation, a YOLOv8 detector pretrained on GNN-positioned synthetic data achieves +30% mAP50 over the baseline and +11% over empirical means when fine-tuned on 200 real handwritten staves, showing that positioning accuracy matters for downstream OMR.
Human video scales easily and does not need robot hardware, but existing methods require depth sensing, overlay the robot onto 2D video, or generate data only for the demonstrated objects. We present Real2Sem2Real (R2Sem2R), a real-to-semantic-to-real pipeline that turns one monocular RGB video of a human demonstration into thousands of robot demonstrations across the semantic object instances of a category. R2Sem2R tracks the SE(3) pose of each object with DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth. R2Sem2R then aligns digital cousins, each generated from one or a few photographs, to the demonstrated objects with a semantic alignment. This alignment matches whole objects and the functional regions of target objects, such as a coffee machine's tray. R2Sem2R maps the human hand to a feasible robot grasp that extends to the cousins. Robot demonstrations are generated either in IsaacSim, for photorealistic rendering, or MuJoCo, for contact physics, and R2Sem2R can generate over 2,000 episodes in about two hours. On two real robots and four tasks, we train a vision-language-action policy on data from one human video per task, and policies trained on our data match or exceed policies trained on teleoperation across diverse object instances. The project page is available at https://real2sem2real.github.io/.
Instruction-guided video generative models offer a scalable solution for stress-testing autonomous driving systems by simulating diverse environmental conditions. However, driving scenes operate within a dense Operational Design Domain (ODD) governed by strict traffic semantics. When applied to global driving-condition edits, general-purpose instruction-guided video editors frequently corrupt these semantics, leading to structural hallucinations and ``temporal shock.'' This work distills and protects traffic semantics during video editing at three levels. First, we construct a hybrid dataset by combining temporally aligned real-world videos with synthetic edits, balancing foundational scene structure with unambiguous appearance signals. Second, during training, we adapt advantage conditioning from offline reinforcement learning to align the model using dense rewards. We extract dense, pixel-level rewards from the hybrid data to serve as a localized signal and actively steer generation toward structurally reliable pixels. Finally, our model mitigates temporal shock by introducing thinking frames, a transition mechanism that seamlessly reconciles static image conditioning with dynamic video context. Extensive experiments demonstrate that our method prevents the corruption of multi-actor dynamics, establishing state-of-the-art spatiotemporal consistency, yielding consistent user preference (39% relative improvement in content preservation and 12% in temporal consistency), and achieving a significant 5% gain in downstream road line segmentation and 7% gain in downstream multi-object detection.
We present WorldStream, an open-ended 3D world model with three simultaneous capabilities: it extends playable terrain beyond any pre-specified map, acts on open-ended language in the running world, and restores visited places with their edits. A hierarchical planner over a typed decision model emits structured decision state: Macro fixes settlement, ecology, transport, and hydrological structure across a group of tiles; Micro specializes that contract into tile semantics under measured terrain and accepted neighbor commitments. Deterministic local code compiles accepted plans into terrain, roads, settlements, vegetation, and entities. Streaming separates their reach: plans run ahead along the predicted corridor, compilation only where movement will soon need it, geometry behind the user released. One world record retains plans, seeds, object identities and mutable state, boundary commitments, and edits by attribute, revised as the user advances and edits; returning tiles recompile from it. The same record grounds interaction: the model acts on existing entities, including stateful NPCs that update as they act, edits the present region, or attaches intent to unplanned regions. Together they yield a world expanding under bounded budgets, connected across independently planned tiles, and interactive.
In vision and scientific domains involving spatial data, a key question is whether node feature values vary independently across space or form coherent spatial structures. Classical spatial autocorrelation statistics such as Moran's $I$ and Geary's $C$ quantify this phenomenon but operate on a single scalar variable with fixed neighbourhood weights, limiting their use for high-dimensional graph-like data. We introduce SAIL (Spatial Autocorrelation Index Learner), a differentiable framework that generalises spatial autocorrelation into a learnable operator by jointly learning feature projections defining candidate scalar fields and inter-node neighbourhood affinities. SAIL produces interpretable local and global spatial coherence descriptors, enabling supervised prediction and unsupervised discovery of spatial autocorrelation modes and hotspots. We show that SAIL exactly recovers classical spatial statistics under appropriate parametrisations and, for tasks driven by latent spatial autocorrelation, can approximate the Bayes-optimal predictor with an inductive bias matched to such tasks. Experiments on synthetic data and large-scale biomedical graphs demonstrate statistical correctness and discovery of spatial coherence with applications across vision, geospatial analysis, and biomedicine. The code repository is available at: https://github.com/pkeller00/SAIL.
Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.
Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON
Deploying a trained 3D Gaussian Splatting scene under shrinking compute budgets requires one primitive set to remain effective when rendered at smaller prefixes. In this continuous level of detail (LoD) setting, the challenge is to decide which Gaussians should retain capacity as the budget decreases. A pilot study on 3DGS-MCMC reconstructions shows that image complexity, computed from the input views before optimization, predicts where reconstruction error concentrates in image and 3D space. It also reveals that simple regions contain too many Gaussians and degrade as much as complex ones at lower budgets, despite being inherently cheaper to represent. Subsequently, in KoalaGS, during training we use this complexity prior to bias relocation toward poorly reconstructed areas while ensuring smooth degradation across LoDs. A primitive importance score then enables inference at low-capacity, while preserving the scene structure. Tested across three datasets, KoalaGS maintains competitive full-capacity quality while significantly improving quality across the budget spectrum, yielding smoother transitions across LoDs, and enabling faster progressive streaming.
Video generation models can create realistic-looking video footage from images and text prompts. Some of the recent advances have raised concerns about the reproduction of copyrighted visual content. However, copyright leakage detection in generated videos remains largely unexplored. In this paper, we introduce a new benchmark data set for copyright leakage detection, comprising 50 copyrighted concepts, each with 5 images. For each concept, we have 5 videos created using the image+prompt video generation model and the prompt-only video generation model. Following that, we develop an AI toolset based on a contrastive retrieval framework that formulates copyright leakage detection as a video-to-image retrieval problem by matching aggregated video frame embeddings against a database of copyrighted image embeddings. Our method uses a single similarity threshold and retrieves the most similar copyrighted reference as supporting evidence. We conducted experiments showing strong retrieval and detection performance, achieving 99.6\% detection accuracy with only a single false negative. These results show the effectiveness of the retrieval-based method for scalable and explainable copyright leakage detection in AI-generated videos. Data can be accessed publicly from https://zenodo.org/records/21386117 and the source code for this project is publicly available at https://github.com/Shaif95/video-cp-Leakage.
Video generation and feed-forward 3D reconstruction typically use separate representations, requiring RGB decoding and visual re-encoding to recover geometry from generated latents. We propose Latent Bridge, which connects video generative latent spaces with geometric feature spaces to directly predict 3D geometry from video latents. We develop representation-specific bridges for Wan2.1 and V-RAE, enabling feed-forward prediction of cameras, depth, and 3D point clouds without the intermediate RGB pathway. Each bridge is trained on RGB videos through feature distillation, while the video encoder and geometry model remain frozen. Training requires neither ground-truth geometry nor execution of the downstream geometric aggregator or prediction heads. At inference, the same video latents support both RGB decoding and geometry prediction. The same trained bridge can be reused for image-to-video and text-to-video generation within the same video latent space, without bridge retraining. Experiments on Co3Dv2 and WildRGB-D demonstrate competitive geometric reconstruction quality, efficient end-to-end generation and reconstruction, and faster latent-to-geometry decoding, highlighting latent alignment as a practical and reusable interface between video generation and 3D reconstruction. Our code is available at: https://github.com/hustvl/Latent-Bridge.
Object-goal navigation (ObjectNav) requires an embodied agent to find an instance of a specified category in an unseen environment and stop near it. The agent typically relies on an object detector but must judge online from a limited egocentric video stream, sometimes a single frame, whether a detected candidate is the target. Detectors often confidently mistake similar objects, such as a sofa for a bed, and stopping at a false positive fails the episode. Prevailing systems also approach candidates greedily, leaving them occluded or partially observed. We present CLOSER, an evidence-aware active verification framework that treats stopping as closed-loop evidence acquisition, coupling vision-language model (VLM) verification with active perception. CLOSER fine-tunes an open-source VLM to label each view of a candidate tracked across frames as target, non-target, or uncertain, selects complementary views with an embedding-based diversity gate, and fuses their per-view probabilities with a lightweight network. If the fused judgment remains uncertain, a diffusion policy drives the agent toward the candidate using only the tracked mask and depth until the decision becomes decisive. CLOSER runs locally and plugs into existing ObjectNav systems without retraining them. On Gibson and Habitat-Matterport 3D, it raises average success rate by 3.6 points and success weighted by path length by 2.0 points, showing that verifying evidence before committing improves embodied navigation.
AI coding tools can generate code quickly, but speed alone does not solve the hardest software engineering problems. Developers still need to provide context, validate results, protect maintainability, and remain accountable for what ships. To learn how experienced developers strike that balance, we spoke with three Java Champions, all with over a decade of experience in using Java. They shared concrete examples of using GitHub Copilot in production and open source, along with the practices they use to turn AI-generated output into reliable software. Here are the tips they shared with us. Meet the Java Champions Vlad Mihalcea , founder of Hypersistence and author of the book High-Performance Java Persistence Patrick Baumgartner , software crafter at 42talents with 20+ years of Java experience Frank Delporte , senior technical writer at Azul and maintainer of Foojay.io with 15+ years of Java experience Vlad Mihalcea GitHub profile: @vladmihalcea How has GitHub Copilot helped you recently? I’ve been using it to analyze, design, and implement various features in both Hypersistence Utils and Hypersistence Optimizer. For example, I’ve been wanting for a long time to develop a web application that provides Optimizer event history so we can see how performance improvement efforts are materializing in application performance. Before AI, I kept postponing this task because it would have taken me weeks to develop. But using Copilot is like having a new team member, so I used it to create an app that will ship in the next version of Hypersistence Optimizer. Instead of weeks, this task took just two days to implement. What’s your typical usage? Are you using GitHub Copilot as an IDE plugin or the GitHub Copilot app? I use GitHub Copilot in IntelliJ IDEA because it’s convenient to run while reviewing source code or during a debug session. It’s very useful as a code assistant, and it helps me speed up both investigations and building prototypes for features that need validation. I still review the plans and the code that is generated by AI since their goal is to improve my productivity, not replace me entirely. While AI provides the right answer in many situations, there are still times when the response is not satisfactory and requires my intervention. Can you share some tips and best practices? For example, which skills are you using? I created my own skills so that whatever text Copilot generates looks like something I wrote by hand. This skill captures my voice from 65 of my published articles, so a draft comes back sounding like my blog instead of like a language model. This is really useful when generating documentation for this project, since it lets the LLM learn from 12 years of blogging and provide examples that follow the same principles, such as expressiveness, testability, and simplicity. Throughout my career as a software engineer, I’ve learned that we spend far more time on code maintenance than on creating it. Therefore, because maintenance and stability are critical for existing products, I developed skills that help AI mimic existing code practices in accordance with the principle of least astonishment. One tip is to change the model type based on the type of operation. For thorough thinking, I use Opus with the 1 million context window and Extra High or Max reasoning. However, for simpler tasks, such as generating reports, release notes, or verifying documentation, I use lighter models that consume fewer tokens. Patrick Baumgartner GitHub profile: @patbaumgartner Your work involves coaching teams. What is your approach? A lot of my work happens while the normal business still has to keep running. You cannot stop everything for six months to “transform.” We usually start by solving immediate problems together, understand where the team currently stands, define where we want to go and how we will know that we are improving, and then move there step by step. Depending on the situation, that can mean Software Craft practices such as TDD, refactoring, continuous delivery and XP; architecture and modularisation with Spring Boot, Spring Modulith; Spring Security; optimising Java and Spring Boot applications for containers; or increasingly, agentic engineering. The important part for me is that I work with the team. We make decisions together. I don’t want a team to become dependent on me. Ideally, after some time they no longer need me except perhaps as a sparring partner. And maybe that explains why I find this new generation of AI tools so interesting: I like working with people, learning, experimenting, and solving problems. AI gives us another very powerful tool for doing exactly that. Can you share a good story about how GitHub Copilot has helped you recently? There are actually three very different examples that changed how I think about working with GitHub Copilot. Let’s start with the first one. One was a migration of around 900 teaching slides across 15 presentations from Reveal.js to Marp. These were not simple slides. Many contain code, animations, and even interactive examples where I can change JavaScript live while teaching. More importantly, the presentations had already been battle-tested with students. The flow worked. Losing one small piece could break the story I use in the classroom. Doing that migration manually would probably have taken me a week or more—and realistically, I might simply have avoided doing it because it was such tedious work. With Copilot, I approached it differently. We first analysed the existing material, defined the target, created a migration plan, established constraints, provided examples, and defined what had to remain intact. Then we worked from the big structural changes down into the details. The really important part was verification. We generated screenshots of the Reveal.js and Marp versions and compared them visually and algorithmically. We built a small application where I could run both slide decks next to each other to verify animations manually. We even compared the generated PDFs. That meant the agent did not just say, “Migration complete.” We had evidence. The entire migration took me roughly eight to ten hours. I still owned the content, the teaching intent, and every meaningful change. Copilot could fix a typo automatically, but if content, versions, dates or meaning changed, that was something we discussed. For me, that experience reinforced an important principle: delegate execution, not responsibility. I own the result. Can you share the second example? A second example came from building a terminal UI application with Tamboui. It initially worked perfectly in Linux under WSL, and I had completely forgotten that somebody might run it directly in Windows CMD or PowerShell. And then a real user did. They discovered that the Enter key was simply ignored in some forms. That is another principle I strongly believe in: ship software early, because you learn from how people actually use it. We used Copilot to turn that feedback into reproducible UI tests. On Linux, Copilot was navigating the application with tmux. Through working with it, I learned about psmux and could execute similar flows in PowerShell. We reproduced the problem, created a local fix, traced the real problem into Tamboui, and I submitted the upstream fix, which was merged. During this work, the test descriptions themselves became too large for the available context. So I asked Copilot to help me redesign the way the tests were represented. Instead of verbose tables containing identifiers, names and repeated descriptions, it proposed a much more compact arrow notation: step → step → input, with additional information only where it was critical. That surprised me. We were not just using AI to execute information; we were working together on how information should be represented for execution . It did not have to be maximally human-friendly. It had to preserve the semantics while being efficient for the agent. How about the third example? The third example is completely different again. Someone close to me needed to practise mathematics, and I thought: why don’t we build a small learning app? It covers addition, subtraction, multiplication, division and division with remainder for grades one through four. But I did not want to build another simple quiz application. Together with Copilot, I researched learning approaches such as spaced repetition, immediate feedback, adaptive difficulty, motivation, cooperative learning and gamification without dark patterns. Interestingly, during that research Copilot challenged one of my original assumptions. I initially wanted the application to be generic for children everywhere. The research led us toward looking at the actual school curriculum instead. I realised that the order in which topics are introduced, the operations and number ranges differ enough that making it “universal” immediately was unnecessary. So we focused on Switzerland first and aligned the progression accordingly. Because Switzerland is multilingual, the application is available in German, French, Italian and English. It also has cooperative and competitive two-player modes, and when someone gets an answer wrong, the application explains one way to solve it and brings similar questions back through spaced repetition. I deliberately chose an architecture without a backend. There is no reason for me to collect children’s personal data on a server, so the data stays on the device and the application can simply be hosted on GitHub Pages. What started as a small experiment is now being used and shared among friends. It accidentally became a little product. That experience showed me another side of Copilot: don’t only use it to implement your solution. Use it as a sparring partner to expand and challenge the solution space before you start building . How do you typically use Copilot? I actually don’t use the classic Copilot autocomplete experience anymore. My work has moved almost completely toward agentic workflows. I use VS Code when I need a tight visual feedback loop—when I want to look closely at documents, code, presentations or intermediate results and adapt the process as I go. For larger execution-oriented tasks, I often work with the coding agent from the terminal. There I am more comfortable delegating a piece of work, letting it execute, and then inspecting the complete result. The interface I choose depends mostly on the feedback loop I need. I also had a fascinating experience using the coding-agent workflow through GitHub while travelling to JavaLand. I had started exploring an idea for a Maven plugin that optimises which Spring Boot auto-configurations are loaded. During roughly a four-hour journey, I was steering the agent from my phone: giving it a task, reviewing what it had done, thinking about the next step, and sending the next instruction. By the time I arrived, I had a working Maven plugin, a CI pipeline, performance tests against a sample application, and reports that could demonstrate whether the optimisation actually worked. That was the first time I had worked that way, and I was amazed that it actually worked. I wasn’t trying to write Java on my phone. I was reviewing, deciding and steering while the agent executed . I also use MCP servers when they give me useful capabilities or context. For example, I have used Context7 for documentation, integrations with my Logseq knowledge base, and I have experimented with an MCP integration for Adobe Premiere Pro. But I don’t use MCP just because I can. If a good CLI exists, I often prefer the CLI because it feels more direct and faster to me. The agent is also not a “junior developer” in my mental model. I think of it much more as a sparring partner. But just like anyone joining a project, it first needs to be brought up to speed. Context is king. Can you share some tips and best practices? The biggest thing I would recommend is: stop one-shot prompting. For anything non-trivi
In the summer of 2010, postdoc Chris White worked with a multidisciplinary team at the Defense Advanced Research Projects Agency, or DARPA, to determine if a new approach to data analysis could prevent violence and protect troops during the war in Afghanistan. At the conclusion of the program, the agency asked White to prove the value of his work—on the ground. In this episode, Principal Technical Program Manager Weishung Liu talks with White, director of Catalyst Lab at Microsoft Research, about his unexpected break from academic research to help lead advanced tech operations in Afghanistan and his later decision to again work on tech with immediate, real-world implications: dark-web search tools used to help combat human trafficking. Liu and White explore the influences that led to these experiences plus the important role of problem selection in innovation and remaining hopeful after seeing the darkest parts of humanity. Subscribe to the Microsoft Research Podcast : Apple Podcasts Email Android Spotify RSS Feed Transcript [MUSIC] CHRIS WHITE: I have seen a lot of really bad humanity. You know, I’ve seen terrorism and trafficking and sophisticated organized crime, you know, designed to take things from people. Designed to hurt people. And I just know that I can do something about it, you know, and that I have and that we can. And so I think that that light, you know, the light of making a difference, you know, allows me to hold and handle, you know, the dark parts. [THEME MUSIC] STANDARD INTRODUCTION: This is the Microsoft Research Podcast , where Microsoft researchers—driving advancement through fundamental science and technology research—explore the who, how, and what’s next in computing and AI. [MUSIC ENDS] WEISHUNG LIU: Hi, and welcome. I’m Weishung Liu, a principal technical program manager here at Microsoft Research, and today I’m talking to Dr. Christopher White, who leads Microsoft Research’s Catalyst Lab . Chris has one of those careers that sounds almost made up: a Harvard postdoc, on-the-ground advanced tech lead during the war in Afghanistan, a DARPA program manager, a dark web search pioneer, and a public-good technologist. And his work has been featured in Popular Science (opens in new tab) as well as recognized with honors, including the 2016 Presidential Award for Extraordinary Efforts to Combat Trafficking in Persons (opens in new tab) . But what I’m most interested in today is the person behind that résumé. How someone who has seen some of the worst as part of his work still comes to the office with a smile and a commitment to building tools that make the world better. Chris, thanks for joining me. WHITE: Thanks for having me, Wei. Nice to see you. LIU: Yeah, good to see you, too. And so let’s get started. So I’ve known you for, I think, maybe around five or maybe six years, and now I work with you in Catalyst Lab. I love your story, and when I tell people about you, I say that you lead the newly formed Cat Lab, or Catalyst Lab, in Microsoft Research, you were a senior executive at DARPA, and you are married to an incredibly accomplished woman and scholar. And I personally think that your life story would make a very watchable Hollywood film. So when you tell your own story, though, where do you start? WHITE: Well, you start from where you are. And where I am today is leading research at Microsoft during a period of great fear and hope about changing technology and society. LIU: But, like, that’s now. There’s clearly lots of time from now into before, you know, the before times. So, like, tell me a little bit about the before times. Like, where were you at—I think you grew up in Oklahoma maybe. Tell me about that. Tell me about Chris White, the child. WHITE: Well, I was nerdy. I was a nerdy Eagle Scout. [LAUGHTER] I liked to build my own computers and have local area network [LAN] parties with my friends. But I also really liked being outside. I liked being in nature and camping and rock climbing. Those were all important to me. LIU: OK, you mentioned LAN parties. Tell me about … what did we play with our friends? WHITE: Well, you know, we played everything. I preferred playing sports games, but I lost a lot of the other ones. LIU: [LAUGHS] Wait, so you played like LAN party sports games, like, in basements? Like, with your … were we all in the same … ? WHITE: Yeah. Attics. It was attics. Yeah, not basements. [LAUGHTER] Mm-hmm. Yeah. Everybody had to bring their own computer or bring their own Ethernet cable. And the more that you had your own machine that was higher powered, the better you were going to play. LIU: So did you have the best machine? WHITE: I had the best machine. I built the best machine. Until someone built a better machine. But it was the best for a while. And I had a budget for it. And my family didn’t understand that the budget also included two giant speakers, like the size of a computer tower, like subwoofer speakers. They didn’t know that was part of the computer building process. But it turns out that excellent audio-visual is a critical part of a well-functioning machine. LIU: I love it. But, OK, we’re going to focus a little bit … continue focusing on childhood. So you mentioned nerdy kid, which means—I’m going to generalize here—debate team or math competition team or chess team? WHITE: Oh, it was the debate team. Yeah. Debate was a really wonderful activity. It taught me how to understand many sides of an issue and taught me how to think about evidence and research in that way. LIU: So, Chris, take me back again. Like, we’re talking a little bit about Oklahoma, and, you know, at some point, you chose researcher. Like, I—for example—when I was a kid, I was convinced I would be an architect. I am not an architect as you can see. But did you always think you were going to be a researcher? Was there an epiphany there? Like … or was it always kind of like a slow hunch that you were going to pursue research as a career? WHITE: Yeah, I kind of always felt like it. You know, as I was applying for school, those were the kinds of jobs I was interested in. You know, the kinds of things that you do, you know, once you finish degrees that have research in them. Those were the people, when I visited their work, I was inspired by. The people that were discovering, you know, that were inventing, that were innovating, that were … they’re pushing the boundaries that were kind of in that unknown, you know, space and making sense of it. And so I’ve always had that curiosity, you know, and that approach to discovery. And so as I increased my education, it just changed the nature of what research meant and the ways in which I imagined that process evolving. LIU: Was a doctorate always on the cards then? Or did that come, like, as you were in college and then gradually learned and thought, you know, I could keep going? WHITE: Yeah, you know, I didn’t know about doctorates really when I went to college. I wasn’t as familiar with higher education. And so I knew that I wanted, you know, advanced degrees. And I knew that I wanted more than was being taught in the undergraduate level. But I didn’t know, you know, for example, that when you apply to PhD programs, that often you get paid to go. You know, you get admitted and you get a stipend. You know, I was familiar with law and medicine and that you take out loans and you have to do it this way. And so for me, it was a surprise to know that I could apply to PhD programs directly from college. And from then, I just really was excited, you know, that I could explore different universities and different areas of focus and build on, you know, strong technical foundations of math and science and physics. And it was really amazing mentors that helped me and opened doors and inspired me. You know, mentors at Oklahoma State University, Johns Hopkins, and Harvard, really people that took their time and helped me. I’m really grateful for my teachers. LIU: So let’s talk about your first mentor of your life, and I’m going to assume … Dad ? WHITE: Yeah, Dad’s a big influence. LIU: OK, Dad’s a big influence. And so we’re going to, I’m going to go down this path. So Catalyst Lab landed a huge commitment from our executive, kind of, leadership team on a project that has been, I think, like a decade in the making. And then … so we all, I know we all came together on a call. We’re like, “Wow, let’s debrief. Let’s talk about it.” Like, oh, energy’s up. And then at the very end of the meeting, you said, “OK, now I’m going to go clean some toilets.” WHITE: [LAUGHS] Mm-hmm. Mm-hmm. LIU: And I think all of us kind of thought, like, wha-what does that mean? So before we get kind of more into the specifics of mentorship and the, like, follow-on in your career, what— please tell me —does cleaning toilets have to do with success? WHITE: Well, you know, like I said, Dad’s a big influence, and he taught me so many things, and he taught me about respecting people. And most importantly, he taught me about humility and service. And so his way of making sure I understood that was every time something really good happened to me, he’d remind me to go clean some toilets and, sort of, embody that. And so I did. I did it last weekend actually. I went to a community event and volunteered to do bathroom clean. LIU: Is this like … are you … so in, like, the household chores, are you, like, the bathroom cleaner? Like, is this just always your job when we’re dividing up chores in the house? WHITE: No, no, no. We trade off, and also we have a complicated household, and so we have a lot of help that also helps with cleaning. And so this is something that I choose to do, especially if I can do it in a community context, because it again reminds me, you know, about humility and about service and of respect for other people and other jobs. LIU: So I think this … there’s some Eagle Scout vibes that I’m getting from here, as well, based on sort of humility and service. What is the Boy Scout badge you are most proud of? WHITE: Oh, that’s really hard. Each badge is something that you earn, including the Eagle Scout badge. And the Eagle Scout badge for me was maybe the most difficult to earn because it requires an Eagle project. And I was really close to my Eagle project. I was, I was … had it set up and it was ready to go. And it was in this church in downtown Oklahoma City in 1995. And right as I was about to do this project, the Murrah Building blew up in a terrorist incident (opens in new tab) , … LIU: Whoa. WHITE: … and it blew up part of the church. And, therefore, my Eagle project was not realistic, and it was not a priority, and it got shelved. And it took me several years after that to then come back to plan and execute an ambitious Eagle project right before I turned 18. And so that badge, … LIU: Wow. WHITE: … you know, completing the project, overcoming significant setbacks, including being a teenager and having other interests, to come through and finish it, I’m really proud of finishing it. LIU: That’s awesome. So you … so that was, so there were a few years then in between when you ideated, had the idea for the project, and then … I appreciate the story around resilience because of when you completed the project. Like, what is the biggest takeaway that you took from there? Like what is that big lesson? You turned 18, you finished it, you celebrate. Actually, you know, maybe I’ll ask you, how did you celebrate? WHITE: Oh, with my family. You know, the Boy Scouts is a long process. It’s many years, many kinds of roles, many kinds of learning, and to finish it—I finished it with another of my fellow scouts that had been together for a long time. And so our families, we had a ceremony, and it was something we could celebrate. And every time I think about it, I think about the things that you keep repeating when you attend the scout things, like the motto, “being prepared,” and the, kind of, laws about being respectful and courteous. And thos
### Added - Generate images with Codex CLI using its saved sign-in, independently of the conversation agent, from canvas image nodes or any agent's `generate_image` tool. ([#332](https://github.com/WhiteTowerAI/ohmygame/pull/332) by @dogxii) - Use Gemini 4 Argon with Google Gemini and Vertex AI, including Low, Medium, and High reasoning, through the bundled Pi 1.3 upgrade. ([#316](https://github.com/WhiteTowerAI/ohmygame/pull/316) by @dihuangdh) - Preview Markdown files with project-relative links and images in Code, and start publishing with the `/publish` command. ([#309](https://github.com/WhiteTowerAI/ohmygame/pull/309) by @dogxii) ### Changed - Load long conversation histories, generate Scene thumbnails, and export media-heavy projects more efficiently. ([#329](https://github.com/WhiteTowerAI/ohmygame/pull/329) by @82228) - Label coding-agent quota percentages as used, keep slash-menu usage compact, and show agent brand icons in model selection. ([#326](https://github.com/WhiteTowerAI/ohmygame/pull/326) by @dogxii) - Kimi Coding now uses OpenAI Chat Completions through Pi 1.3; existing custom Base URL overrides for `kimi-coding` must include `/v1`. ([#316](https://github.com/WhiteTowerAI/ohmygame/pull/316) by @dihuangdh) ### Fixed - Merge uncontested canvas node conversions without reporting a board conflict, while preserving conflict choices for incompatible concurrent edits. ([#336](https://github.com/WhiteTowerAI/ohmygame/pull/336) by @dihuangdh) - Keep canvas connections available after importing images and saving. Generation prompts support Enter to generate and Shift+Enter for a new line, with input-method composition and video reference selection protected from accidental submission. ([#334](https://github.com/WhiteTowerAI/ohmygame/pull/334) by @dogxii) - Include prebuilt Interactive Story runtime resources in desktop packages so story creation, legacy upgrades, and previews work inside the packaged app. ([#328](https://github.com/WhiteTowerAI/ohmygame/pull/328) by @ElenaYU-77) - Discover Codex CLIs bundled with the macOS Codex and ChatGPT apps, refresh discovery on rescan, and explain unavailable manually selected executables. ([#327](https://github.com/WhiteTowerAI/ohmygame/pull/327) by @ElenaYU-77) - Keep differential update downloads within bounded multipart Range batches to reduce CDN-triggered full-download fallbacks. This takes effect after installing this version; older installed versions keep their existing downloader. ([#323](https://github.com/WhiteTowerAI/ohmygame/pull/323) by @dihuangdh) - Prevent the Design canvas viewport from flashing when opened, and correct project settings typography and copy. ([#335](https://github.com/WhiteTowerAI/ohmygame/pull/335) by @dihuangdh, [#324](https://github.com/WhiteTowerAI/ohmygame/pull/324) by @82228)
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.
Despite rapid progress in text-to-motion generation, existing methods are commonly designed for single-shot generation. When users are dissatisfied with specific motion details, they are often forced to regenerate the entire sequence from scratch, which is computationally inefficient and potentially destroys previously established kinematic context. A practical generation system should instead support iterative user feedback and refine only the affected spatio-temporal regions while preserving the remaining motion. Therefore, we introduce a conversational motion generation paradigm that requires the system to coordinate instruction understanding, motion generation, localized editing, and quality assessment. Recent advances in LLM-driven multi-agent systems provide a promising foundation for such role-specialized coordination. To tackle these challenges, a Self-evolving Multi-Agent RefinemenT Framework (SMART) is proposed for conversational motion generation. Within SMART, to preserve semantic consistency, we design a semantic-guided masked motion editing mechanism, which precisely grounds natural language feedback into spatiotemporal motion regions to enable localized refinements without altering unedited regions. To reduce redundant exploration, we propose a memory-based self-evolving strategy that dynamically accumulates motion editing knowledge from interaction history and reuses past successful experience. Extensive experiments demonstrate that SMART effectively supports multi-turn conversational motion generation and enables efficient motion refinement.
Pose-free 3D Gaussian Splatting (3DGS) aims to bypass cumbersome Structure-from-Motion (SfM) by directly reconstructing scenes from unposed image sequences. In this work, we present CrossFit-GS, a deferred-commitment progressive framework that mitigates cumulative trajectory drift and breaks the self-reinforcing loop of distorted-map error propagation. Specifically, we partition the input sequence into interleaved subsets and maintain two complementary submaps. Each candidate pose is cross-refined against a frozen opposite submap that has never been optimized using the target frame, thereby providing an independent geometric reference for pose estimation. Checkerboard validation and spatial coverage safeguards further reject unreliable updates, and only verified poses are committed to their corresponding owner submap. Once the trajectory has been refined, the intermediate submaps are discarded and the final global representation is trained from scratch using the fixed refined poses. Extensive evaluations across multiple benchmarks demonstrate that CrossFit-GS substantially reduces cumulative trajectory drift and consistently improves both camera pose estimation and novel-view synthesis over existing methods.
Existing feature lifting approaches for open-vocabulary segmentation rely on simple backprojection to assign features to primitives. In the case of 3D Gaussian splatting reconstruction, the primitives overlap heavily, leading to suboptimal reconstructions, governed by many primitives per ray. Backprojection, however, reduces to a least-squares solvable problem, exactly when no two primitives share a ray. Motivated by this insight, we propose OV-Foam, which uses PowerFoam as the underlying representation. This representation uses a power-diagram to partition the scene, resulting in zero overlap between primitives and a substantial reduction in primitives per ray, which ultimately simplifies the lifting problem and allows for closer-to-optimal feature assignment. We show that the space partitioning primitives of OV-Foam reach near-optimal assignment, and our experiments demonstrate improved open-vocabulary segmentation over Gaussian baselines.
Inline quality assurance for Sheet Molding Compound (SMC) manufacturing is hindered by the lack of large-scale, accurately labeled image datasets for training machine-learning models. This work presents a process-informed, photorealistic digital twin for carbon-fiber SMC strip deposition. A rigid-body simulation reproduces the deposition kinematics (settling, friction, belt motion), while a physically grounded polarization-rendering model produces realistic appearance, so that every image carries exact, mutually consistent labels in synchronized formats: oriented bounding boxes, point annotations, density maps, and polarization renderings (Angle and Degree of Linear Polarization). Because the joint distribution of many overlapping strips cannot be specified analytically, simulation yields exact, scalable ground truth that manual annotation cannot provide. The twin produces over 16,000 images across 3,300 frames, and the synthetic data replicate near-isotropic fiber orientation (axial concentration r < 0.05) consistent with as-deposited commercial SMC. Trained purely on synthetic data, YOLOv11 and CSRNet reach detection mAP50 up to 0.91, with a dedicated oriented-bounding-box detector confirming the oriented labels are directly trainable (mAP50 = 0.83); on real manufacturing images, the synthetic-to-real difference in edge and texture structure (Kolmogorov-Smirnov distance 0.46, distribution overlap 0.56) is of the same magnitude as the difference between two real acquisition series of the same material (0.41, overlap 0.57). The approach substantially reduces the labeling burden for camera-based inline inspection of fiber orientation and volume content.
Scaling data, diversity and model capacity has improved embodied learning in physical environments, where diverse experiences remain grounded in shared physical laws. Whether these gains extend to virtual worlds with different visual styles, dynamics, controls, and objectives remains unclear. We investigate this question through a controlled study of continued pre-training (CPT) across three axes: data volume, diversity, and model size. Our 120 training configurations span four model sizes from 0.8B to 27B, six data budgets from 50 to 1,600 hours, and five mixtures ranging from Minecraft-only data to diverse multi-world corpora. We assess performance and generalization using three-level validation losses and gameplay benchmarks from in-corpus worlds to unseen worlds. Our results show: (1) more data improves in-corpus performance, diversity improves transfer to related worlds with limited gains in distant worlds, and model size improves data efficiency but generalizes worse to distant worlds under single-world training; (2) we observe an early interface phase with broad cross-world benefits and a later world phase with gains narrowed to in-corpus and related worlds, and fit an empirical scaling law relating validation loss to data volume, diversity and model size; and (3) our models are competitive with Minecraft specialists and frontier models in 3D worlds, and perform best in real-time settings. However, the limits of imitation and persistent gaps in distant worlds suggest that building generalist embodied agents requires richer training stages beyond continued pre-training.
Camera design heavily influences perception task performance, yet cameras are typically designed by domain experts through costly experimentation, in isolation from the tasks they serve. Existing joint-design methods either rely on differentiable simulation limited to pre-captured 2D images, restricting them to optics or image-processing pipeline design, or use non-differentiable camera simulators optimised with derivative-free or reinforcement-learning methods that scale poorly with the number of design parameters. We present a differentiable framework for jointly designing complex camera systems and downstream perception tasks. We use it to design two coded-aperture depth-from-defocus cameras, a monocular and a stereo configuration, by jointly optimising the coded aperture mask, camera parameters, and a depth estimation network end-to-end. Our framework represents scenes with 3D Gaussian Splatting reconstructed from real-world captures, providing a differentiable image formation process for gradient-based optimisation over design spaces intractable for prior differentiable approaches. Results demonstrate that our differentiable method improves over existing coded aperture design and non-differentiable camera design baselines, offering a step forward in automated and principled camera design for autonomous systems.
Anticipating how users will interact with their surroundings is essential for proactive systems that provide timely assistance in AR, VR, and embodied applications. Prior work typically forecasts hand-object interactions directly from historical observations, leaving intention and contact geometry implicit. However, similar observation histories can precede semantically or geometrically distinct futures. Inspired by behavioral evidence that gaze can provide early cues for underlying intention, we propose Gaze2HOI, a gaze-aware framework that formulates hand-object interaction forecasting as a hierarchical problem comprising intention inference, contact grounding, and coupled hand-object motion generation. Specifically, Gaze2HOI first weights gaze cues by fixation strength and recency, and uses them to modulate the visual representations of a vision-language model (VLM) through Temporal Gaze Saliency Modulation (TGSM), thereby reducing semantic ambiguity in intention inference. Subsequently, an Anticipatory Contact Grounding (ACG) module maps these structured intentions into explicit contact, support, and placement fields, effectively mitigating geometric ambiguity. Driven by these explicit conditions, a diffusion-based model jointly generates the 6-DoF object trajectories and future articulated motion of both hands. To evaluate Gaze2HOI, we further introduce EgoAIM, a phase-aware benchmark that organizes egocentric hand-object interactions into four phases defined by evolving hand-object relations and provides fine-grained intention annotations. Extensive experiments demonstrate that Gaze2HOI achieves state-of-the-art performance in both intention inference and motion forecasting, even approaching oracle-level end-to-end performance in specific interaction phases. Project page: https://gaze2hoi.github.io/.
Procedural generation plays a key role in synthetic data creation and game development, offering greater controllability, editability, and diversity than static mesh representations. However, VLM-based code generation often produces relatively simple 3D assets, while sophisticated photorealistic procedural systems require extensive manual development. Existing VLM-guided procedural methods remain largely constrained by predefined factories and APIs, limiting their ability to create novel asset types. We introduce InfiniDream, a VLM-driven framework for open-ended procedural 3D world generation. First, we propose Symbolically Constrained Asset Generation, which leverages existing procedural algorithms to create novel, photorealistic assets while preserving editability and seed-based diversity. Second, we develop an end-to-end text-to-world pipeline that combines semantic scene planning with Deterministic Frustum-Guided Grid Placement to construct large-scale 3D worlds with precise object placement, procedural population, and animation. Extensive evaluations on 100 object and 20 scene generation tasks demonstrate improvements across four perceptual metrics. Diversity analysis, ablation studies, and human preference evaluations further validate the effectiveness of our framework. Project page: https://yichuanh.github.io/infinidream-page/
Neural Radiance Fields (NeRF) achieve high-quality novel-view synthesis by representing a scene as a continuous volumetric radiance field. However, the geometry encoded by the density field of vanilla NeRF can be noisy and inaccurate, since different combinations of shape and radiance may explain the same input images. To reduce this ambiguity, we augment vanilla NeRF with a geometry-optimization branch based on self-supervised cross-ray consistency. Specifically, this branch samples candidate surface points from the density field, generates virtual rays that intersect at each point from different directions, and renders their appearance using the NeRF network. Based on the visibility-weighted color consistency across these rays, we constrain the implicit geometry. We further regularize the implicit geometry by penalizing opacity at candidates with inconsistent appearance predictions and encouraging local normal smoothness. These geometric constraints are derived from the NeRF predictions and require no ground-truth depth or surface-normal supervision. Experiments on eight Replica scenes show that the proposed method reduces the mean Chamfer distance by 30.8% compared with vanilla NeRF.
Long-horizon human-scene interaction requires coordinated spatial reasoning, physical feasibility, and the ability to recover from compounding execution errors. Text-driven motion models are largely scene-agnostic and lack closed-loop correction, while geometry-conditioned approaches depend on paired scene-motion training or explicit 3D preprocessing that limits generalization to novel layouts and open-vocabulary goals. We present InteractAgent, an agentic multimodal LLM system for long-horizon human-scene interaction that plans, executes, observes, and reflects in a closed loop, using a single MLLM as both planner and evaluator and a frozen MotionLLM as the executor. InteractAgent integrates Online Observation for step-wise recovery decisions (continue/redo/replan) and Memory-Augmented Reflection for multi-round plan refinement with adaptive stopping. On UniHSI, InteractAgent improves task success and reduces penetration compared with training-free baselines. Code: https://github.com/AIGeeksGroup/InteractAgent.
Long-horizon human-object-scene interaction (HOSI) is central to embodied AI, animation, and game development. Existing human-object interaction (HOI) and human-scene interaction (HSI) models excel in their respective domains, but none alone covers tasks requiring multiple interaction skills. Training unified HOSI models is hindered by scarce and costly data. We introduce World2HOSI, a multi-expert orchestration harness that generates continuous HOSI data from 3D Gaussian Splatting (3DGS) scenes and natural language instructions. Our key insight is that long-horizon HOSI hinges not on a stronger single model but on orchestration. Given a shared, structured scene state and an orchestration harness grounded in it, existing experts can jointly accomplish tasks beyond the reach of any of them alone. World2HOSI realizes this insight as a closed loop around a persistent scene state built from decoupled background and object assets. A harness plans action stages over this state, invokes experts via their native conditioning interfaces, and writes each action's effects back, so that every invocation is grounded in the evolving world. To reduce physical artifacts, bounded residual optimization and contact-driven physics simulation further ensure the physical plausibility of human motion and object manipulation, respectively, while preserving interaction intent. Experiments and a perceptual user study show that World2HOSI achieves the best task completion and interaction quality on the evaluated HOSI tasks, while generating more diverse data.
3D scene graphs support language-guided navigation by organizing physical entities and their containment and spatial relations into hierarchical maps. Building such structured maps requires a scene representation that consistently distinguishes semantic regions and object instances across viewpoints. To this end, we present PanoGraphGS, a unified framework that integrates panoptic 3D Gaussian representation learning with hierarchical 3D scene graph construction. The framework learns semantic and instance features on shared Gaussian geometry. Prototype-based semantic learning and association-based instance learning produce consistent panoptic predictions across viewpoints. The learned representation supports novel-view panoptic segmentation and direct Gaussian-level panoptic decoding. A semantic-aware scene graph module further recovers a floor-room-stuff-object hierarchy with containment and spatial relations. We further release NaviGS, a real-world dataset for panoramic scene understanding. The dataset provides 3DGS, mesh, RGB, semantic, and depth data. For 3D scene graph identification, PanoGraphGS outperforms the strongest evaluated baseline in object-node accuracy by more than 50\% points on average across Replica and NaviGS. PanoGraphGS also achieves state-of-the-art performance among the evaluated methods in 2D panoptic segmentation. These results demonstrate the effectiveness of PanoGraphGS in both hierarchical 3D scene understanding and 2D panoptic segmentation. The project website is available at: https://github.com/gaojh0333/PanoGraphGS.
Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK provides the underlying computational engine for low-level geometries, part composition, differentiable parameter refinement, and physics validation. At the same time, NeoMCP exposes these capabilities as structured, tool-callable APIs for vision-language agents to visually inspect, edit code, and iteratively refine programs. Through a hierarchical coordinator-worker architecture, the framework scales seamlessly from standalone articulated objects to complex compositional scenes. Evaluations across object-level and scene-level reconstruction benchmarks demonstrate that NeoWorld-3 achieves state-of-the-art geometric fidelity and physical readiness. Crucially, manipulation trajectories planned within our reconstructed twins achieve successful zero-shot execution on a physical dual-arm robot, completing the Real-Sim-Real loop.
In object goal navigation (ObjectNav), an embodied agent must find a target object (e.g., "find a bed") in a previously unseen environment, relying on spatial commonsense such as object co-occurrence. Large Language Models (LLMs) encode this commonsense, but deploying them on robots exposes a trade-off between reasoning quality and efficiency: cloud-based models incur latency and privacy risks, whereas small on-device LLMs are fast but prone to hallucinated spatial judgments. Hybrid reasoning LLMs offer both a fast non-thinking mode and a slow thinking mode, which raises a key question for embodied agents: when is slow thinking worth its cost? We propose CovNav, a training-free framework that answers this question with Conformal Prediction (CP), turning dual-process (System 1 / System 2) inference into an uncertainty-aware decision. Much like recent decision models such as Jev, CovNav casts each spatial query as a structured binary decision rather than free-form generation: a local LLM in fast mode judges whether the target is likely near an observed object, CP converts its token probabilities into a prediction set, and only ambiguous queries are escalated to slow chain-of-thought reasoning. The object-pair formulation keeps calibration tractable and supports open-set objects through off-the-shelf detectors. Experiments on Gibson and HM3D show that CovNav surpasses the strongest zero-shot baseline by 0.8 and 1.7 percentage points in success rate on HM3D v1 and v2, while invoking slow reasoning for only 12.4% of queries and retaining most of the gain of always-on thinking at a fraction of its latency. Real-robot experiments on a TurtleBot 4 further validate the approach. CovNav shows how calibrated uncertainty can decide when an embodied agent should think slowly, offering a principled form of adaptive test-time reasoning for LLM-based robots.
2.4.0 (Oct 19, 2026) Bug fixes and improvements Enhancements Add SPZ v4 read support ( #484 ) ( @udwinj , @mrxz ) Decouple and abstract paged splat load logic from PagedSplat into FetchSource that retrieves data from in-memory file, URL or any other custom mechanism ( #463 ) ( @mrxz , @xohm , @oscarlorentzon ) Switch to flate2 from zlib-rs backend for RAD deflate compression. ~50% faster compression, ~2-5% smaller outputs and ~20% faster decoding ( #474 ) ( @mrxz , @oscarlorentzon ) Replace miniz_oxide with flate2 for Gzip (de)compression of SPZ files. Remove miniz_oxide as dependency ( #485 ) ( @mrxz ) Address stutter during animations. Skip unnecessary splat sortings if the splat buffer needs or has changed like for example when LoD levels change or splats animate / move ( #347 ) ( #460 ) ( @oscarlorentzon , @mrxz ) Refactor PagedSplats into its own separate file ( #462 ) ( @mrxz ) Remove outdated Github workflows CI commands ( #464 ) ( @mrxz ) Add Rust linter ( #414 ) ( @mrxz ) Improve tests ( #483 , #482 ) ( @mrxz ) Improve docs ( #478 ) ( @oscarlorentzon ) Bug Fixes (WebXR) Use camera projection projection matrix to calculate LoD pixel limit. The THREE.Camera fov value was incorrectly used before: It doesn't match the one reported by the WebXR API ( #488 ) ( @oscarlorentzon ) Adress view-dependent color precission loss due to unwanted SH coefficinets clamping ( #465 ) ( @oscarlorentzon ) Dispose the LoD index textures when renderer is disposed ( #479 ) ( @oscarlorentzon ) Fix issue with Level 3 Spherical Harmonics (sh3) dropped due to incorrect type assertions. It introduces explicit types in PackedSplats and ExtSplats instead of loose types + assertions. ( #467 ) ( @mrxz , @oscarlorentzon ) (WebXR) Fix issue with splat size and anti-alias blur rendered at half the correct width. Use whole viewport width for calculations instead of half ( #487 ) ( @oscarlorentzon ) Ignore errors / rejections after SparkRenderer is disposed. This can happen when for instance there's pending LoD sorting in parallel in the worker when the renderered is disposed ( #476 ) ( @oscarlorentzon ) Replace workers that hit a fatal error / crash / trap. The pool tried to reuse those in a broken state ( #480 ) ( @oscarlorentzon ) Fix issue with long thin splats drawn off axis, too wide or too narrow. Compute 2D covariance eigenvectors without cancellation ( #486 ) ( @oscarlorentzon ) Stop retrying a failed level of detail tree on every update ( #477 ) ( @mrxz , @oscarlorentzon ) Address raycasting precission and speed issues ( #469 ) ( @n3ur0m4n , @oscarlorentzon ) 官方 Release：v2.4.0。请查看 Release notes、依赖和许可证变化。
Online LiDAR-camera calibration remains a fundamental yet challenging task in the field of robotics. Recently, deep learning-based data-driven approaches have emerged, offering high-precision, targetless calibration capabilities. However, existing methods often suffer from a critical dataset shortcut phenomenon due to the lack of extrinsic diversity in their training data. Networks tend to overfit to fixed nominal extrinsic configurations, leading to severe performance degradation and poor generalization in real-world deployments. To address this limitation, we propose a highly generalizable data generation framework tailored for training LiDAR-camera calibration networks. Our framework leverages existing omnidirectional image datasets annotated with depth ground truth to synthesize multi-modal calibration data under rigorous extrinsic constraints. Extensive experiments demonstrate that the data generated by our pipeline significantly enhances the generalization and robustness of typical learning-based calibration networks. Furthermore, we conduct comprehensive real-world evaluations across various camera-LiDAR configurations, validating the zero-shot transfer capability of models trained exclusively on our synthetic data. To facilitate future research and community use, the code for data generation, the pre-trained models, and online benchmark datasets will be made publicly available.
Machine learning benchmarks act as guideposts for model development and spur the creation of training datasets intended to improve benchmark scores. We introduce BenchDoor, a family of data poisoning attacks that exploits this incentive. Beyond preserving benign performance, BenchDoor requires benchmark gains that encourage adoption of data that also implants a backdoor. The adversary either manipulates a benchmark's training or test data, creating an adversarial benchmark, or supplies an independent training dataset while leaving the benchmark unchanged. We formalize five attack scenarios, including training on rewritten test examples and applying similarity-based decontamination. Across multiple-choice Q&A, text classification, and open-ended Q&A with Qwen and Gemma models, BenchDoor achieves positive benchmark gains for all 30 evaluated configurations, with held-out attack success rates of 82.33% to 100.00% at poisoning rates of 2% to 10%. Both objectives remain achievable after synthetic data generation and similarity-based decontamination, with attack success rates of at least 90.33%. These findings show how benchmarks can serve as attack vectors both through their own data and through training resources that deliver improvements on them.
Cross-structural motion retargeting aims to transfer motion between different skeletal topologies. Despite recent progress, existing state-of-the-art models struggle with reliability in zero-shot settings, i.e. skeletons with different topologies which were unseen during training, and recent Transformer-based attempts have failed to outperform specialized geometric methods. We bridge this gap with a Transformer Autoencoder that learns a topology- and translation-invariant latent space. Our core contribution is a learnable flattening of skeletal graphs that captures both local dependencies and global structure. Unlike the standard transformer architecture, which adds positional information to token content, we integrate graph-based positional encodings multiplicatively, a design choice that follows directly from our flattening formulation. The resulting model handles diverse skeletal topologies within a single unified architecture and trains in a fully unsupervised manner, requiring no paired retargeting data. Ablation studies show, that the graph encodings, multiplicative formulation, and Transformer backbone is critical for the performance. In zero-shot evaluations, our method reduces global joint position error by $43-47\%$ over current benchmarks. A user study ($n = 37$), including expert animators, further ranks our approach highest in motion alignment and physical plausibility ($p < 0.05$). These results demonstrate that our model design is key to making transformer architectures effective for motion retargeting, outperforming existing approaches.
Producing synthetic data for comparison testing a migrated COBOL application is a major challenge that agentic migrations face today. Our prior paper, the Agentic Method for Deterministic Validation of Legacy Code Migration, presents a deterministic oracle to facilitate such migrations, the Locksmith Loop. We now propose a novel method for achieving high branch and paragraph coverage through generated synthetic end-to-end test cases, by means of entry-replay test-case certification by observing the code execution trace on a surrogate COBOL program that mocks external interactions via a chronological external-operation tape. We applied our method to a very large COBOL program with 33,429 lines of code to generate 849 synthetic end-to-end test cases covering 2,973/3,806 source branches (78.1%) and 989/1,033 main-AST paragraphs (95.7%). Using the Locksmith Loop's projected-agreement validation, the generated Java code matched the COBOL surrogate on the specified observations across all 849 certified synthetic end-to-end test cases. During the campaign we detected 61 defects and fixed the Java generator accordingly. To test whether the coding agent could silently introduce business logic bugs, we seeded 450 defects into the Java code, such as deleted writes, deleted paragraph calls, and altered rounding, to verify how many of these would be detected by the gating methodology. The gate detected 47.6%, including all deleted writes and paragraph calls in executed code, but was unable to detect the rounding and precision defects.