跳到正文
只看一手
10月23日周五
  1. AWS Machine Learning Blog

    Announcing new ways to customize models with SageMaker AI, without managing GPUs

    Companies getting real results from AI aren’t only using foundation models (FMs) off the shelf. They’re customizing open-weight models using post-training techniques such as reinforcement learning. A model trained on your business context, product catalog, or clinical vocabulary will be more accurate for your task and cost less to run. When you build agents, your data, workflows, evaluations, and success criteria also shape how the model behaves. However, model customization is difficult. You must choose the right technique, set up and manage GPU infrastructure, and, for agents, build environments for multi-turn reinforcement learning. We launched Amazon SageMaker AI serverless model customization to solve these challenges, and today, we’re expanding it with the general availability launch of training sessions. With training sessions you get control over the post-training logic without managing the underlying infrastructure. You write and coordinate the training loop, including custom loss functions or reward signals. SageMaker AI handles GPU allocation, distributed execution, fault recovery, and checkpoint storage. Customize without the hassles of infrastructure management With SageMaker AI serverless model customization, you don’t choose instance types, manage clusters, or write distributed training configuration. We didn’t want to limit you . You can use techniques such as SFT, DPO, RLVR, RLAIF, or multi-turn RL. You can choose how much of the model to update with Low-Rank Adaptation (LoRA) or full fine-tuning and chain techniques together through continuous customization. We didn’t want you to waste time . Our SageMaker AI training sessions cookbook provides examples and use cases, along with model customization agent skills. You describe your use case, and the agent handles data prep, technique choice, training, evaluation, and deployment. The agent works where you work, in SageMaker Studio, Kiro, Claude Code, Codex, or GitHub Copilot. “We adopted SageMaker AI Serverless Training Sessions to build a healthcare claim-denial triage model, and the experience validated the platform from day one. The low-level API surface, direct control over the forward and backward passes, sampling, and the optimizer step gave our scientists the fine-grained control they need to pursue reinforcement learning on a demanding, accuracy-critical use case. Training Sessions gives us the right primitives to push toward production-grade performance.” — Rob Gontarek, Senior Vice President, Data Science, ImagineSoftware Write your own post-training loop One important shift in agent training is toward multi-turn reinforcement learning. For example, an agent that books travel, triages support tickets, or runs a multi-step research task needs several rounds of interaction before you can judge whether it reached the right outcome. However, the training loop can get complicated quickly. You may train the model on one set of infrastructure, run inference on another to generate responses, score those responses, and feed the results into the next update. Each stage needs different compute, and connecting them across separate tools and clusters adds work. SageMaker AI already offers multi-turn RL as a managed, serverless technique. But we heard from customers who wanted to define their own loss functions, shape reward signals, or RL stages in one loop. Training sessions give you that flexibility. “With SageMaker AI Serverless Training Sessions, we fine-tuned gpt-oss-20b for agentic, multi-turn tool calling against our Amazon Neptune knowledge graph — and the platform handled the infrastructure for us. Setup just worked, with no manual tuning and no out-of-memory wrangling. Checkpoint-and-resume was a standout: resuming from an exact step was completely reliable. It gave our developers the flexibility they wanted while taking the operational burden off their plate.” — Nitin Wagh, CEO, Karini AI How it works Your code runs the training loop and reward function in your environment and connects the agent to its tools or simulators. The training session performs the GPU-heavy work: a sampler generates responses, and a trainer calculates gradients and updates the model weights. The exact sequence depends on the algorithm you write. For example, let’s look at Group Relative Policy Optimization (GRPO) implementation. GRPO is a reinforcement learning method that generates several responses to the same prompt, scores them, and uses their relative performance to update the model. One pass through the loop works like this: Sample responses – Your loop asks the sampler to generate several responses for each prompt using the current model. Score them – Your reward function scores each response in your environment using a verifier, reward model, or test harness. Update the model – Your loop sends the scores back to the trainer. forward_backward() uses them to calculate gradients, and optim_step() applies the update to the model weights. Refresh and repeat – Your loop asks the session to refresh the sampler with the updated weights, then starts the next batch. Your code repeats these steps until training is complete. You pay for the tokens SageMaker AI processes. The meter stops while your reward function scores a batch in your environment, and there is no capacity to reserve. Keep control of your data and models Each session uses an AWS Identity and Access Management (IAM) execution role that you create, and you can add your own AWS Key Management Service (AWS KMS) key and virtual private cloud (VPC) configuration. Your data stays in your own Amazon Simple Storage Service (Amazon S3) buckets in the AWS Region that you choose, and your checkpoints land there too. We don’t use your data or your customized weights to train another model. Before deployment, you can evaluate the model for accuracy, bias, and safety. After you deploy, you can apply guardrails and monitor it with Amazon CloudWatch. You can keep using the governance processes your teams already have. Get started today Write your own training loop on SageMaker AI serverless model customization training sessions, available today with support for gpt-oss-20b , gemma-4-31B-it , Qwen3.6-27B , and Qwen3.8-27B . Two ways to get started: In your IDE: Install the SageMaker Python SDK (on Linux or macOS with Python 3.10 to 3.12), clone the cookbook, and start developing. In SageMaker Studio: Launch a ready-to-use AI development environment in under 20 seconds. Sign in, open a notebook, connect your coding agent (Kiro, Claude Code, Codex, or Copilot), download the agent skills and cookbook, and you’re ready to begin. Resources Read the full technical post Training sessions examples Training sessions documentation Training sessions cookbook About the authors Pooja Karadgi Pooja is a Senior Product Manager at AWS on the SageMaker AI team, where she works on developer experiences and model customization, including reinforcement learning. Her focus is making advanced fine-tuning techniques practical and accessible for builders. Sumedha Swamy Sumedha is a Senior Manager of Product Management at Amazon Web Services (AWS), where he leads several areas of the Amazon SageMaker, including SageMaker Studio – the industry-leading integrated development environment for machine learning, developer and administrator experiences, AI infrastructure, and SageMaker SDK. Jason Berkowitz Jason is the Head of Product Marketing for model training and inference at Amazon Web Services (AWS), where he leads positioning, messaging, and marketing for SageMaker AI. He brings over 25 years of experience in analytics and AI building, deploying, and taking AI products to market.

  2. AWS Machine Learning Blog

    Write your own training loop — without managing GPUs — with Amazon SageMaker training session APIs

    Fine-tuning a large language model (LLM) today usually forces a tradeoff between training loop control and operational simplicity. With fully managed services you select a base model, provide data and hyperparameters, and the service owns everything else, which works well when a standard, prepackaged recipe fits your problem. Less managed services give you more control, but you also manage more of the underlying infrastructure and training frameworks. Sometimes a standard recipe isn’t enough: you need a custom loss or data curriculum for supervised fine-tuning (SFT), your own preference optimization, a reinforcement learning (RL) reward function, or an alignment algorithm published last week. As soon as you need to shape the training procedure yourself, you take on that infrastructure work to gain algorithmic freedom. Today, serverless model customization in Amazon SageMaker AI expands with training session APIs, which remove that tradeoff by drawing the abstraction boundary one level lower. Instead of abstracting away the algorithm, they abstract away the compute. Operations like forward_backward() , optim_step() , and sample() become SageMaker API calls that you compose into whatever procedure your technique requires: SFT, preference optimization, RL, or something new. The calls target a live, stateful session, so the model stays loaded on the GPUs across steps, and you drive as many calls as you need against it. The service handles runtime setup, distributed execution, gradient synchronization, fault recovery, tenant isolation, and the operations of running GPUs at scale. You write the training loop, and SageMaker AI runs the infrastructure. Alongside the service, we’re releasing the SageMaker training sessions cookbook , a set of ready-to-run SFT and RL recipes that you can use as blueprints for your own training loops. In this post, we introduce the programming model and the three SDK clients, and show where training session APIs fit among the model training and customization options in SageMaker AI. We then take apart a minimal RL loop on grade-school math problems (GSM8K) to show how the operations fit together and finish with a real-world workload from the cookbook: reinforcement learning on competitive programming problems, where generated code is graded by running it against unit tests. What are SageMaker AI training session APIs? With SageMaker AI training session APIs, you can drive a live, GPU-backed session from your own Python process on your local machine. You call the operations, and SageMaker AI runs them on managed accelerators loaded with your chosen base model plus your Low-Rank Adaptation (LoRA) adapter. There’s no config file describing the whole job up front and no cluster creation step. You control what happens at each step, interactively, from code, so you can iterate and debug quickly. The division of responsibilities looks like this: You focus on You write SageMaker AI handles Your data, RL environments, agents, and algorithm A Python training loop that calls operations such as forward_backward() , optim_step() , and sample() for the steps that need GPUs GPU allocation, model and LoRA loading, distributed execution, fault recovery, checkpoint storage, security The API is organized around three clients in the SageMaker Python SDK , created in sequence. A ServiceClient holds the platform-level settings (execution role, model package group, Amazon Simple Storage Service (Amazon S3) output location, and optionally an AWS Key Management Service (AWS KMS) key and virtual private cloud (VPC) configuration, which you can also override per session) and starts a session. The ServiceClient instantiates a TrainingClient (training and checkpointing operations such as forward , forward_backward , optim_step , and save_weights_for_sampler ), which in turn instantiates a SamplingClient ( sample and compute_logprobs , for generation and scoring during rollouts and evaluation). Each API call maps to one operation in your loop. An active session also serves an OpenAI-compatible chat completions endpoint, so existing OpenAI client code can sample from it. Figure 1: The SageMaker Python SDK client hierarchy for training session APIs For reinforcement learning , the loop has a characteristic shape. The following figure shows one training step and where each part runs: the steps in green call APIs on service-managed GPUs, and the steps in purple, your loop control and reward logic, run on your machine. Figure 2: One training step, with sampling, gradient, and checkpoint operations on managed GPUs and loop control and reward logic on your machine Each step loads a batch of prompts and (1) samples from the current model on the GPU. Your Python code then (2) scores those outputs: a verifier, a reward-model call, a test harness, anything you can express in code. (3) forward_backward and optim_step turn the scores into a gradient update on the GPU, and (4) save_weights_and_get_sampling_client() refreshes the sampler so the next step draws from the updated model. You own the data, the loop control, and the reward; SageMaker AI runs the GPU-bound sampling, gradient, and checkpoint operations. The same primitives cover supervised fine-tuning (SFT) and other LLM training techniques . For SFT, drop the sampling and reward steps and call forward_backward with a cross_entropy loss over labeled examples. Who is this for and where does it fit within Amazon SageMaker AI? Training session APIs are the right starting point when you need fast iteration cycles and algorithmic freedom, such as a custom loss, a new RL reward signal, or a multi-stage pipeline, but don’t want to provision or operate GPU infrastructure yourself. Usage is billed per token processed, so you’re charged only for the compute your loop consumes. If your reward function takes 10 seconds to score a batch on your machine, no GPU time is billed while it runs. Compared with hourly GPU billing, this has two practical advantages. First, there’s no idle-GPU cost: reward computation, data loading, logging, and debugging happen off the clock, and you’re billed only when you call a GPU-bound operation. Second, cost scales with your dataset and generation volume, not with wall-clock time or cluster size, which makes spend easier to forecast and experiments easier to compare. The savings over reserving GPUs for the full session can be significant for workloads with heavy local computation, such as complex reward models, multi-tool agentic rollouts, or human-in-the-loop evaluation. Training session APIs are built for teams whose work is defined by the training algorithm, not only the dataset. Because you compose the loop yourself, the same primitives support supervised fine-tuning (SFT) with a cross_entropy loss over labeled data, reinforcement learning with ppo , importance_sampling , or cispo (GRPO, PPO, and other group-relative methods), and custom objectives , such as preference optimization or distillation, built from the same pieces. They fit: Post-training and alignment researchers implementing RL methods (GRPO, PPO, importance-sampling variants), custom advantage estimators, or new objectives. Teams doing supervised fine-tuning or preference optimization who need a non-standard loss, a data curriculum, or a multi-stage pipeline that config-driven recipes don’t express. Agentic and multi-turn RL builders who need custom rollout logic, reward shaping, and curricula. Machine learning (ML) engineers who’ve outgrown config-driven fine-tuning but don’t want to own the operational burden of GPU clusters. Researchers porting existing training-loop code, since the operation-based programming model maps directly onto loops written against low-level forward_backward , optim_step , and sample APIs. The supported base models and the AWS Regions where each is available are listed in the training session APIs documentation . Where training session APIs fit within Amazon SageMaker AI Training session APIs sit at a distinct point among the model customization options in SageMaker AI: substantially more control over the training algorithm than fully managed Serverless Model Customization , and far less operational ownership than training on Amazon SageMaker Training Jobs or Amazon SageMaker HyperPod . Option Control over training Infrastructure SageMaker Serverless Model Customization – Managed APIs Low – pick a model, configure available hyperparameters and supply data Serverless, no instance selection SageMaker Serverless Model Customization – SageMaker AI training session APIs High – you write the loop, call primitives Serverless, no instance selection SageMaker Training Jobs Full Managed, ephemeral jobs where you select instance type SageMaker HyperPod Full Managed, persistent clusters with advanced fault recovery Figure 3: Where training session APIs fit among the model customization options in SageMaker AI Anatomy of a training loop: GRPO on math problems This section takes apart a minimal reinforcement learning loop to show how the operations fit together. The task is GSM8K , a benchmark of roughly 8,500 grade-school math word problems, each with a single numeric answer, as in the following example: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? → 72 We train a LoRA adapter for OpenAI’s GPT-OSS 20B with Group Relative Policy Optimization (GRPO): for each question the model generates several responses, each response is scored for correctness, and responses that beat the group’s average are reinforced. The complete, runnable example is in the Amazon SageMaker examples repository . Before you open a session, complete the prerequisites in the documentation. Open a session Create a session, then get the tokenizer and a SamplingClient from it: from sagemaker.train.training_session import ServiceClient, SamplingParams, AdamParams svc = ServiceClient(role_arn=role, region=REGION) training_client = svc.create_lora_training_client( "openai/gpt-oss-20b", # a name from svc.list_base_models() training_session_name=SESSION_NAME, rank=32, alpha=64, # LoRA rank and scaling factor model_package_group_arn=MODEL_PACKAGE_GROUP_ARN, s3_output_path=OUTPUT_S3_PATH, ) tokenizer = training_client.get_tokenizer() sampling_client = training_client.create_sampling_client() Define the reward and the advantage The task-specific logic is two short functions that run on your machine: def compute_reward(response_text, ground_truth): predicted = extract_answer(response_text) # parse the number after "#### " return 1.0 if predicted == ground_truth else 0.0 def compute_advantages(rewards): rewards = np.array(rewards) advantages = rewards - rewards.mean() # center within the group (the GRPO baseline) return advantages / (rewards.std() + 1e-8) # normalize The loop Each step follows the four stages shown in Figure 2. draw , build_prompt , and build_training_datum are short helpers in the sample: for step in range(NUM_STEPS): datums, pending = [], [] # (1) sample K responses per question in one request each (GPU); returns a future right away for question, ground_truth in draw(PROMPTS_PER_STEP): # data loading is local prompt = tokenizer.encode(build_prompt(question), add_special_tokens=False) pending.append((prompt, ground_truth, sampling_client.sample( prompt, num_samples=K, sampling_params=SamplingParams(max_tokens=512, temperature=0.7, top_p=0.9)))) for prompt, ground_truth, future in pending: result = future.result() # wait once the batch is in flight # (2) score each response, then center the rewards within the group (local) rewards = [compute_reward(tokenizer.decode(s.tokens), ground_truth) for s in result.sequences] advantages = compute_advantages(rewards) datums += [build_training_datum(prompt, s, a) for s, a in zip(result.sequences, advantages)] # (3) one gradient step over the batch (GPU); the optimizer step is sequenced behind it fwd_bwd_op = traini

  3. arXiv · 场景与世界生成

    Reading, Writing, Sharing, and Interacting with the World through Video

    Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.

  4. arXiv · 资产数据集与许可

    Gate3DGen: A Pre-processing Model that Selectively Purifies Real-World Images for 3D Generation

    Single-image 3D generation has achieved high fidelity on curated benchmarks, yet degrades sharply on real-world images with uneven illumination, blur, and occlusions. Purifying such inputs requires deciding when to intervene and preserving the visual cues needed for 3D generation, while leaving clean inputs unchanged. We introduce Gate3DGen, a selective pre-processing model that gates image inputs to existing 3D generators without retraining them. A degradation-aware router bypasses inputs classified as clean and extracts features to guide purification of the remaining inputs. Conditioned on these features, a 3D-oriented one-step purifier purifies degraded inputs with spatial losses that encourage preservation of object identity, contours, and textural cues. To provide paired supervision, we construct a synthetic data engine that renders clean views of 3D assets and synthesises composite degradations. On the synthetic benchmark, the purifier improves image fidelity and downstream 3D geometry in only 2.3s on average, reducing TRELLIS.2 mesh distance by 61.5% relative to degraded inputs. VLM evaluation and human preferences on real photographs further support identity preservation and degradation correction. More demos and code are available at https://supra-3d.github.io/gate3dgen.

  5. arXiv · 资产数据集与许可

    UniVOE: A Removal-Driven Framework for Interaction-Aware Universal Video Object Editing

    Video object removal and injection are reciprocal editing operations between object-present and object-absent video states, yet they differ substantially in training difficulty. Removal can be effectively bootstrapped from video inpainting and further adapted to erase object-related interaction effects. Injection, however, requires large-scale paired supervision to model object appearance, dynamics, and scene-dependent effects, while perfectly aligned real-world pairs are difficult to collect and synthetic data lacks sufficient realism and diversity. To address this challenge, we present UniVOE, a removal-driven framework for training both video object removal and injection models. We first train an effect-aware remover by transferring video inpainting priors and finetuning on object-present/object-absent pairs. The trained remover converts real object-present videos into object-absent counterparts, creating large-scale paired supervision for injection. An injection model is trained on these pairs in the reverse direction to recover objects together with their dynamics and interaction effects. Moreover, since pixel-level reconstruction alone cannot adequately capture perceptual realism, we introduce a reward-based post-training method that aligns injection results with human preferences using a proprietary reward model. Experiments against strong open-source and commercial systems demonstrate that UniVOE achieves state-of-the-art performance on both object removal and injection, particularly in challenging interaction-effect cases.

  6. arXiv · 场景与世界生成

    WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models

    Video diffusion models have learned rich 3D/4D priors, making them promising for scene generation and video re-rendering. However, their use in spatial tasks is limited by imprecise camera control, spatial and temporal inconsistencies, and entangled camera motion with scene dynamics. Existing approaches, such as task-specific fine-tuning and post-process warping, often introduce visual artifacts, generalize poorly, or incur high computational costs. We propose WorldForge V2 to achieve accurate camera control while preserving the generative capabilities of pretrained video models. We first develop a training-free guidance framework using warped references to iteratively refine generation along a target camera path. Optical-flow analysis identifies motion-related latent channels for selective guidance, helping separate motion control from appearance. A dual-path strategy further compares guided and unguided predictions to correct artifacts from imperfect references. Together, these components guide viewpoint changes while preserving scene appearance and object motion. To reduce refinement costs, v-cache approximates the correction using the preceding denoising prediction, avoiding an additional model evaluation. We further extend motion localization from latent channels to network blocks and introduce Trajectory LoRA, which concentrates adaptation on blocks most responsive to trajectory guidance. A two-stage training strategy teaches these adapters to denoise the states encountered during guided generation, using limited supervision effectively. This enables reliable camera control with very little training data, with a single training scene already supporting generalization to unseen scenes. Quantitative and qualitative evaluations across multiple datasets show that our method achieves state-of-the-art generation quality and trajectory accuracy in camera-controlled video generation.

  7. NVIDIA Developer Blog

    How to Read Data into GPU Memory in Python with NVIDIA GPUDirect Storage

    GPU applications are processing ever-growing datasets with increasingly powerful accelerators. According to the 2025 Stanford AI Index, compute for notable AI... GPU applications are processing ever-growing datasets with increasingly powerful accelerators. According to the 2025 Stanford AI Index, compute for notable AI models has been doubling approximately every five months, while the size of datasets used to train LLMs has been doubling approximately every eight months. As computation and data scale together, moving data efficiently from storage becomes… Source

  8. arXiv · 可控视频与镜头

    Self-Supervised Multibeam Sonar Restoration via Coherent Forward Modeling

    Multibeam imaging sonar produces intensity images degraded by coherent speckle and system-induced blur, while clean reflectivity targets are rarely available. Existing despeckling methods often rely on simplified intensity-domain noise models, synthetic clean targets, or blind-spot self-supervision, and typically do not jointly model deblurring and speckle suppression. We propose a physics-consistent self-supervised framework for joint despeckling and deblurring of multibeam imaging sonar. A stochastic coherent image-formation model yields a conditional-mean forward operator that relates latent scene reflectivity to expected intensity through the squared magnitude of the system point spread function. We then use registered multi-frame sonar sequences to train without clean ground truth images, showing that, under an idealized inter-frame conditional-mean condition, the expected paired loss equals the corresponding forward-prediction loss up to a network-independent constant. To stabilize the ill-conditioned inverse problem, we further adapt robust equivariant imaging using stochastic coherent simulation and introduce a paired forward prediction error metric for ground-truth-free evaluation. Experiments on two real multibeam sonar datasets demonstrate a favourable balance between forward-prediction agreement, homogeneous-region speckle suppression, and preservation of visible structural boundaries compared with representative baselines.

  9. NVIDIA Developer Blog

    Running Frontier-Class AI Models Across 8 NVIDIA DGX Spark Nodes

    AI developers aren’t just running a single model anymore—they’re orchestrating multiple models in parallel, chaining inference calls, and experimenting... AI developers aren’t just running a single model anymore—they’re orchestrating multiple models in parallel, chaining inference calls, and experimenting with frontier-class architectures that demand hundreds of gigabytes of memory. NVIDIA DGX Spark was built to bring AI development to the desk, and now you’ll be able to link even more Spark nodes together. At GTC Berlin… Source

  10. Microsoft Research

    Flint-interactive: A language for creating interactive data visualizations

    At a glance Brings rich interactivity to AI-generated charts: Flint-interactive enables agents to create data visualizations that users can explore directly through clicking, filtering, zooming, highlighting, and cross-chart interactions. Makes complex interactions easier to build: Instead of requiring low-level visualization code, Flint-interactive lets users and agents define behaviors using high-level semantic events and update operators. Improves reliability and reduces development effort: By using Flint’s compiler to handle interaction logic and visual dependencies, the system helps avoid many of the layout, rendering, and debugging challenges common in custom interactive visualizations. Connects charts, text, tables, and applications: Flint-interactive supports interactive data stories, dashboards, and connected experiences where actions in one view can update charts, documents, widgets, or application state in another. AI agents make it easy to analyze data and generate reports with data visualizations. They help users examine patterns more efficiently than by reading verbose text. However, when we’re feeling exploratory, we want to go beyond static visualizations. : We want to interact with charts to zoom in on details, compare data across charts, or even highlight related text in the document from a chart selection, and vice versa. This requires agents to be able to create interactive data visualizations that communicate information while letting readers explore beyond the author’s chosen view, ideally with the quality of interactive stories in sophisticated data journalism. This leaves us wondering: (As a user) Can I interact with charts directly in chat to explore the data and ask follow-up questions based on what I select? (As a developer) Can I enable the AI agents I build for my customers to generate interactive visualizations that help them explore and understand their data? Yes, it’s possible, but there is a significant practical gap. Even with capable frontier models, we still face: High generation cost. Agents rarely produce a fully satisfactory interactive visualization in one turn. Repeated rounds of reasoning, coding, and debugging consume tokens and time. Reliability. Interactive visualizations are challenging enough that expressive interactive charts introduce poor layouts or broken interactions easily, and we risk introducing new problems when trying to fix them. These challenges help explain why consistently generating high-quality interactive visualizations remain difficult in real-world use. Meanwhile, built-in interactions in some visualization libraries are not expressive enough on their own to create bespoke interactive charts. And these challenges are results of the lack of an expressive yet high-level language to specify visual interactions. Flint-interactive Flint-interactive (opens in new tab) is a language for creating interactive data visualizations on top of Flint. It allows users to create dashboards, data stories, and interactive artifacts more easily. The power of Flint-interactive comes from Flint’s compiler, which translates user interactions into semantic events that capture the meaning behind what users select. It then uses high-level operators to update visualizations in response to those events, making sophisticated interactions far easier to create and maintain. This shifts the heavy burden of resolving complex dependencies between interaction and visual logic to the compiler, so the user (or the agent) can easily describe expressive interactions with high-level specifications. Example 1: We can enable the user to brush one visual view, and the chart highlights all groups of country selected by the user with a simple preset interaction linked-brush. How to use 1. Use Flint in your chat Use Flint Interactive directly in your chat or canvas environment through Flint-MCP and MCP Apps, following the MCP setup guide. With the latest release, tell your agent what to chart and which interactions you want; it can generate an interaction_spec alongside the Flint chart spec and render the interactive chart in your conversation. Example 2: Using Flint-interactive in the chat environment, allowing the agent to produce interactive charts more reliably with simple Flint specifications. It also enables the user to select and pass context to the follow-up turns. 2. Build an interactive app with Flint Build interactive charts in a dashboard, data story, or other application with Flint Interactive Canvas. Your app can use event handlers to process semantic events emitted by Flint Canvas, for example to filter a table, update another chart, or change application state. To let your app’s widgets control how the chart updates, send update operators to Flint Canvas. Example 2: Flint-interactive allows developers to define interactions between charts and external elements (e.g., table, paragraph). Here, the user can click the table to highlight points in the chart and verse versa. 3. Create bespoke interactions To create bespoke interactions beyond Flint’s presets, define a trigger and a custom handle function with the programming API that maps semantic events to custom chart update sequences. For example, a handler can highlight a clicked country and label it by name. It combines set-style and set-annotation without manipulating renderer-specific marks. Example 4: The agent can create bespoke interactive visualizations with Flint’s programming API. In this DimpVis example, the user can drag the point along the time path to visualize how all cities’ temperature and precipitation changes. How it works Flint Canvas is a state machine that tracks user interactions on the canvas and manages visual rendering state. Flint makes interaction specification easy by separating visual logic, managed by the compiler, from interaction logic, specified by the user at the semantic level: Visual logic: The compiler automatically resolves raw triggers, such as clicks, drags, and key presses, into semantic events that identify chart elements and data. It also translates update operators into rendered canvas changes, handling hit testing, and low-level visual updates. Interaction logic: The user specifies interactions by mapping semantic interaction events to chart update operators, such as set style to highlight data, set annotation to label a point, or set viewport to pan or zoom. The user can also use those semantic events to update external tables, widgets, or application states. Figure 1: Flint interactive allows the agent to describe interaction logic by mapping “what semantic event is the user interacting with” to “what updates they want for the canvas surface”. The compiler takes the heavy lifting of resolving visual updates and manages the runtime, which makes interaction authoring more reliable c and cost effective. This separation means users do not need to reverse-engineer rendering logic or manipulate renderer-specific marks, making interactions easier to specify and maintain. Furthermore, because event generation and visual updates are compositional, the same model also connects Flint with tables, text, other charts, and widgets to build interactive stories and dashboards. Try it now! Chart users: Set up Flint in your chat (opens in new tab) and ask your agent to create an interactive chart. Find inspiration in the interaction gallery. App builders: Build interactive dashboards, data stories, and connected views with the programming API (opens in new tab) . Share the interaction-author skill with your coding agent to add presets to your app. Interaction designers: Explore the bespoke examples (opens in new tab) , design a new interaction, and turn it into a reusable preset. Propose a preset on GitHub and share your prototype or implementation to help expand Flint’s interaction vocabulary. Opens in a new tab The post Flint-interactive: A language for creating interactive data visualizations appeared first on Microsoft Research .

10月22日周四
  1. arXiv · 创作 Agent 与装配

    A Sparse Topometric Graph for Scalable Cross-Session Indoor Navigation

    A conventional embodied agent in an indoor environment builds a map by accumulating noisy real-world measurements and results from heavy inference to guide navigation. While the detailed measurements may differ due to scene changes or sensor settings, the unique topological structure of navigable passages or the connectivity between rooms may provide training-free guidance that complements local observations. We propose a sparse topometric graph as an efficient structural representation for robust indoor navigation, alleviating the need for fine-grained matching, which is computationally expensive, memory-intensive, and prone to real-world variations. The map consists of nodes that encode local 3D geometry and distance-weighted edges that capture traversable connectivity. A learned graph-matching network finds consistent subgraph structures across sessions by considering both node-level geometric descriptors and neighborhood connectivity. The holistic contextual matching is robust to local ambiguities caused by weak distinctiveness or repetitive structure, and establishes stable correspondences despite local scene changes, different sensor modalities, or limited spatial overlap. We analyze the practicality of the graph through cross-session point-goal navigation, where a goal specified in a prior reference map must be reached from an unknown initial pose in a later session. Experiments show that our method achieves the highest success rates (77.1% on Gibson and 40.0% under 3D-FRONT scene changes), while reducing memory by more than two orders of magnitude compared to point-level representations. The proposed approach further supports cross-session alignment and goal-directed navigation with strong scalability and robust real-world transfer.

  2. NVIDIA Blog

    NVIDIA Unveils Minerva to Bring End-to-End Observability for AI Factories

    Launched today, NVIDIA Minerva is an AIOps platform built to help teams find problems earlier and keep AI factories productive. It brings together telemetry from infrastructure and AI workloads, then applies purpose-built AI models to turn that data into timely, actionable insight. AI factories are among the most complex operations in the world, with vast numbers of GPUs working across high-speed networks to train models and serve AI applications. At that scale, a single degraded component can slow an entire job. The warning signs may be easy to miss, such as a GPU running hotter than its peers or an NVMe drive losing performance under load. The platform focuses on AI factory yield, a measure of the useful work infrastructure produces through GPU hours, goodput, cluster uptime and token production. Read All NVIDIA in Europe Announcements Delivering a Dynamic View of an AI Factory Traditional monitoring tools often divide the data center into separate views. Networking, compute, site reliability and machine learning teams each see a different part of the operation. Minerva connects those signals across NVIDIA GPUs, networking, storage, schedulers and workloads. With these signals, Minerva builds a dynamic knowledge graph that logically maps the infrastructure — GPUs, SuperNICs, switches and cables — and the jobs running on it. This context helps operators trace a slow training job to the component affecting it, even across a factory with hundreds of thousands of GPUs. It analyzes billions of data points per hour using supervised and unsupervised machine learning. It learns normal behavior, detects previously unseen anomalies, predicts emerging failures and correlates related events based on the factory’s topology. Instead of sending every symptom to an operator, it can group an event storm into a focused incident with a likely root cause. Minerva comes with a native AI assistant and connects to external agentic AI tools. This adds a natural-language interface grounded in live telemetry. Operators can investigate cluster health and receive recommended remediation through conversation. Teams can keep a person in the loop to manually implement fixes or connect Minerva to automation through infrastructure application programming interfaces and external systems. Results That Maximize AI Factory Yield The ability to correlate workload and infrastructure issues along with its agentic AI capabilities help simplify AI factory operations. Instead of dealing with a blizzard of alerts, Minerva aggregates related problems into a single actionable item. This allows AIOps teams to move from reactive to proactive maintenance. And all of this helps AI factories maximize GPU uptime. Minerva delivers value for the key metrics AI factories focus on: Longer time between interruptions : Minerva spots trouble before it happens, so jobs run longer without incident. Faster time to repair : When an incident does happen, Minerva pinpoints the problem, leading to a significantly quicker resolution. Greater insight : Modern AI infrastructure generates a flood of data; Minerva filters it out, leaving only the handful of alerts that actually matter. IREN is among the NVIDIA partners using Minerva to optimize its AI factories. “As we scale our AI factories, the key to providing the best service for our customers is maximizing the productivity of our GPUs,” said Denis Skrinnikoff, chief technology officer of IREN. “NVIDIA Minerva gives our operations team one view across the network, the GPUs and the jobs running on them, so we focus more on proactive maintenance rather than troubleshooting. That visibility strengthens our operating model for the infrastructure we’re building next.” Joining IREN , Aolani , Bitdeer AI , IBM , Jio, Lambda , Quake AI and Tesla plan to deploy Minerva. In addition to Dell Technologies , Lenovo and Supermicro will offer Minerva as part of their AI factory operations portfolios. Learn more about NVIDIA Minerva . See notice regarding software product information. Read All NVIDIA in Europe Announcements

  4. arXiv · 渲染、神经场与导出

    RecallGS: Geometry-Grounded Recall for Online Semantic Gaussian Splatting

    Online semantic 3D Gaussian Splatting (3DGS) enables open-vocabulary scene understanding from streaming observations. However, existing methods recover geometric context missing from image-derived semantic features using an additionally learned 3D network, incurring extra training and runtime overhead. We observe that such geometric-semantic evidence may already exist in the agent's observation history and can be retrieved rather than learned. Based on this insight, we introduce RecallGS (Geometry-Grounded Recall for Online Semantic Gaussian Splatting), a lightweight, training-free mechanism that treats past observations as a non-parametric geometric-semantic prior. RecallGS follows a recall-ground-fuse pipeline: CLIP retrieves relevant observations, alpha-compositing weights identify geometrically supported views, and aggregate their semantic features over each Gaussian's rendered footprint. On ScanNet and ScanNet++, RecallGS achieves strong semantic segmentation performance at 2.48 FPS, demonstrating that geometric-semantic context can be effectively recovered from observation history without an additional learned 3D branch.

  5. GitHub · WhiteTowerAI/ohmygame

    WhiteTowerAI/ohmygame 发布 OhMyGame v0.0.0-beta.4.12

    ### Added - Connect external Claude Code, Codex, and other MCP clients to OhMyGame's media, canvas, project, and playtesting tools through Settings → MCP Server. ([#325](https://github.com/WhiteTowerAI/ohmygame/pull/325) by @dogxii) - Explore optional Beta plugins for Blender, Godot, TapTap Maker, Cocos Creator, Unity, Unreal Engine, Aseprite, Tiled, WeChat Mini Games, and Douyin Mini Games. Enable them from the sidebar's Plugins library; these workflows remain experimental. ([#278](https://github.com/WhiteTowerAI/ohmygame/pull/278) by @dogxii) ### Changed - Set up a model provider or installed coding agent during first-run onboarding, choose a default model, and optionally connect creative tools before starting. ([#331](https://github.com/WhiteTowerAI/ohmygame/pull/331) by @dogxii) - Browse Community in a responsive cover grid with visible titles, creators, engagement counts, search, sorting, and matching cards on creator profiles. ([#357](https://github.com/WhiteTowerAI/ohmygame/pull/357) by @dogxii) - Publish Interactive Stories up to 100 MiB, with a longer upload timeout and media optimization in a temporary copy that preserves the original assets. ([#333](https://github.com/WhiteTowerAI/ohmygame/pull/333) by @82228) - Connect Codex Image using its saved sign-in without an extra capability check, and simplify provider detail controls. Availability and quota errors are reported when generating an image. ([#356](https://github.com/WhiteTowerAI/ohmygame/pull/356) by @dogxii) - The Windows installer is shown in English or Simplified Chinese, matching the app's languages; other system languages get English. ([#377](https://github.com/WhiteTowerAI/ohmygame/pull/377) by @dihuangdh) ### Fixed - Preserve recoverable canvas drafts when concurrent changes would merge into invalid node references, and offer conflict choices instead of losing edits. ([#342](https://github.com/WhiteTowerAI/ohmygame/pull/342) by @jsdhwfmax) - Avoid a Windows checkpoint write failure when multiple files have the same contents and are saved at the same time. ([#373](https://github.com/WhiteTowerAI/ohmygame/pull/373) by @xinyang6879) - Keep the previous story picture visible while the next node's media loads, reducing black flashes between story nodes. ([#343](https://github.com/WhiteTowerAI/ohmygame/pull/343) by @82228) - Improve Windows title bar and sidebar appearance, keep installer progress moving forward, allow more time for first-start plugin preparation, and prevent a failed process check from incorrectly blocking installation. ([#352](https://github.com/WhiteTowerAI/ohmygame/pull/352) by @82228, [#369](https://github.com/WhiteTowerAI/ohmygame/pull/369) by @82228, [#377](https://github.com/WhiteTowerAI/ohmygame/pull/377) by @dihuangdh) - Keep composer attachment and context menus usable near window edges and in short viewports, with scrolling when the full menu does not fit. ([#352](https://github.com/WhiteTowerAI/ohmygame/pull/352) by @82228, [#361](https://github.com/WhiteTowerAI/ohmygame/pull/361) by @xinyang6879) - Make plugin Skills available to installed coding agents and prevent Codex startup failures caused by reserved tool names. ([#278](https://github.com/WhiteTowerAI/ohmygame/pull/278) by @dogxii) - Remove the confusing Off map badge from scene cards while keeping visibility controls available. ([#346](https://github.com/WhiteTowerAI/ohmygame/pull/346) by @82228)

  6. arXiv · 可控视频与镜头

    QICOM: Open-Source Software for Reproducible Image Quality Evaluation, Blinded Assessment, and Radiomic Comparison

    QICOM (Quality Image COMparator) is open-source software for quantitative image comparison, paired radiomics, and blinded assessment. It saves processing settings, image identities, measurements, and response mappings so a comparison can be reopened and recalculated. We evaluate reference agreement, responses to controlled perturbations, and recovery of saved analyses. Synthetic experiments cover 37 quantitative descriptors using independentnumerical references and separate upstream integration comparisons. The QICOM/MIRP backend meets the reference tolerances for all 482 numeric IBSI 1 digital-phantom entries and all 1,622 entries across configurations A-E of the public CT reference case; these comprise 1,804 feature/aggregation comparisons and 300 preprocessing diagnostics. Restricting masked SSIM and HFEN to complete kernel support removes the observed dependence on excluded pixels. In a synthetic filtering example, stronger smoothing increases SSIM while worsening MSE and HFEN relative to milder smoothing and changing GLCM contrast more strongly. Reopened radiomics projects reproduce all 43 feature pairs per comparison, and two blinded deliveries return four correctly mapped scripted responses. These experiments document the calculations and data transfers linking the three workflows.

  7. NVIDIA Blog

    Cloud Deployment Incoming: ‘Call of Duty: Modern Warfare 4’ Arrives on GeForce NOW

    Lock in and load out this GFN Thursday. Call of Duty: Modern Warfare 4 arrives on GeForce NOW at launch, bringing a new campaign, multiplayer action and DMZ, high-stakes extraction mode to supported devices . The campaign also gets an enhanced look: GeForce RTX 5080-class performance arrives for Ultimate members, bringing support for ray-tracing features in the cloud. There’s a new way to join the cloud, too: Existing supported Fire TV Stick users can now purchase one-month Performance or Ultimate memberships directly through Amazon, making it easier to sign up and start streaming on the big screen. Plus, Square Enix’s FINAL FANTASY RESONANCE brings cinematic pixel art and strategic battles to the cloud, and The Elder Scrolls Online’s Echoes of Skyrim Season adds the entire Greymoor Chapter free for all ESO Standard Edition owners. It’s all part of six games joining the GeForce NOW library this week. Deployment Without the Downloads Squad up and answer the call on the cloud. Activision and Infinity Ward’s Call of Duty: Modern Warfare 4 launches today on GeForce NOW. Members can drop into the trenches in Korea, close-quarter combat in New York, high-octane chases through Paris, night raids in Mumbai and city-wide engagements to reclaim occupied territory. Fight as Private Park, a young South Korean soldier facing live combat for the first time alongside his squad as a routine patrol turns into a desperate fight for survival. Beyond the front lines, operator-turned-outlaw Captain Price is operating outside the system he once served, pursuing revenge through covert missions as the conflict spreads. Multiplayer pairs precise combat with fluid movement, a dozen core 6v6 maps from around the world and battlegrounds that are shaped by the campaign’s global conflict. The DMZ extraction mode sends players into contested territory to recover valuable technology, weigh objectives and decide when to get out. Loot, fight, negotiate, betray and extract — it’s all fair in the cloud and battleground. Ultimate members can now experience the campaign with ray tracing, powered by newly added GeForce RTX 5080-class performance across PCs, Macs, handhelds, mobile devices, browsers, supported TVs and more. With game downloads and installs handled by GeForce NOW, members can spend less time preparing a local rig and more time preparing the next loadout. A New Way to Play A new way to join GeForce NOW on Fire TV. GeForce NOW in-app purchase is rolling out to existing supported Fire TV Stick users, making it possible to purchase a membership directly through Amazon using an existing account. Transactions are processed through Amazon’s billing system, creating a more seamless path from setup to gameplay. GeForce NOW transforms Fire TV into a powerful PC gaming rig in the cloud. Members can stream supported PC games they already own or purchase new titles from digital stores including Steam, Epic Games Store, Ubisoft Connect, Xbox PC Game Pass, GOG and EA. The library includes thousands of ready-to-play games, including many of the world’s most-played titles and more than 100 free-to-play games. No downloads, installs, patches or updates stand between a new game and playtime. An Amazon Fire TV Stick, the GeForce NOW app and an Ultimate membership are all it takes to level up the living room with GeForce RTX 5080-powered gaming. A Pixel-Perfect Fantasy Classic roots. A new adventure. Square Enix’s FINAL FANTASY RESONANCE launches today on GeForce NOW, bringing the series’ classic roots into a cinematic pixel-art world. Travel with companions bound by fate and uncover the truth behind a force threatening to bring everything to ruin. Strengthen the party with Visions of beloved FINAL FANTASY characters, each bringing unique skills and abilities to battle. Dynamic camera angles and pixelized 3D models give familiar fantasy a fresh perspective. Exploit enemy weaknesses to stagger foes, break their defenses and seize extra turns before finishing the bonus phase with powerful Resonance attacks. Airships, crystals, espers and moogles bring familiar series favorites to a world filled with secrets to discover. GeForce NOW handles the game’s downloads and storage in the cloud, leaving more space on supported devices for everyday essentials. Save the inventory management for potions and Phoenix Downs. Step Into a New Season in Tamriel Familiar lands. Fresh reasons to return. The Elder Scrolls Online ’s second season , Echoes of Skyrim , launched Wednesday, Oct. 21, free for all ESO players and ready to stream through GeForce NOW. Season Two gives everyone who owns ESO Standard Edition access to its Skyrim content and the Antiquities treasure-hunting system. The Markarth downloadable content follows Monday, Nov. 9, rounding out the additions with two classic zones to explore. The Season also introduces Solo Dungeons, reimagining challenges previously designed for groups. Moon Hunter Keep and March of Sacrifices have been redesigned and rebalanced for players ready to tackle their dangers alone. Earn new rewards and take on new challenges at your own pace without needing to find a group. A new Tamriel Tome also launched Oct. 21, offering both free and premium battle pass reward tracks. In addition, members can look for the following titles joining the GeForce NOW library this week: We Were Here Tomorrow (New release on Steam , Oct. 20) FINAL FANTASY RESONANCE (New release on Steam , Oct. 22) Call of Duty: Modern Warfare 4 (New release on Oct 23, available in the Call of Duty app) CONTROL Resonant ( Epic Game Store ) Crimson Desert ( Epic Games Store ) Planet Zoo 2 ( Xbox app , supports XBOX Play Anywhere) Dates listed above reflect when games are released on their respective stores. GeForce NOW availability may vary, as games are onboarded after release and added throughout the week. Keep an eye on GeForce NOW channels and GFN Thursdays for availability updates on announced titles. Gamers who want to test out high-performance GeForce NOW cloud gaming can start with a day pass to try premium cloud gaming before committing to a membership. Better yet, the cost of the day pass can be applied toward a first monthly membership — making it easy to try GeForce RTX-powered cloud gaming with the latest releases, then level up for more. What’s on the mission board this weekend? Let us know on X or in the comments below.

  8. arXiv · 场景与世界生成

    Inside-JEPA: From Prediction to Relational Grounding in Latent World Models

    Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.

  9. arXiv · 创作 Agent 与装配

    HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

    Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.

  10. arXiv · 场景与世界生成

    Dual Exam: Controlled Evaluation of Multimodal Agents and World Models

    World Models (WMs) can support Multimodal Agents (MAs) through simulated experience, while Agent trajectories can inform World Model refinement. Evaluating this interaction requires separate evidence about action selection and state evolution. We introduce Dual Exam, which tests Agents under fixed environment response mechanisms and World Models under fixed action interventions across physical, digital, social, and scientific domains. The Agent track compares 10 configurations on 1,088 items in 15 reporting categories. The World Model track contains 400 cases across eight scenarios; 15 configurations have full coverage and one covers six scenarios. GPT 6 Astra leads the case-weighted Agent Index at 31.3, and GPT-Image-2 leads the full-coverage WM Overall at 21.7. These scores use different tasks and metrics and are not directly comparable. A 150-case pilot of one coupled pairing records zero task successes and motivates component-level diagnosis. Results reveal uneven task performance and failures to preserve task-relevant state. Their value for predicting coupled success or sustained reciprocal learning remains to be established.

  11. arXiv · 可控视频与镜头

    Bound-Conditioned Latent Inference for Progressive Image Compression

    Progressive image compression reconstructs images from prefixes of a single bitstream and therefore involves sequential latent inference under partial observations. Existing trit-plane codecs mainly treat decoded trits as discrete symbols for probability prediction, without fully exploiting their role as interval constraints on the underlying continuous latents. We propose bound-conditioned latent inference (BLI), a framework that reformulates progressive trit-plane coding as sequential latent inference under shrinking interval constraints. At each coding step, the trits decoded so far define lower and upper bounds for each latent element. BLI uses the center and width of these bounds, together with hyperprior information shared by the encoder and decoder, to jointly refine the Gaussian parameters $(μ,σ)$ for entropy modeling and the latent estimate $\hat{y}$ for reconstruction. As additional trits are decoded, the feasible intervals shrink monotonically, yielding tighter constraints for the next inference step and enabling flexible in-plane refinement schedules. Experiments on Kodak, CLIC, and JPEG-AI show that BLI achieves state-of-the-art rate-distortion performance among progressive image codecs, reducing BD-rate by $17.78\%$ over DPICT and by $7.96\%$ over CTC on Kodak, while replacing CTC's plane-specific predictors with a shared model that uses about $5.3\times$ fewer parameters and achieves lower latency than CTC.

  12. arXiv · 可控视频与镜头

    GateFormer: Forecast-Space Decomposition for Intra-Hour Solar Irradiance Prediction from Sky Images

    Sub-hourly dispatch, balancing, and storage control increasingly depend on forecasts that anticipate rapid variability in solar generation. Accurate sky-image-based irradiance forecasting requires a combination of visual cloud dynamics with recent irradiance state while adapting to different short-term irradiance-change regimes. To address this, we introduce GateFormer, a GHI-conditioned dual-decoder ViT for intra-hour global horizontal irradiance (GHI) forecasting through forecast-space decomposition. GateFormer embeds observed GHI into the visual token stream, allowing interaction with sky-image patch features through self-attention. From this shared representation, a Trajectory Head predicts future GHI sequence, while a Delta Head predicts final-horizon change from the latest observed GHI. The two final-horizon forecasts are adaptively weighted in prediction space, with magnitude- and direction-supervision targeting the remaining error. On the NREL-TSI benchmark, GateFormer attains 125.01 W/m^2 RMSE at +30 min, improving over a state-of-the-art transformer baseline by 9.6% under paired-timestamp evaluation. Across NREL-TSI, Chilbolton, and SIRTA, GateFormer achieves the lowest final-horizon RMSE among the evaluated baselines, with forecast skill of 22.97-28.86% relative to smart persistence. Component ablations show adaptive weighting outperforms both individual decoders and fixed averaging, distinguishing GateFormer from conventional forecast combination, while the supervised correction further reduces RMSE by 1.68 W/m^2 and MAE by 4.86 W/m^2. Ramp-regime diagnostics show larger gains during operationally important positive ramps, reducing RMSE by 26.27 W/m^2. These results identify forecast-space decomposition as an alternative forecasting formulation, with potential relevance to other multimodal short-horizon forecasting tasks and deployment using only sky images and measured GHI.

  13. arXiv · 可控视频与镜头

    Vision Transformers for Estimating Irradiance Using Data Scarce Sky Images

    Accurate estimation of diffuse horizontal irradiance (DHI) is critical for optimising photovoltaic system performance and energy forecasting yet remains challenging in regions lacking comprehensive ground-based instrumentation. Recent advancements using Vision Transformers (ViTs) trained on extensive sky image datasets have shown promise in replacing costly irradiance measurement equipment, but the scarcity of long-term, high-quality sky imagery significantly restricts practical implementation. Addressing this critical gap, this study proposes a novel dual-framework approach designed for data-scarce scenarios. First, calculated atmospheric parameters, including extraterrestrial irradiance and cyclic time encodings, are integrated to represent sky conditions without utilising any instrumentation. Next, a sequential pipeline initially predicts synthetic global horizontal irradiance (GHI) and uses it as a feature, to refine DHI estimation. Finally, a dual-parallel architecture simultaneously processes raw and overlay-enhanced fisheye sky images. Overlays are generated through unsupervised, physics-informed cloud segmentation to highlight dynamic sky features. Empirical validation is performed using data from the Chilbolton Observatory, chosen for its temperate climate and frequent cloud variability. To simulate data-scarce conditions, models are trained on a single month (e.g., January) and evaluated across a temporally disjoint, full-year test set. Under this setup, the sequential and dual-parallel frameworks achieve RMSE values within 2-3 W/m^2 and 1-6 W/m^2, respectively, of a state-of-the-art ViT trained on the complete dataset. By combining physics-informed modelling with unsupervised segmentation, the proposed method provides a scalable and cost-effective solution for DHI estimation, advancing solar resource assessment in data-constrained environments.

  14. arXiv · 可控视频与镜头

    H-2DGS: Height-Field Gaussian Splatting for Sharp and Deformable Neurosurgical Surface Reconstruction

    Neurosurgical navigation commonly presents magnetic resonance imaging (MRI) as 2D slices, making it difficult to relate the anatomy in those slices to the cortical surface observed through the surgical microscope. We seek a common 3D visualization that combines a photorealistic reconstruction of the exposed cortex during tumor surgery with whole-brain MRI context and diffusion tractography pathways. This requires the reconstructed cortex to form a clear boundary around the brain anatomy rather than a diffuse volume that obscures internal structures. Conventional 3D Gaussian Splatting (3DGS) can reproduce observed views but, under limited surgical view coverage, may distribute opacity through a thick and geometrically ambiguous band. We introduce height-field 2D Gaussian Splatting (H-2DGS), a three-stage pipeline that concentrates a robust 3DGS reconstruction, fuses its depth into a shared truncated signed distance field (TSDF) surface, and represents that surface with curved, surface-oriented Gaussian primitives. Separate height, color, and opacity bases provide local geometry and appearance, while a neighbor-transition objective coordinates overlapping patches. The explicit surface further supports postoperative alignment: MRI-derived depth deforms the reconstruction, and tracked-color supervision with alternating optimization preserves its appearance. Across five real neurosurgical cases, H-2DGS gives low multi-view depth inconsistency, a compact 7.97MB representation, and interactive rendering at 29--42ms per frame. The resulting sharp, deformable surface integrates clearly with tractography and MRI anatomy, including a postoperative resection cavity.

  15. arXiv · 场景与世界生成

    DRILL: Training World Models to Improve Policies, Not Predict Pixels

    World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically trained to predict pixels one step ahead, even though their role here is not merely to render visually faithful videos but to produce learning signals from imagined rollouts that improve the policy. We show that this mismatch is substantial: lower pixel MSE can produce worse policy gradients, and a world model trained for policy utility can yield better downstream policies despite 17% higher pixel MSE on the next frame. We propose the Downstream-Return Imagination Learning Loop (DRILL), a bilevel framework that updates the world model to maximize the policy's return after an inner GRPO update on imagined rollouts. DRILL has two instantiations. DRILL-VWI is a closed-form surrogate, requiring no additional simulator rollouts, that reweights prediction errors by policy visitation, the group-relative GRPO advantage, and model confidence. DRILL-IMG is a full meta-gradient method that differentiates through the inner update using truncated Hessian-vector products. We show that DRILL-VWI recovers the local one-step component of the DRILL-IMG meta-gradient under score-compatibility and small-residual assumptions. Across five manipulation simulators and two VLA backbones, DRILL-IMG raises mean success rate by 13.1 points over WMPO and 7.9 points over RLVR-World, and complements RLVR pretraining rather than competing with it. More broadly, we believe video world models for VLA reinforcement learning can be productively trained by the policy gradients they produce, not only by pixel-level prediction.

  16. arXiv · 场景与世界生成

    An Introduction to World Models for Generalist Agents

    World models have recently emerged as a promising direction toward developing generalist agents capable of understanding, predicting, and interacting with diverse environments. However, despite growing interest, the concept of a world model remains loosely defined, encompassing approaches ranging from predictive dynamics models and learned simulators to models that jointly generate future observations and actions. This introductory paper aims to clarify the concept of world models and provide a unified perspective on their role in developing generalist agents. Starting from the partially observable Markov decision process (POMDP) framework, we connect world modeling to classical control, dynamical systems, signal processing, and reinforcement learning. We introduce different approaches to world modeling, emphasizing the knowledge they capture, their learning objectives, and how they support decision-making. We further explore the relationship between world modeling and policy learning, highlighting how these capabilities can be unified through conditional prediction without requiring separate models. Rather than offering an exhaustive survey of existing methods, this paper focuses on fundamental concepts, modeling principles, and connections, with the goal of providing an accessible and unified introduction to world models for generalist agents.

  17. arXiv · 场景与世界生成

    DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models

    Before acting on an object, a robot needs to anticipate how its hand motion will change the physical world and what the resulting scene will look like. Existing approaches typically entangle action-conditioned dynamics, camera motion, and visual synthesis within a video generation model, or represent dynamics as point trajectories without explicit object structure. We introduce DeXplicit, a dexterous manipulation world model that instead predicts how the world evolves explicitly in metric 3D before rendering its visual consequences. Given an initial scene and a future hand trajectory, a lightweight 3D dynamics network predicts object-centric motion and deformation, providing structured and geometrically grounded future world states. These predictions then serve as a strong scaffold for video generation: appearance is transported through the predicted 3D motion and projected into future views, allowing the renderer to focus on completing uncertain and newly visible content rather than implicitly learning world dynamics from pixels. This explicit-3D-first decomposition also enables efficient causal generation, requiring only two denoising steps per block while conditioning on generated history. Across HOT3D, OakInk2, and PhysTwin, DeXplicit reduces moved-point error by 27-40% relative to PointWorld and improves object fidelity IoU+ by 47% on HOT3D under a shared renderer. Our design with two-step causal generation achieves a 57x speedup from our 14B bi-directional configuration in video synthesis.

  19. arXiv · 资产数据集与许可

    Evaluation of Front-End Feature Matching for Lunar Visual SLAM

    Lunar rovers will primarily rely on cameras for localization until navigation satellite services are deployed, and will continue to need them afterwards for relative navigation, hazard detection, and surface operations. Visual odometry (VO) and Simultaneous Localization and Mapping (SLAM) mostly estimate motion by matching features between frames. However, feature matchers are developed on urban, indoor and driving scenes, planetary studies compare them on orbital registration or within complete SLAM systems, and real lunar imagery with reference poses is scarce, so matching between rover frames on low-texture, harshly lit terrain, and its effect on VO, has not been compared. We benchmark eleven classical and learned detector-matcher combinations on nine synthetic, lunar-testbed, and terrestrial-analogue datasets, and a feature tracker, scoring pairwise pose accuracy against baseline and the trajectory error of the resulting VO. Under a common rotation-only metric, the SuperPoint pairings outperform SIFT with ratio-tested FLANN matching on both lunar testbeds and one analogue site, with smaller or unresolved differences on synthetic data. When rotation and translation direction are scored jointly, this SIFT pairing exceeds both SuperPoint pairings on four synthetic datasets, so synthetic results alone do not establish relative performance on real rover imagery. In stereo VO, the SuperPoint pairings are not consistently more accurate than SIFT, but only they keep tracking on all eight sequential datasets. Pairings with classical detectors depend on lighting while SuperPoint changes little: a low spotlight makes them more accurate than diffuse light on the same testbed views, consistent with more contrast and detected keypoints. SuperPoint with LightGlue is the fastest learned combination (16 ms per frame), and DINOv2 retrieval proposals still need geometric verification.

  20. NVIDIA Blog

    Leading European Financial Services Organizations Embrace Open Models to Build Proprietary Intelligence

    At GTC Berlin, banks, payment companies and financial data providers are showing what production-scale intelligence looks like — built on NVIDIA accelerated computing, powered by open weight models and running on their own infrastructure. Revolut is rebuilding its technology stack on a single foundation model PRAGMA , powered by NVIDIA accelerated computing. Using the NVIDIA cuDF toolkit and Dynamo-Triton software on NVIDIA Blackwell Ultra GPUs , Revolut is scaling PRAGMA in production, servicing an initial 80 million customers and engineered to scale 3x. “Revolut processes billions of transactions — the opportunity was always there, but the question was whether foundation models could operate at that scale and across domains. PRAGMA answers that,” said Pavel Nesterov, head of AI at Revolut. “Built in collaboration with NVIDIA, it’s the clearest proof yet that the gap between a research breakthrough and a production banking system can be closed.” Read All NVIDIA in Europe Announcements Training on Accelerated Infrastructure Starling Bank , the first bank to train an agentic model on U.K. sovereign cloud infrastructure, has developed “Starling Intelligence” — the intelligence layer that underpins its AI capability — on Civo’s U.K. cloud using the bank’s own transaction data. Lloyds Banking Group built ATUM , a transaction foundation model that converts billions of banking transactions into reusable intelligence — accelerated with NVIDIA RTX PRO 6000 Blackwell GPUs and improving efficiency across bank-wide use cases. SBS launched a banking sovereign intelligence platform so banks can run generative and agentic models without their data leaving their own walls. Open Models at a Fraction of the Cost Open weight models give European financial institutions proprietary intelligence built on data they own, at costs they control. NVIDIA’s accelerated computing platform is the engine running beneath them. Neobank bunq’s agentic assistant Finn now runs on open weight models. Powered by NVIDIA accelerated computing, deploying on Nebius matches the quality of leading closed models at 60% of the cost per answer. NatWest is building technology designed to recognize the full range of British accents, dialects and financial terminology, developed with NVIDIA’s speech models, including Nemotron 3.5 ASR. Modulr is building specialized models for distinct parts of payment operations, based on the NVIDIA Nemotron-3-Ultra-550B open model. Santander is an early adopter of the new NVIDIA Kumo family of open weight foundation models for structured financial data, using them to build specialized intelligence for core banking workloads. Accelerated Computing Transforms Financial Research LSEG is working with NVIDIA to help financial institutions analyze large volumes of historical tick data up to 10x faster with the Polars GPU engine , accelerated by NVIDIA cuDF on NVIDIA RTX PRO 6000 Blackwell GPUs. RavenPack built a dataset generation pipeline on Bigdata.com that takes a plain-language research hypothesis and returns a labeled, backtest-ready dataset — running on a fine-tuned NVIDIA Nemotron model. Investors can use these datasets to identify, analyze and visualize market trends. Speeding Data Pipelines to Improve Fraud Detection RTX PRO 6000 Blackwell Server Edition GPUs and NVIDIA cuDF help accelerate data processing across multi-terabyte-scale, siloed, regulated sources, compressing the time from data to decision for fraud and other scenarios. HSBC provides near-real-time client insights that support customer processes and on-demand decision-making while minimizing risk and fraud, powered by 5x faster data processing on Starburst Enterprise’s GPU Engine . Stripe , a global provider of programmable financial services, sees 42x faster data pipelines and 92% cost savings for its Stripe Radar fraud-detection platform with NVIDIA GPU-accelerated Ray Data in offline testing. This enables Stripe Radar’s machine learning pipelines to continuously optimize model training and improve fraud detection. Learn more about NVIDIA accelerated computing in financial services . See notice regarding software product information. Read All NVIDIA in Europe Announcements

  21. NVIDIA Blog

    NVIDIA Cosmos Opens New Tools to Train, Evaluate and Deploy World Models

    To help developers rapidly build world models , which help physical AI systems understand and simulate the real world, NVIDIA has expanded the Cosmos platform with new training tools, simulation checkpoints and Cosmos Data Factory blueprints. Ranking top five on six different open models leaderboards, the NVIDIA Cosmos platform gives world model builders the frontier open models — as well as data, training and optimization tools and recipes — to scale training data, adapt models, test behavior and optimize for performance before deployment. Build Better Training Data Building world models begins with multimodal, multisensory data. NVIDIA Cosmos Data Factory brings together the tools and skills for labeling, curating, augmenting and evaluating data at scale for training physical AI. Cosmos Curator, a GPU-accelerated solution for curating video training datasets, has now added support for multimodal multisensory inputs and processing blocks. Read All NVIDIA in Europe Announcements Cosmos Data Factory now offers the following five cloud-native blueprints and is available through GitHub , Microsoft Azure , Nebius and CoreWeave : Autonomous Vehicle Dataset Generation: Curates and labels driving data, then adds synthetic examples of hard-to-capture scenarios to expand training coverage. Image Attribute Augmentation: Varies object appearance to improve image search. Event Video Generation: Turns still images into synthetic videos of rare or unsafe events. Defect Image Generation: Generates printed circuit board, metal and glass defect images for inspection models. Video Data Augmentation: Changes weather and lighting in existing footage while preserving scene structure. https://blogs.nvidia.com/wp-content/uploads/2026/10/CDF-visual-Oct20.mp4 Cosmos Data Factory accelerates data curation, processing and synthetic data generation with agentic skills. Companies are already putting Cosmos Data Factory blueprints to work to speed their physical AI development workflows. Porsche is leading the development of a shared AV data flywheel across Volkswagen Group brands, using Cosmos Data Factory through Voxel51 on Microsoft Azure. This is expected to reduce repeated physical test drives by up to 90% and shorten development cycle times by up to 85%. https://blogs.nvidia.com/wp-content/uploads/2026/10/Porche-Cosmos-Data-Factory.mp4 Porsche searching and augmenting autonomous vehicle data footage in Voxel51, powered by Cosmos Data Factory. ANYbotics generated over 1,500 synthetic video clips with Cosmos Data Factory blueprints, curating real-world data and validating outputs with Encord’s platform, to train its industrial-inspection robots to recognize liquid spills and other changes between facility visits. https://blogs.nvidia.com/wp-content/uploads/2026/10/ANYbotics-Cosmos-Data-Factory.mp4 ANYbotic s generating anomaly clips with Encord powered by NVIDIA Cosmos Data Factory. Ultralytics , a computer vision company, retrained its YOLO model with real images and targeted synthetic defects generated using Cosmos Data Factory, improving defect localization and recall. Train, Adapt and Optimize Models With training data prepared, developers may train their own architecture or start with a pretrained model. At GTC Berlin, NVIDIA extended Cosmos Framework training, optimization and deployment tools and recipes as agent skills to more open models, along with Cosmos 3, giving world model builders a faster path from experimentation to deployment. Developers and agents can now adapt open models like GR00T , Alpamayo , QWEN, Pi0.5 and other compatible Hugging Face models through fine-tuning and reinforcement learning, optimize the model with distillation and quantization, and deploy with high-performing inference. Cosmos Framework is an open, modular framework of model training and optimization tools and skills to accelerate world model development. To develop a world model specialized for egocentric manipulation, robotics research lab MicroAGI post-trained Cosmos 3 on videos of humans manipulating objects and distilled its 32-billion-parameter reasoner down to an 8-billion-parameter Nano model with no loss in accuracy. Developers can label data and post-train Cosmos 3 on Roboflow for their specific environments and tasks, then deploy those models to RoboFlow hosted or on-premises NVIDIA GPUs to bring video reasoning into factories and warehouses. Test Policies and Run Models Faster Alongside these tools, NVIDIA continues advancing foundational technology through its work on Cosmos models. New Cosmos 3 Nano model variants for simulation enable developers to test autonomous policies in repeatable, hard-to-capture, multiview multisensor scenarios. Accompanying code allows developers to adapt the models to train and evaluate their own policies for autonomous vehicles and bimanual robots in interactive simulations before real-world testing. Multiview and multisensory video data generation for evaluating policies in closed loop simulation. Developers can deploy core Cosmos models using updated NVIDIA Cosmos 3 NIM microservices, which deliver up to 2x faster inference with no quality loss in validated configurations, while continuous video analysis enables sub-second alerts. For vision AI developers, the new NVIDIA Metropolis Blueprint for video search and summarization (VSS) 3.3 , built on Cosmos NIM microservices, turns one prompt into an efficient video AI application and helps teams monitor 40% more video streams with the same GPU. ADNOC uses Fogsphere’s MistIQ agentic platform , built with Cosmos and the NVIDIA Metropolis VSS Blueprint , to detect safety hazards in real time across 65 sites. For rare events, Fogsphere also uses the Cosmos Data Factory to generate synthetic training data, improving detection without needing to physically stage dangerous situations. https://blogs.nvidia.com/wp-content/uploads/2026/10/ADNOC-Cosmos.mp4 The Cosmos platform is fully modular and open. Developers can use Cosmos Data Factory to generate training datasets, Cosmos Framework to train and adapt world models, then deploy supported Cosmos models through NIM microservices . Read All NVIDIA in Europe Announcements

  22. NVIDIA Blog

    How Companies Are Using NVIDIA DSX to Break the Power Bottleneck

    Companies around the world are building on the NVIDIA DSX AI Factory Platform to plan, simulate and run the grid and the AI factory as one system, from the substation to the rack. Their adoption of the platform responds to an acute problem. Power is now the biggest bottleneck in infrastructure. More than $300 billion in equipment is backlogged, and over 200 gigawatts of projects are waiting in U.S. interconnection queues. One root cause of the bottleneck is siloed systems. The grid, the facility and the compute inside have long been designed independently from one another, creating the conditions for delays, risk and stranded capacity. Read All NVIDIA in Europe Announcements To help address this, NVIDIA DSX brings these systems together: NVIDIA DSX Sim and SimReady assets from leading energy manufacturers let engineers simulate the grid, substation and AI factory together, validating generators, switchgear, power systems and cooling before breaking ground. NVIDIA DSX Flex syncs AI factory demand with grid conditions and onsite generation, turning AI factories from static loads into flexible grid assets. NVIDIA DSX MaxLPS and DSX Exchange squeeze more compute from every watt and securely share signals across compute, networking, energy, cooling and operations. GE Vernova is deploying its grid-to-facility and rack power expertise to complement NVIDIA DSX systems, enabling customers to transform power constraints into competitive advantages while maximizing compute and token output, sustainability and grid reliability. Hitachi Energy ’s battery energy storage system has qualified as NVIDIA DSX Ready , helping customers identify power infrastructure that meets applicable NVIDIA DSX reference architecture design requirements for resilient, reliable and scalable AI factories. Schneider Electric , together with AVEVA and ETAP, is advancing NVIDIA DSX with validated NVIDIA Vera Rubin power and cooling reference architectures and an NVIDIA Omniverse -powered AI factory digital twin. Combining AVEVA ’s engineering software expertise with NVIDIA SimReady assets, OpenUSD and Omniverse enables connecting simulations, real-time operational data and AI-driven optimization across AVEVA, Schneider Electric and third-party solutions. Together, these capabilities help organizations design, simulate, operate and optimize data centers across their lifecycle, improving energy efficiency and workload performance while accelerating time to token. Siemens Energy is expanding its collaboration with NVIDIA by developing a reference design for resilient, scalable and grid-aware power infrastructure across power generation, transmission and distribution. This design is aligned with the requirements of the NVIDIA DSX Vera Rubin AI factory design. The company is also exploring usingNVIDIA technologies such as RAPIDS, Metropolis and Isaac Sim for its Noedra digital twin platform, evaluating how the technologies could enhance grid infrastructure monitoring, identify emerging risks and support unplanned outage prevention. Phaidra integrated DSX Flex, DSX Max-Q and DSX Exchange into its self-learning agents, optimizing cooling and power allocation to maximize tokens per watt, improve grid flexibility and deliver about 10% more compute within the same power envelope. Emerald AI integrated its Conductor platform into DSX Flex to help utilities tap underused grid capacity, an approach modeled to unlock up to 100 gigawatts on the existing U.S. grid. Explore the NVIDIA Omniverse DSX Blueprint and the Vera Rubin DSX AI Factory Reference Design . Learn more from NVIDIA and energy leaders who are accelerating power-flexible AI factories to strengthen the grid at GTC Berlin. Visit Hitachi Energy (booth 7004, hall 7), Schneider Electric (booth 3041, hall 3) and Siemens Energy (booth 3048, hall 3) and tune into sessions with Schneider Electric on Wednesday, Oct. 21: From Grid to AI: Engineering the Future of AI Factories and Energy Intelligence: Powering the AI Era . Read All NVIDIA in Europe Announcements

  23. NVIDIA Blog

    NVIDIA Transforms European Retail, From Digital Discovery to Physical Operations

    European retailers are using open models and physical intelligence across use cases — from a commerce platform cutting inference costs by 90% to to a fulfillment center where collaboration of different robotic fleets is simulated inside a digital twin. Frontier Results With Open Models Retail leaders and technology companies across Europe are using NVIDIA-accelerated open and frontier models to advance agentic commerce, digital product discovery and retail R&D. Shopify post-trains open models to reach frontier-level accuracy at a fraction of the cost. Its engineers use the NVIDIA Megatron Bridge library for faster fine-tuning and NVIDIA NeMo RL for reinforcement learning. Shopify has post-trained NVIDIA Nemotron 3.5 Lightning for the GraphQL subagent integrated into Sidekick, Shopify’s AI assistant for merchants. The subagent retrieves store data for several Sidekick features, including Sidekick Pulse, which researches a merchant’s store and recommends ways to improve it. Read All NVIDIA in Europe Announcements On Shopify’s internal benchmark, the post-trained model surpasses a leading frontier model in accuracy while cutting serving costs by 90% and delivering responses 7x faster. Beyond Sidekick, NVIDIA CCluster vLLM inference optimization software helped Shopify achieve 4.5x faster performance and up to 12x lower inference cost across five additional use cases: catalog enrichment, merchant assistant, e-commerce shopping simulation, product taxonomy classification, and merchant and buyer recommendations. Zooplus , Europe’s leading online pet retailer, with Infosys and NVIDIA technologies rebuilt its product discovery stack for the agentic era. A production ready prototype deployed inside Zooplus ’ Amazon Web Services (AWS) environment achieved 86% search accuracy using the NVIDIA Nemotron Retriever library for semantic search and the NVIDIA Blueprint for retail catalog enrichment for generated, search-ready product attributes. Photoroom results from its deployment with Vestiaire Collective , where its Image Editing API now standardizes every primary image, about 40,000 a day across 70 countries, at the point of listing. Listings processed by Photoroom sold at a 5% higher rate of liquidity, with 3% to 7% more unique views per product page . The pipeline runs on NVIDIA GPUs with TensorRT and Triton Inference Server, and NVIDIA Dynamo Snapshot can help users quickly respond to demand spikes. Omi launched ProductDrop, a workflow that anchors every generated image to an approved digital twin, letting brands scale creative output across entire catalogs without introducing product errors . L’Oréal is using this approach, combining Omi ’s digital twins with generated brand environments within its CreAItech program. Omi runs TRELLIS 3D asset generation on NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, speeding up inference by 10x and reducing pipeline latency by 4x. In addition, L’Oréal is accelerating the discovery of the best ingredient combinations, moving from months of experimentation to hours of computational screening, with the integration of NVIDIA ALCHEMI into its research workflow. L’Oréal is also testing this approach with CurieOS, Orbital Industries ‘ operating system for science and engineering, using OrbMol-v2, its open source molecular simulation model. Integrated with the NVIDIA ALCHEMI Toolkit and powered by NVIDIA accelerated computing, OrbMol-v2 is five times faster than comparable models and delivers 21% lower error than the next-best model of its kind. This AI-based workflow enables researchers to identify the most promising ingredient combinations for solubilizing UV filters and optimizing performance before committing to laboratory testing. Mirakl , a software company for intelligent commerce, runs open models on NVIDIA GPUs, both on its own infrastructure and through European cloud providers like Scaleway . This keeps its technology sovereign, avoids vendor lock-in and delivers the speed and scale its customers expect . Mirakl ’s Catalog Transformer pairs fine-tuned open models with proprietary services and has onboarded more than 70 million products, turning messy vendor data into accurate, well-organized catalogs. Physical Intelligence Enables Training in Simulation NVIDIA physical AI, digital twins and open models help simulate retail environments, automate warehouse operations and deliver virtual try-on experiences for European retailers and fashion brands. Otto Group advanced its autonomous warehouse under human supervision initiative — built with Reply on NVIDIA Omniverse libraries and running on Google Cloud — to the group’s largest fulfillment center in Haldensleben, Germany. New capabilities let logistics teams design, test and optimize warehouse operations entirely in simulation before any physical implementation . ALLSIDES launched its 3D Digital Twin Factory: an automated pipeline that transforms physical objects into SimReady 3D digital twins in minutes, capturing sub-millimeter geometric accuracy, measured physical properties and fully relightable materials that maintain accuracy under any lighting condition. Those digital twins serve as the foundation for simulating entire store layouts in 3D, as well as for tracking products reliably throughout the shopping journey and at checkout regardless of ambient light. Built on NVIDIA CUDA, OptiX and TensorRT, the platform already serves brands including adidas , Amazon , Nike , Zalando and Zara . CATCHES achieved production results from its RealFit deployment with AMIRI of conversion up 55%, add-to-cart up 80% and gross sales up 56% , all after seasonal adjustment against like-for-like styles. The company launched PredictiveFit, extending physics-based virtual try-on to retailers, as well as CASI, a personal fashion agent that shops, styles and sizes for customers over WhatsApp, powered by open weight models including NVIDIA Nemotron 3, built on NVIDIA Omniverse and the Newton Physics Engine. Explore New NVIDIA Retail Blueprints at the Nebius Booth At GTC Berlin, Nebius is showcasing three NVIDIA Blueprints for retail , all available in early access and running on Nebius infrastructure, accelerated by NVIDIA: Agent Harness for Employee Productivity : An agentic assistant that consolidates scheduling, task management, inventory alerts and operational insights for store managers. Inventory Simulation and Optimization: A blueprint that can help retailers reduce overstock and stockouts by modeling demand scenarios, testing replenishment strategies and optimizing stock levels before users need to commit in the real world. Agentic Finetuning Flywheel: A self-improving loop that uses agentic workflows to continuously generate training data, fine-tune open source models and feed real-world performance back into the next training cycle. Request early access to the blueprints and learn more about NVIDIA technologies for retail and consumer packaged goods . See notice regarding software product information. Read All NVIDIA in Europe Announcements

  24. NVIDIA Blog

    Pioneers Across Europe Deploy NVIDIA Earth-2 Open Models to Power Real-World Weather and Climate Prediction

    Weather forecasts shape more than daily plans; timely weather information is crucial for energy producers, insurers and transport operators to plan around changing conditions. A global forecast can show a storm taking shape, but decisions about solar farms, shipping routes or insured property depend on what happens locally. NVIDIA Earth-2 gives developers and researchers a family of open AI models and tools they can customize with their own data. Running on NVIDIA computing, these technologies support global weather prediction, detailed regional forecasts and environmental research. Read All NVIDIA in Europe Announcements StormScope EU is the latest model to join Earth-2 — a regional version of the StormScope model for short-term forecasting, trained over Europe weather and climate data. It uses EUMETSAT satellite observations to predict evolving cloud conditions across 16 Meteosat channels at 10-minute intervals. At GTC Berlin, NVIDIA is showcasing a live StormScope EU pipeline in an interactive weather demo built in partnership with Earthmover . The demo uses Earth2Studio to run forecasts and Earthmover’s data platform to manage and serve the real-time results, which are updated as soon as new satellite observations come in, every 10 minutes. Earthmover serves these results as map tiles, letting visitors explore predictions across Europe without downloading entire datasets. Developers can combine Earth-2 models in workflows tailored to regional needs, using StormScope for short-term forecasting and CorrDiff to add local detail to coarser weather data. Developers can also pick up these model architectures and train them for new environmental applications such as air-quality modeling. The open source NVIDIA PhysicsNeMo framework provides training and customization tools so teams can build applications that incorporate proprietary data and domain expertise to meet regional needs. Organizations building on Earth-2 capabilities include: AXA is using NVIDIA Earth-2’s FourCastNet 3 to study regional cyclone risk beyond the historical record across more than 1,000 simulated seasons, generating insights that can inform risk assessments and help customers target prevention efforts. Vaisala Xweather is combining Earth-2 with its proprietary observations , lightning data and forecasting expertise to enhance its weather intelligence platforms for government, energy, maritime, transportation and insurance customers. Its work includes adapting StormScope with lightning data to advance short-term forecasting lead time and make these warnings accessible through Vaisala Xweather’s software-as-a-service offerings and AI-assistant integrations. https://blogs.nvidia.com/wp-content/uploads/2026/10/Vaisala-Xweather-Nvidia-StormScope.mp4 Video courtesy of Vaisala Xweather. University of Manchester researchers adapted the Earth-2 CorrDiff model for U.K. air-quality modeling. It trained the model on the NVIDIA-powered Isambard-AI supercomputer and ran it locally on NVIDIA DGX Spark systems, bringing environmental modeling from a national supercomputer to the desktop. TotalEnergies is exploring NVIDIA Earth-2 for solar-energy forecasting, evaluating StormScope-based solar-radiation nowcasts at two photovoltaic sites and developing models to provide weather information at subhourly intervals. The work explores how more detailed forecasts could help anticipate changes in solar-power generation. Space42 developed SARDiff4Height using CorrDiff and NVIDIA PhysicsNeMo. The research estimates surface heights from a single radar image and produces uncertainty maps, helping analysts understand where estimates need closer scrutiny. Explore NVIDIA Earth-2 open models and tools , and explore with Earth2Studio’s forecasting examples . Read AXA’s hurricane-risk case study and Manchester’s air-quality story to learn how teams are adapting weather AI to specialized challenges. Read All NVIDIA in Europe Announcements

  25. arXiv · 场景与世界生成

    Anticipating Social Consequences with Distributional World Models

    Social world models predict how interactions develop after an action, helping agents negotiate, cooperate and mediate. In these situations, one action can lead to several plausible futures, hence predicting only one future can overlook consequential alternatives, while exploring them through repeated simulations increases inference cost. We propose DSWM, a Distributional Social World Model that predicts alternative responses, their probabilities and immediate and final outcome values in one output per candidate action. We train a 4B model on repeated simulations using supervised fine-tuning and GRPO. At inference time, the actor uses these possible consequences to choose an action. Across three seen and two unseen environments, with interaction backbones absent from training collection, DSWM achieves the best average performance on 13 of 15 metrics. Ablations show that retaining multiple futures and their predicted probabilities improves outcomes in all five environments.

  26. arXiv · 可控视频与镜头

    Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing

    Autonomous video editing from open-world resources remains a challenging problem in multimedia content creation. Retrieval-based methods are constrained by fixed source material and lack creative flexibility, while generative models struggle with long-horizon coherence and often diverge from user intent. Both paradigms fail to replicate the structured, multi-stage reasoning that underlies professional post-production. To address these limitations, we present Auto-Director, a tool-augmented multi-agent framework that mirrors professional editorial workflows: a Planning Agent decomposes abstract topics into fact-grounded scripts via retrieval-augmented planning; a Material Agent indexes open-world footage through shot segmentation and visual tagging; an Editor Agent assembles semantically aligned clips with MLLM-driven spatial adaptation; and a Critic Agent governs an iterative Reflect-and-Revise loop that checks scene-level alignment and pacing. To rigorously evaluate this new paradigm, we introduce VEdit-Bench, a benchmark of 100 expert-curated topics with human-annotated Golden Scripts and a five-dimensional metric suite covering Narrative Logic, Factual Accuracy, Semantic Alignment, Compositional Quality, and Stylistic Cohesion. Extensive experiments demonstrate that Auto-Director significantly outperforms baseline methods in both automated metrics and human evaluation, effectively bridging the gap between amateurs and professional-grade video production.

  28. arXiv · 场景与世界生成

    From Single-Actor Environments to Multi-Actor Event Worlds: Can LLMs Be Event World Models?

    World models help agents assess the consequences of their actions before acting. Many real systems, however, involve multiple actors who make their own decisions, can evolve even when the agent takes no action, and are only partially observed through event logs. We formalize these systems as event worlds and ask whether large language models can predict future observations from the log and the agent's proposed actions. We build complementary benchmarks from human Diplomacy games and real Polymarket trades. Our hierarchical framework compares prediction with and without other actors' actions, then tests multi-step rollouts, predictions for actions never taken, and usefulness to agents. Fine-tuning and more data improve prediction when all actors' recorded actions are given. With only the agent's action, models struggle when other actors strongly affect the outcome, and larger models and longer histories help little. This gap also limits multi-step prediction and agent decisions, with models overestimating plan success. Yet imagined experience and initialization from world models can still improve policy learning. Anticipating other actors' actions remains a central challenge for event world modeling.

  29. arXiv · 可控视频与镜头

    AVAtt: A Free-Viewing Eye-Tracking Dataset and Benchmark for Visual Attention on Paintings

    Saliency and scanpath models are developed almost exclusively on natural photographs, yet paintings are composed by artists to guide the viewer's eye and differ from photographs in texture, colour and abstraction. We introduce \avatt, a free-viewing eye-tracking dataset on \nImages{} paintings spanning sixteen art movements and five centuries, with \nScanpaths{} scanpaths and \nFixations{} fixations from \nParticipants{} participants (\spPerImageMin--\spPerImageMax{} observers per painting). For every painting we release individual scanpaths with fixation timing and eye--screen distance, fixation maps, saliency maps smoothed at one degree of visual angle, temporal saliency videos and rich art metadata. We benchmark classic and deep saliency models against human agreement, revealing substantial headroom for art-aware models, and rank 44 scanpath-similarity measures, including a new language-free family of image-encoder metrics built on ten CNN, ViT and self-supervised encoders. With controls that remove viewing biases, we show that faces are fixated \faceEnrich$\times$ more than expected and within half a second, that local structure draws the eye, that composition rules fitted from image content predict where people look, and that art style shapes how much viewers agree. Dataset, code and a Python package are available at: https://huggingface.co/datasets/Amine-CV/AVAtt.

  30. arXiv · 创作 Agent 与装配

    Externalise the Memory: A Pre-Registered Study of Where an Embodied Agent's Experience Should Live

    Should an embodied agent's experience be baked into its policy weights, or kept external and retrieved at inference? A memory-augmented teacher (a VLM that retrieves lessons, with their episode images, from its own past attempts) produces trajectories on a vision-conditioned reach task; from the same trajectories we distil LoRA student policies and evaluate them under matched protocols, including one that re-attaches the teacher's frozen retrieval buffer to a fixed set of student weights. We pre-registered the hypotheses, arms, tests, and analysis rules before training and before any p-value. Distilling the memory-augmented behaviour into weights beat an action-only control by +1 pp (n.s.); the registered test proves underpowered for its own minimum effect, so we withdraw - rather than refute - the memory-in-weights claim. Attaching retrieval to a fixed set of student weights added +34 pp (z=5.15), reaching the pooled memory-augmented teacher reference (49.3%); the pre-registered protocol contrast agrees (+23 pp, z=3.36); under a pre-registered auto-reset sensitivity count the two are +36 and +29 pp. An exploratory behavioural probe with pre-registered per-arm predictions is consistent with behavioural distillation transferring the marginal correction but not the conditional, situation-specific correction that retrieval supplies. The recipe that follows is to externalise the memory: in follow-up controls registered before they ran, the same frozen buffer lifts the base model with no adapter to 48% (sensitivity 48%) and the other three adapters from 19-26% to 53-55% (sensitivity 17-21% to 50-55%), and the adapter adds nothing detectable once the buffer is attached. On a second, harder task, the retrieval advantage points the same direction but falls short of significance (+6 pp, 95% CI [-4.5, +16.4]); exploratory diagnostics tie the attenuation to memory coverage.