### Added - Per-turn file diffs with Undo and Reapply, plus a read-only Git Changes view in Code. ([#246](https://github.com/WhiteTowerAI/ohmygame/pull/246)) - Export Interactive Stories as editable projects. ([#253](https://github.com/WhiteTowerAI/ohmygame/pull/253)) - A background video timeline in the Scene workbench, with playback, scrubbing, and Mark to attach a specific moment to an agent message. ([#264](https://github.com/WhiteTowerAI/ohmygame/pull/264)) - Doubao speech generation with voice selection and preview, plus speech, sound-effect, and music nodes on Asset Canvas. Audio generation requires a compatible connected provider. ([#259](https://github.com/WhiteTowerAI/ohmygame/pull/259)) - Skills in the composer's slash menu. ([#257](https://github.com/WhiteTowerAI/ohmygame/pull/257)) ### Changed - Plugins have Public and Personal collections, separate AI setup sessions, enable/disable controls, Personal plugin editing, and ZIP import/export. ([#250](https://github.com/WhiteTowerAI/ohmygame/pull/250)) - Model selection and reasoning controls share a compact composer control. ([#252](https://github.com/WhiteTowerAI/ohmygame/pull/252)) - The Asset Canvas Library picker starts with the current project selected. ([#261](https://github.com/WhiteTowerAI/ohmygame/pull/261)) - OpenAI and Anthropic browser sign-in options explain subscription limits. ([#267](https://github.com/WhiteTowerAI/ohmygame/pull/267)) ### Fixed - Desktop apps check for updates periodically and on return to the window, with retries after failed checks. ([#260](https://github.com/WhiteTowerAI/ohmygame/pull/260)) - Provider settings open without waiting for cloud verification. ([#262](https://github.com/WhiteTowerAI/ohmygame/pull/262)) - Leaked thinking tags no longer appear in agent conversations or Canvas text. ([#263](https://github.com/WhiteTowerAI/ohmygame/pull/263)) - The mouse wheel pans the canvas over node content. ([#255](https://github.com/WhiteTowerAI/ohmygame/pull/255)) - New project can open an existing workspace folder. ([#251](https://github.com/WhiteTowerAI/ohmygame/pull/251)) - Removed the empty-state prompt overlay from the Story canvas. ([#254](https://github.com/WhiteTowerAI/ohmygame/pull/254)) **Full Changelog**: https://github.com/WhiteTowerAI/ohmygame/compare/v0.0.0-beta.4.8...v0.0.0-beta.4.9
Existing video multimodal large language models (MLLMs) organize observations primarily in time, but embodied spatial reasoning requires organizing them in space. Consequently, their visual representations can remain locally consistent across nearby frames while lacking a persistent spatial context that is stable across viewpoints and time. We exploit depth and camera pose, which are naturally available to embodied agents, to explicitly establish this spatial correspondence. We introduce Cambrian-BEV, which extends Cambrian-S with a parameter-free geometric tokenization scheme that back-projects visual patch features from each frame into a shared bird's-eye-view (BEV) coordinate frame and averages features that fall within the same spatial cell. Thus, observations of the same physical location, regardless of viewpoint or time step, are aggregated into a common spatial token. In addition to providing a persistent scene representation, this formulation acts as an efficient compression mechanism: 128 frames containing approximately 100K DINOv3 patch features are consolidated into a fixed 27$\times$27 grid of only 729 BEV tokens, cutting LLM prefill time by $2.9\times$ and peak memory by $3.6\times$. Cambrian-BEV requires no architectural modification beyond BEV tokenization and uses a frozen DINOv3 vision encoder with a trainable multimodal projector and LLM. On VSI-Bench, our model reaches 63.48, $+8.5$ points over an identically fine-tuned frame-token control with $12.6\times$ fewer visual tokens, with particularly large gains on room size ($+24.5$) and relative direction ($+34.1$). On ReVSI, it reaches 44.68, $+4.5$ over the corresponding control.
Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic manipulation has been extensively studied, personalized robotic manipulation (PeRM) lacks a controlled benchmark for separating task completion from preference satisfaction. We introduce LIBERO-PeRM, a benchmark for preference learning in embodied agents. LIBERO-PeRM studies five questions: 1) how policies follow preferences in zero-shot settings; 2) how they learn single preferences from demonstrations; 3) how efficiently they acquire preferences as demonstrations increase; 4) how learned preferences generalize across layouts and tasks; and 5) how policies compose preferences and behave under conflicts. We propose a controlled formulation and taxonomy of observable preferences in simulation, covering motion, skill, and user-profile choices. This formulation is restricted to language-expressible preferences that can be instantiated in LIBERO without changing the task goal, evaluated automatically, and reused across tasks. Building on LIBERO, our procedural pipeline generates paired demonstrations with and without preferences. We create five task suites (312 tasks) with high-quality demonstrations, rich metadata, and preference-satisfaction metrics. Experiments show that current VLA policies struggle with zero-shot personalization, while SFT makes preferences learnable. Mixed training achieves better no-preference retention than sequential adaptation and improves performance with contextual prompts compared with preference-only training, but can weaken motion-preference learning; preference generalization, decomposition, and conflict handling remain difficult. The code, datasets, and demos can be found on the project website: https://libero-perm.github.io/.
Recent advances in multimodal large language models (MLLMs) and 3D generation have expanded 3D understanding and synthesis. However, beyond visual fidelity, task-specific correctness grounded in 3D evidence remains a major challenge. We argue that building reliable 3D systems requires not only stronger generation and reasoning, but also verification feedback that identifies unmet requirements and explains their violations, allowing systems to reflect on and refine their outputs. Such feedback must integrate multi-view observations and distinguish genuine inconsistencies from projection-dependent appearances. We make three contributions. First, we introduce 3DVerifierBench, a benchmark spanning 10 tasks that evaluates decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study reveals a 21.6 percentage-point gap between humans and the strongest tested model, GPT-6 Astra, while explanation-aware analysis exposes weaknesses in rejection reliability. Second, we analyze verification across fine-grained factual categories, exposing errors in values and target binding. We develop a scalable pipeline combining multi-view fact extraction, controlled 3D editing, and native annotations to produce 14,126 training examples with contradiction explanations and linked counterfactuals. Third, we examine how verification could support 3D generation through candidate selection and recursive refinement. In human-labeled sixteen-candidate pools, oracle selection improves complete factual success from 26.7% to 90.0%. In a separate eight-input study, human-guided recursive refinement satisfies all seven requirements for every input, compared with three under parallel Oracle@8. Together, our benchmark, analysis, and data pipeline establish a foundation for models to reflect on 3D content through reliable verification feedback.
At a glance A new form of computing . Zero-knowledge proofs let you run a workload and receive, alongside the result, a short proof that the program ran correctly. Anyone can check the proof in milliseconds, without rerunning the program, without trusting whoever ran it, and without needing data on which the program was run. Three grand challenges have stood in the way of broad adoption : making proofs fast enough to produce, making them available to normal programs without cryptographic expertise, and resting them on cryptography that will still be trusted many decades from now. Magnetar makes major progress on all three challenges. It proves programs written in Rust, Go, or C, runs 10 to 39 times faster on average than the prior fastest comparable systems, and rests its security only on standard hash functions. It is the latest result of more than a decade of research at Microsoft Research on making zero-knowledge proofs practical. A new form of computing Every result a computer produces comes with an implicit question: did the program run as claimed and produce correct outputs? Today, answering that question often requires a degree of trust alongside verification. We trust the operator, the vendor, the counterparty, or the agent, and we rely on contracts, audits, and logs to establish that trust. Zero-knowledge proofs offer a fundamentally different approach to establishing trust. A program runs, and along with its output it produces a short proof, a cryptographic certificate that this program, on these inputs, produced this output. The proof is small and fast to check. Whoever receives it can confirm the result in milliseconds without rerunning the program, and can pass the proof on to anyone else who needs the same assurance. With the zero-knowledge property, the proof also reveals nothing about inputs that should stay private. This changes what correctness means in the real world. A result no longer has to be trusted because of who produced it. It can be checked because of what accompanies it. An AI agent acting on someone’s behalf could attach a proof that each action complied with the policy it was given. Regulated financial institutions could prove that transactions were processed correctly without making their transactions public. These become possible once producing a proof is as routine as running the program. Three grand challenges Zero-knowledge proofs were invented in the 1980s, and the ideas behind proving program execution have been around nearly as long. What has kept them from becoming a general tool is three challenges that have resisted being solved at the same time. Efficiency . Producing a proof has historically taken millions of times more computation than running the program itself. Years of research have brought that overhead down, but the fastest systems have achieved their speed by making compromises elsewhere. Generality and usability. Many proof systems require developers to manually rewrite their program as a cryptographic circuit, which is a fixed arithmetic description of that exact computation. This takes months. Every change to the application means a new circuit, and the translation is itself a frequent source of security bugs. A general-purpose alternative is to prove the execution of an ordinary processor, so that any compiled program can be proven as written. Until recently, that generality came at a steep cost in speed or security. Security. A proof is only as strong as the cryptography beneath it. Many proof systems rest on elliptic curves, which can be broken by a sufficiently powerful quantum computer. Microsoft is addressing this problem with its Quantum Safe Program, which will move its products and services to post-quantum cryptography, a goal it recently accelerated to 2029 . Post-quantum proof systems already exist, but some rest on lattice assumptions, and most rest on new cryptographic functions invented to reduce the cost of proving. These new cryptographic functions have not faced the decades of scrutiny given to the cryptography protecting today’s internet, and some rest on mathematical conjectures about error-correcting codes that have not been proven. They have bought their speed with strong assumptions. More than a decade of work For more than a decade, Microsoft Research has worked on making zero-knowledge proofs and verifiable computing practical. Early research followed two approaches, both practical only for small programs. The first (opens in new tab) (opens in new tab) lets a verifier check a long computation through a short conversation with the prover using a technique called the sum-check protocol (opens in new tab) , but this applied only to computations with a particular layered structure. The second (opens in new tab) (opens in new tab) used elliptic-curve cryptography to create very small proofs for almost any computation. But this requires a one-time trusted setup involving secret information which, if leaked, could allow anyone to forge proofs. This approach is also vulnerable to quantum computers. In 2019, a new zero-knowledge proof system called Spartan returned to the sum-check approach and made it work for general-purpose programs without a trusted setup. Spartan can also be instantiated with different cryptographic building blocks. That flexibility is what later allowed the proof system to be built on hash functions alone. A sequence of results then filled in the remaining pieces: Phalanx (opens in new tab) , in 2021, showed how to exploit repetition, such as a processor running the same instruction cycle on every step, to prove computations an order of magnitude faster than Spartan. Magnetar’s processor description is built on this idea. Nova (opens in new tab) , HyperNova (opens in new tab) , NeutronNova (opens in new tab) , Neo and SuperNeo (opens in new tab) showed that stopping Spartan’s proving early, and folding the partial results together, makes it practical to prove computations of any size at far lower cost. Brakedown (opens in new tab) , in 2021, introduced a fast hash-based commitment scheme and combined it with Spartan, giving the first Spartan variant whose security rests on standard hash functions. Spice (opens in new tab) , Lasso (opens in new tab) , Nebula (opens in new tab) , and then Twist and Shout (opens in new tab) made the memory operations at the heart of a general-purpose processor inexpensive to prove, and Jolt (opens in new tab) showed that combining them with Spartan can efficiently prove the execution of a complete processor. Earlier this year, Vega (opens in new tab) applied this line of work to digital identity, proving facts from a government credential in under a tenth of a second . More recently, BinarySpartan (opens in new tab) brought Spartan to binary fields, where a processor’s logic and standard hash functions can be efficiently expressed. BinarySpartan also relies on Ligerito (opens in new tab) , a Brakedown-style commitment scheme with much shorter proofs, and on ring switching (opens in new tab) , which makes it efficient to commit to vectors of bits. Each of these solved a piece of the problem. Magnetar is where the pieces come together. Magnetar Magnetar is a zero-knowledge virtual machine, or zkVM, and it makes major progress on all three challenges at once. Figure 1. Magnetar proves a program’s execution and returns a small proof that anyone can check, with security resting only on standard hash functions. It is general-purpose. Magnetar proves the execution of RISC-V, an open processor architecture with mature compilers for every mainstream language. A developer writes a program in Rust, Go, or C, compiles it as usual, and Magnetar proves it as written. Consider what this means in practice. A business policy, a data pipeline, or an interpreter for the Ethereum Virtual Machine (EVM) is just a program, and the engineers who already maintain it can prove their executions exactly as written. As a result, proving stops being a specialist activity. It is faster than prior systems in the same class. Magnetar proves 7.2 to 8.7 million processor instructions per second on a laptop. Across 15 benchmark programs, from cryptographic hashing to token transfers inside a small EVM interpreter, it is 10 to 39 times faster on average than the fastest existing zkVMs on the same machine, and more than 300 times faster than two others on a token-transfer benchmark. An optimized Magnetar proves 9.5–12.8 million cycles per second, which increases each speedup reported here by about 1.3×. Its proofs verify in about 12 milliseconds. Its security rests on standard cryptography. Magnetar depends on a standard hash function such as SHA-256 or SHA-3, the same primitives that already protect every organization’s systems. It requires no elliptic curves, no lattices, no special-purpose hash functions, no trusted setup, and no unproven conjecture about error-correcting codes. Hash functions are the most thoroughly studied primitives in the cryptographic toolkit, and they are the basis of the stateless hash-based signatures that NIST standardized as its own conservative post-quantum option. For a system that must still be trustworthy in many decades to come, relying only on standard hash functions removes a cryptographic migration from the roadmap. The same design decision pays a second dividend. Checking a Magnetar proof consists mostly of evaluating standard hash functions and binary-field arithmetic, which are exactly the operations Magnetar proves efficiently. Verifying a proof is therefore an ordinary workload for it. This is what allows Magnetar to prove computations of any size, by proving them in pieces and then combining the pieces. This also allows Magnetar to prove a long-running system such as a ledger or a blockchain. The technical paper gives the full design, security analysis, and measurements. Looking ahead Magnetar advances zero-knowledge proofs and verifiable computing to a stage where they can be built on the cryptography that organizations already trust, at speeds that make it usable, by engineers who write normal code. Magnetar enables a future where cryptographic proofs are a routine output of computation. We expect that technology leaders will stop asking whether proofs of execution are practical and focus instead on which results would they prefer to verify rather than taking on trust . If you’re interested in exploring how zero-knowledge proofs can support your use case, we’d love to hear from you (opens in new tab) . Opens in a new tab The post Proof-based general-purpose computing with Magnetar appeared first on Microsoft Research .
Unit quaternions are widely used in computer graphics making their efficient compression critical for reducing memory consumption and bandwidth requirements. In this paper, we present a novel method for encoding uniformly distributed unit quaternions. Our thorough analysis of several quaternion parameterizations compares their theoretical error properties, quantization behavior, computational cost, and memory efficiency and provides both an analytical and empirical characterization. We show that an infinity-norm parameterization with our optimized warp function consistently offers the smallest maximum error and the best storage efficiency, at a decompression cost equal to or lower than that of parameterization methods found in practice. We demonstrate the practical benefits of the proposed approach in several graphics applications, including tangent-space normal mapping, skeletal animation with blend weights, and Gaussian splatting.
中文 | English 中文 新增功能 自定义模型 API 支持通过UI中配置各模型的推理等级，无需手动修改配置文件。 by @LegGasai 允许用户从“自定义插件索引源”中发现和安装插件。 by @turtle1999 问题修复 修复命令与参数交界处输入异常的问题。支持整体选中/删除，并在撤销或刷新页面后恢复。 by @LegGasai 改进 Office 文件交付：避免在回复中引用已删除的预览图，减少 PDF 导出后不必要的工具查找和额外检查。 by @yudshj 修复启用开发者工具后，仍需刷新页面才能打开工具面板的问题。 by @imccyu 减少 Linux 下启动命令前的重复检查，降低命令启动开销，并改善较旧 systemd 环境的兼容性。 by @imccyu 修复命令执行失败时，未呈现有效的状态信息。 by @yudshj 其他变更 第三方插件的版本兼容性优先按 engines.dsh 判断；未声明时沿用原有规则。插件作者需检查声明是否覆盖支持的 DSH 版本。 by @turtle1999 English New Features Configure reasoning levels for individual custom API models in the UI without editing configuration files manually. by @LegGasai Discover and install plugins from custom plugin index sources. by @turtle1999 Bug Fixes Fix input glitches between a command and its arguments. Commands can be selected and deleted as a whole, and restored after undo or a page reload. by @LegGasai Improve Office file delivery: avoid referencing deleted preview images in replies and reduce unnecessary tool searches and extra checks after PDF export. by @yudshj Fix Developer Tools requiring a page reload before its panel can open after being enabled. by @imccyu Reduce repeated checks before launching commands on Linux, lowering startup overhead and improving compatibility with older systemd environments. by @imccyu Fix failed commands not displaying useful status information. by @yudshj Other Changes Third-party plugin compatibility prioritizes engines.dsh when declared and otherwise uses the existing rules. Plugin authors should check that the declared range covers their supported DSH versions. by @turtle1999 Full Changelog: dsh-v0.2.1-alpha.3...dsh-v0.2.1-alpha.4 官方 Release：dsh-v0.2.1-alpha.4。请查看 Release notes、依赖和许可证变化。
## [0.2.17] - 2026-10-15 ### Added / 新增 - The built-in Agent can browse and import local media in a browser-hosted editor once `AGENT_IMPORT_ROOTS` lists the directories to expose, reaching the same probe/hash/dedupe chain the desktop app uses; the desktop app keeps its default access (#185). 浏览器端的内置 Agent 在 `AGENT_IMPORT_ROOTS` 列出允许目录后，也可以浏览和导入本地素材，复用桌面端相同的探测、哈希与去重流程；桌面端保持原有的默认访问行为（#185）。 ### Changed / 调整 - Doubao 配音改走火山新版音频生成接口 `POST /api/v3/tts/create`，`X-Api-Key` 单头鉴权，Base URL 填完整接口地址也能正常请求；配置项改为 `DOUBAO_TTS_API_KEY` + `DOUBAO_TTS_MODEL`（默认 `seed-audio-1.0`），旧的 App ID / Access Key / 音色资源 ID 不再使用，需用新版控制台的 API Key 重填。支持 `speedRatio` / `loudnessRatio` / 原生 `pitch`，`emotion` / `emotionScale` / `performancePrompt` / 方言不再支持。 Doubao TTS now uses the new Volcengine audio-generation endpoint `POST /api/v3/tts/create` with single-header `X-Api-Key` auth, and accepts a full endpoint URL as Base URL. Config is now `DOUBAO_TTS_API_KEY` + `DOUBAO_TTS_MODEL` (default `seed-audio-1.0`); the old App ID / Access Key / resource ID are removed, refill with a new-console API key. Supports `speedRatio` / `loudnessRatio` / native `pitch`; `emotion` / `emotionScale` / `performancePrompt` / dialect are no longer supported. ### Fixed / 修复 - Move left/right to boundary (`Ctrl + E` / `Ctrl + R`) now fires on Windows and Linux; every shortcut catalog row is asserted to match its advertised chord on both platforms (#202, #203). 「左贴边 / 右贴边」（`Ctrl + E` / `Ctrl + R`）在 Windows 与 Linux 上恢复可用；新增断言要求快捷键目录里每一行在两种平台都能被自己的写法命中（#202、#203）。 - Desktop whisper.cpp transcripts group BPE tokens into words: multi-token words (Russian, German compounds, …) no longer split with inner spaces, punctuation stays attached to its word, and scripts without spaces (Chinese, Japanese, Thai) still split per chunk (#206, #207). 桌面端 whisper.cpp 转写按 BPE 规则把 token 还原成词：俄语、德语复合词等多 token 词不再被拆开并插入空格，标点跟随其所属单词；中日泰等无空格文字仍按块切分（#206、#207）。 - Windows verification gates pass on a stock checkout, and verification suites no longer read or write the developer's real profile (`USERPROFILE`) (#208, #209, #210, #211). Windows 上验证门禁在全新检出即可通过；验证套件不再读写开发者真实用户目录（`USERPROFILE`）（#208、#209、#210、#211）。 - Preview: the CC button and the rendered picture share one caption-visibility decision, hallucinated ASR words no longer silence an audio clip, and transcript windows starting past the end of the media are rejected (#213). 预览：字幕开关与画面共用同一可见性判定，幻觉出的 ASR 词不再把音频片段静音，转写窗口越过素材末尾时会被拒绝（#213）。 - Internal: `server/key-probes.ts` split back under the 500-line gate (MiniMax/Doubao HTTP-200 business-code checks moved into `server/key-probe-result.ts`), unbreaking `npm test` on `main`. 内部：`server/key-probes.ts` 拆回 500 行门禁以内（MiniMax/Doubao 的「HTTP 200 业务码」后检移入 `server/key-probe-result.ts`），修复 `main` 上 `npm test` 的红灯。 --- ## What's Changed * fix(shortcuts): match an explicit Ctrl chord on Windows and Linux by @JasonBuildAI in https://github.com/0xsline/OpenChatCut/pull/203 * fix(desktop): group whisper.cpp BPE tokens into words and keep punctuation by @JasonBuildAI in https://github.com/0xsline/OpenChatCut/pull/207 * fix(verify): make four more verification gates pass on Windows by @JasonBuildAI in https://github.com/0xsline/OpenChatCut/pull/209 * fix(verify): isolate the real Windows profile from every verification suite by @JasonBuildAI in https://github.com/0xsline/OpenChatCut/pull/211 * fix(preview): three ways a clip sat on the timeline but never reached the preview by @autober in https://github.com/0xsline/OpenChatCut/pull/213 * feat(agent): browse and import local media from a browser-hosted editor by @JasonBuildAI in https://github.com/0xsline/OpenChatCut/pull/200 ## New Contributors * @JasonBuildAI made their first contribution in https://github.com/0xsline/OpenChatCut/pull/203 * @autober made their first contribution in https://github.com/0xsline/OpenChatCut/pull/213 **Full Changelog**: https://github.com/0xsline/OpenChatCut/compare/v0.2.16...v0.2.17
0.11.0 (2026-10-14) Features blender: play saved audio inside Blender with Play audio and Stop audio ( #466 ) ( 28c685a ) blender: re-enable Generate for an unchanged form after a delay ( #477 ) ( 984627e ) Bug Fixes blender: keep chosen-file uploads through scene updates and refuse incomplete MCP calls ( #441 ) ( 0d266fa ) blender: offer a saved image as World only when its header can be 2:1 ( #465 ) ( 0013ebe ) jobs: match a saved WAV original to its MP3 asset record for server stills ( #488 ) ( 3b69052 ) jobs: refuse a duplicate paid request while one is unconfirmed ( #475 ) ( 971eaa7 ) jobs: save the WAV original of audio results, not the MP3 copy ( #479 ) ( e3da3b9 ) mcp: check Host, time out idle sockets and limit agent uploads ( #476 ) ( 297c916 ) mcp: skip the reverse DNS lookup when local MCP starts ( #443 ) ( 56b03ef ), closes #399 tools: ignore documents as drift sources and list removed ones ( a2b5369 ) ui: keep the result pager's arrows one step wide in Studio ( #471 ) ( 73141c8 ) ui: make the picker's OK take one click, reword its experimental line and name LoRAs ( #464 ) ( 19f3ec0 ), closes #440 ui: say where the panorama World goes and that Undo reverts a reuse ( #491 ) ( a8a38a7 ) ui: title the unconfirmed-job dialog and name Mark reviewed as its next step ( #489 ) ( 9f9eab9 ) Documentation agents: add /knowledge-refresh for the per-version documentation refresh ( #482 ) ( a2b5369 ) docs: add a gallery of Scenario explorations made in Blender to the handbook ( #474 ) ( d3e6589 ) docs: keep the limits only the old evidence records stated ( e60a2d1 ) docs: lead installation with the extension repository ( #438 ) ( be8eba0 ) docs: make the user guide shorter and easier to follow ( #468 ) ( a0a0c92 ) docs: name the next release 0.11.0 in the maintenance records ( #480 ) ( 2858d57 ), closes #440 docs: point open work to #440 and record the 2026-10-13 decisions ( #469 ) ( 29a1b0d ) docs: re-pin the Studio source to 0.2.0 and record its additions ( #458 ) ( ca97ec5 ) docs: record the 0.10.0 publication evidence ( #436 ) ( 34ce22e ) docs: record the first-party origin of the bundled icons ( #470 ) ( e02dada ) docs: refresh documentation for 0.11.0 ( #487 ) ( b3c7ac9 ) docs: review the user guide's Studio, workflow, Library, Environment and update evidence ( #473 ) ( 9399a26 ), closes #440 docs: split the UI style guide and runtime map by area ( #484 ) ( 7d3e55c ) docs: sync release records with the 0.10.0 human checks and settings ( #439 ) ( fdcd91b ), closes #36 #97 #189 #19 tests: record the fixture licence and generating-account rights ( #460 ) ( 3c04fc6 ), closes #9 Continuous Integration ci: give the weekly full native suite a 900-second timeout ( #442 ) ( 70db5c3 ), closes #399 ci: lint and format-check the whole Python tree ( #467 ) ( a1efd28 ) deps: wait seven days before proposing action updates ( #463 ) ( 3221af5 ), closes #39 Tests blender: make two macOS-flaky native tests deterministic ( #459 ) ( 5eefa91 ), closes #457 Code Refactoring address the remaining hand-fix Ruff findings ( #461 ) ( fc8db71 ), closes #28 docs: keep one evidence record per document ( #481 ) ( e60a2d1 ) Styles reformat the Python tree with Ruff (mechanical) ( #462 ) ( 998e17d ), closes #28 官方 Release：blender-plugin-v0.11.0。请查看 Release notes、依赖和许可证变化。
中文 | English 新增功能 Web 版支持在本机 HTTP 访问时登录 DeepSeek 账号。 @lsdsjy 支持识别已发送消息中的密钥和令牌，并默认脱敏按需查看。 @ZiyaZhang 极简模式支持读取图片和自动压缩长对话，Web 与桌面端可通过 /compact 指令手动压缩上下文。 @LogosVeritas 问题修复 修复切换会话、收起工作过程或发送后续消息时，对话区域出现大块空白、滚动跳动的问题。 @imccyu 修复重启应用后部分会话无法继续，以及切换到 DeepSeek 后因历史消息兼容问题导致请求失败的情况。 @imccyu , @yudshj 修复插件更新或重新安装后仍运行旧代码的问题。启用热重载时，可替换的插件会直接生效；无法热重载的情况会提示重启。 @turtle1999 修复 macOS 添加工作区时目录选择器无法粘贴目录路径，以及取消选择仍添加工作区的问题。 @turtle2099 修复原预设被删除或调整后，恢复空白会话、切换工作区可能失败的问题。 @LegGasai 修复部分较旧 Git 版本下无法创建工作树的问题。 @turtle2099 修复远程 Web 在网络拥塞时被误判为断线，以及刷新后语言、主题等偏好丢失的问题。 @turtle2099 , @oraluben 应用意外退出后，未完成的任务会明确显示“任务已停止”，避免被误认为正常完成。 @turtle2099 凭据文件中出现 DSH 不使用的额外字段时，不再阻止应用启动。 @turtle2099 修复卸载会话存储插件时，最后的会话记录可能丢失的问题。 @turtle1999 体验优化 改善会话列表加载和 @ 会话搜索体验 @imccyu 子智能体可通过配置选用独立 Agent Preset 预设，统一设置角色和工具； @grllll , @Dudu-0223 优化侧栏展开规则，让运行中和等待回应的会话更容易找到；会话菜单支持复制会话 ID。 @turtle1999 , @imccyu DeepSeek 账号的模型列表随账号可用模型更新，并隐藏当前客户端版本不支持的模型。 @LegGasai 模型设置更清楚地提示凭据来源和检测状态，支持按需显示输入的 API Key。 @Magolor Office 文档任务按需加载编辑、渲染和导出指引，减少简单内容修改中不必要的检查与预览。 @yudshj 改进子智能体的委派与等待指引：达到委派层数上限时提前提示，等待子任务期间不再消耗新的目标执行轮次。 @ZiyaZhang , @turtle1999 Web 终端更可靠地处理连续输入、粘贴和尺寸调整，并保留进程退出前的最后输出。 @LegGasai 桌面端可用 Cmd/Ctrl + 加号或减号统一调整正文、代码和终端字号。 @koalazf99 优化会话和工作区的悬停操作，统一卡片与菜单样式，并改善页头排列及插件页、自动化任务页的窄窗口布局。 @Yifffan , @Kingwl PTC Mode 的 Node 执行环境新增可选的 in-memory 模式，多次运行代码可复用变量、函数和已导入模块。 @CreatixChu 其他变更 API Key 读取优先级变更：Web端/桌面端优先采用已保存的凭据；headless 默认改用环境变量和 .env，不读取已保存的密钥。 @Magolor 更新第三方模型支持；Azure 提供方名称统一为 azure，使用旧名称 azure-openai-responses 的提供方配置需迁移。 @LegGasai Python SDK 的 Office 引擎和完整创作环境改为按需下载。 @yudshj 使用 --from-default-profile 重建已有自定义配置时，会替换该配置及其插件和设置。 @turtle2099 TypeScript 和 Python SDK 支持 resume 功能，恢复已保存的会话，继续使用原有会话 ID 和工作目录。 @koalazf99 New Features The Web app supports DeepSeek account sign-in when accessed over local HTTP. by @lsdsjy Detect keys and tokens in sent messages and mask them by default, with an option to reveal them. by @ZiyaZhang Minimal mode can read images and automatically compact long conversations. Web and Desktop users can run the /compact command to compact context manually. by @LogosVeritas Bug Fixes Fix large blank areas and scroll jumps when switching sessions, collapsing work details, or sending follow-up messages. by @imccyu Fix failures when continuing some sessions after an application restart, and history compatibility errors when switching to DeepSeek. by @imccyu , @yudshj Fix stale plugin code after updates or reinstallation. With hot reload enabled, replaceable plugins take effect immediately; changes that cannot be hot-reloaded request a restart. by @turtle1999 Fix pasting directory paths into the directory picker when adding a workspace on macOS, and prevent cancelled selections from adding a workspace. by @turtle2099 Fix failures when restoring a blank session or switching workspaces after its original preset was removed or changed. by @LegGasai Fix worktree creation failures with some older Git versions. by @turtle2099 Fix false disconnects during network congestion and lost language, theme, and other preferences after refreshing a remote Web client. by @turtle2099 , @oraluben Tasks interrupted by an application exit now show “Task stopped” instead of appearing to have completed normally. by @turtle2099 Extra fields that DSH does not use in the credentials file no longer prevent the application from starting. by @turtle2099 Fix a case where final session records could be lost while unloading the session storage plugin. by @turtle1999 Improvements Improve session list loading and @ session search. by @imccyu Configure spawned subagents with independent Agent Presets to manage their roles and tools together. by @grllll , @Dudu-0223 Improve sidebar expansion so running sessions and sessions awaiting a response are easier to find. Session menus support copying the session ID. by @turtle1999 , @imccyu Keep the DeepSeek account model list up to date with the models available to the account, while hiding models unsupported by the current client version. by @LegGasai Model settings show clearer credential sources and detection status, and let users reveal an entered API key. by @Magolor Load Office editing, rendering, and export guidance as needed, reducing unnecessary checks and previews for simple content edits. by @yudshj Improve subagent delegation and waiting guidance: announce delegation depth limits in advance and wait for running subtasks before starting another goal round. by @ZiyaZhang , @turtle1999 Handle continuous input, pastes, and resizing more reliably in the Web terminal, and preserve the final output when a process exits. by @LegGasai Use Cmd/Ctrl + plus or minus on Desktop to adjust text, code, and terminal font sizes together. by @koalazf99 Refine session and workspace hover interactions, unify card and menu styling, and improve header ordering and narrow-window layouts on plugin and automation pages. by @Yifffan , @Kingwl The Node runtime for PTC Mode adds an optional in-memory mode that reuses variables, functions, and imported modules across code executions. by @CreatixChu Chores Change API key precedence: Web and Desktop prioritize saved credentials. Headless uses environment variables and .env files by default, without reading saved keys. by @Magolor Update third-party model support and rename the Azure provider to azure. Provider configurations using azure-openai-responses must be migrated. by @LegGasai The Python SDK’s Office engine and full authoring environment are now downloaded on demand. by @yudshj Using --from-default-profile to recreate an existing custom profile replaces that profile, including its plugins and settings. by @turtle2099 The TypeScript and Python SDKs support session resume, restoring saved sessions with their original session IDs and working directories. by @koalazf99 Full Changelog: dsh-v0.2.1-alpha.2...dsh-v0.2.1-alpha.3 官方 Release：dsh-v0.2.1-alpha.3。请查看 Release notes、依赖和许可证变化。
### Added - Community search, sorting, and time filters, creator profiles, and owner controls to edit or unpublish a game. ([#242](https://github.com/WhiteTowerAI/ohmygame/pull/242)) - MiniMax H3 and H3-Max video generation with your own MiniMax China or Global API key. ([#240](https://github.com/WhiteTowerAI/ohmygame/pull/240)) - Release notes in Settings → About, available offline and shown with available updates. ([#239](https://github.com/WhiteTowerAI/ohmygame/pull/239)) - Option-drag (Alt-drag on Windows and Linux) duplicates canvas nodes. ([#230](https://github.com/WhiteTowerAI/ohmygame/pull/230)) - Image generation accepts a resolution and aspect ratio. ([#244](https://github.com/WhiteTowerAI/ohmygame/pull/244)) ### Changed - New installations connect directly by default, and network settings are simpler, with clear mode hints and unsaved-change feedback. ([#238](https://github.com/WhiteTowerAI/ohmygame/pull/238)) - Provider detail pages are reorganized, with a breadcrumb, a single connection row, a compact model list, and an Advanced section. ([#231](https://github.com/WhiteTowerAI/ohmygame/pull/231)) - First-run setup selects the connected provider's recommended model, and Settings and setup list providers in the same order. ([#228](https://github.com/WhiteTowerAI/ohmygame/pull/228), [#229](https://github.com/WhiteTowerAI/ohmygame/pull/229)) - Back from a project returns to the page it was opened from. ([#241](https://github.com/WhiteTowerAI/ohmygame/pull/241)) - The agent runtime is updated to Pi 1.2.0, and Azure provider settings are migrated automatically. ([#237](https://github.com/WhiteTowerAI/ohmygame/pull/237)) ### Fixed - New projects show the empty preview instead of an error. ([#232](https://github.com/WhiteTowerAI/ohmygame/pull/232)) - A preview stopped while restarting no longer stays marked as ready. ([#243](https://github.com/WhiteTowerAI/ohmygame/pull/243)) - The light canvas background outside onboarding is restored. ([#245](https://github.com/WhiteTowerAI/ohmygame/pull/245)) **Full Changelog**: https://github.com/WhiteTowerAI/ohmygame/compare/v0.0.0-beta.4.7...v0.0.0-beta.4.8
Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation agents with different adaptation mechanisms, we find that policies exhibit distinct weakness profiles and benefit from different training environments. Weakness-conditioned adaptation consistently outperforms matched random and hard-example alternatives on ProcTHOR, while the resulting gains for Uni-NaVid and Gemini also extend to HM3D after ProcTHOR adaptation. Iterative re-diagnosis further shows that dominant weaknesses can persist or shift as the policy improves, providing updated guidance for subsequent environment construction. Together, these results show that policy failures are not only diagnostic signals, but actionable guidance for constructing training environments that target a policy's current weaknesses.
Learn how Google DeepMind and GovTech Singapore uses lightweight on-device Gemma models to intercept conversational scams in private chats with zero cloud data egress.
Multimodal Large Language Models (MLLMs) have been widely applied in embodied agents and robotics for vision tasks. However, MLLMs suffer from limitations in spatial reasoning and perception. Post-training techniques have shown promise for improving these capabilities; yet, there are limited data-driven surveys of such approaches. In this survey, we consider approaches to fine-tuning pretrained MLLMs for spatial reasoning tasks, ranging from supervised to reinforcement learning, with an emphasis on the datasets used. We investigate themes of dataset construction, data modality and annotation, and evaluation of the resulting models. Based on these themes, we categorize research into three groups: dataset design, dataset construction and evaluation. We find four main methods for constructing fixed spatial annotations: rules and templates, LLM prompting, procedural generation, and human-in-the-loop. We also observe substantial variation in evaluation approaches, including different scoring methods, tolerance metrics, and benchmark types. Finally, we identify opportunities for future research in unified dynamic annotation, efficient spatial reasoning, standardization of evaluation, expanded spatial range, and human-centered spatial interactions.
OUR PIPELINE VS. THE PREVIOUS LLM PIPELINE Smarter secret scanning, at a fraction of the cost 11.5% higher acceptance Owner-weighted alert precision 86% lower median scan execution time Post-push scanning 98% lower serving cost Compact model vs. remote LLM calls Secret scanning works best when developers are confident in the findings. Catch too little, and real credentials can remain in a repository. Alert too often, and developers may spend time investigating strings that aren’t secrets. Generic secrets make that balance especially difficult. The same string could be a real credential or a documentation placeholder, so detecting it requires understanding the surrounding code and text. To address this challenge, we trained a secret classifier specifically for GitHub’s product needs: distinguishing real credentials from lookalikes while meeting the speed and efficiency requirements of scanning at scale. In our companion post, Secret protection must scale with software , we explained why secret protection must keep pace with AI-accelerated development. Here, we go behind the scenes of the context-aware classifier powering that effort: how we designed, trained, and evaluated it to find more real secrets with less noise. What we enabled GitHub secret scanning operates at two key moments: Push protection helps prevent secrets from entering a repository by scanning changes before a push is accepted, giving developers an opportunity to address the finding. Post-push scanning works after code has been committed, continuously scanning repository content and creating alerts when potential secrets are detected so developers and security teams can investigate and remediate them. We developed a fast classifier that reduced median post-push scan execution time by 86% compared with the previous LLM-based pipeline. That efficiency made it practical to extend generic secret detection to push protection, returning results before developers push changes, not only after code is committed. Secret scanning blocks a commit with push protection after the model detects a secret on line 5 of the file. Why secrets are hard to detect Structured credentials often include a distinctive prefix, length, alphabet, or checksum. Generic secrets may not. Their syntax overlaps with build identifiers, checksums, asset hashes, sample values, and other non-sensitive strings. Context is the deciding signal. Likely real credential Likely harmless lookalike A password assigned beside a production database URL The same password-shaped value in a test fixture A connection string in deployment configuration A localhost connection string in a setup guide A high-entropy value bound to an API credential A high-entropy checksum, build ID, or asset hash These cases are hard negatives: strings that match a credential pattern but are not usable credentials. Adding more rules can suppress some of them, but each exception risks hiding a real secret. A learned classifier can instead use signals such as file path, secret type, variable names, and nearby code or text. That is why we shipped a new scanner using a two-stage pipeline: Candidate generation : Regex filters scan content for potential secrets with high recall. This stage is optimized to catch broad families of credential-like patterns quickly and deterministically. Context-aware classification : A fine-tuned ModernBERT classifier evaluates each candidate and its surrounding context. It is trained to recognize the signals that separate real secrets from false positives, including placeholder values, examples, documentation snippets, generated content, and non-sensitive identifiers. Because the model is small and efficient to run on our inference infrastructure, it costs much less to serve and is simpler to scale than an LLM-based pipeline. Impact: 11.5% user acceptance improvement, reducing false positives. 86% faster post-push secret detection execution time. More than 98% cost savings compared with the LLM-based pipeline. How our pipeline works As content is scanned, regex patterns cast a wide net and flag anything that looks like a credential, so real secrets aren’t missed at the start. Each candidate then goes to a fine-tuned ModernBERT classifier, which reads the value together with its file path, variable names, and surrounding code. The classifier estimates how likely it is to be a real secret. High-confidence findings become alerts, and lookalikes such as placeholders, test fixtures, and checksums are filtered out before they reach developers. HOW OUR PIPELINE WORKS Broad first, careful second STEP 1 Scan Repository content, commits, pull requests, and other scanned surfaces. STEP 2 Generate candidates Regex patterns identify possible tokens, keys, and credential-like strings. STEP 3 Classify Fine-tuned ModernBERT evaluates candidate + context and assigns likelihood of a true secret. STEP 4 Alert or suppress High-confidence findings move forward; likely false positives are filtered. The important design choice is that the model does not replace deterministic scanning. Instead, it complements it. Regexes remain the right tool for finding candidate strings quickly. The classifier is used where machine learning provides the most leverage: distinguishing “looks like a secret” from “is likely a real secret.” The classifier model makes a best effort to distinguish “looks like a secret” from “is likely a real secret.” A hard-negative problem: secrets that almost look real Most false positives are not random. They are hard negatives: strings that match a credential pattern but are not usable credentials. Examples include documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs, and generated fixtures. These cases are easy for a regex to match and hard for a purely rule-based system to suppress without accidentally losing real secrets. ModernBERT helps because it can read the candidate together with nearby code and text providing contextual understanding comparable to an LLM without the latency and computational overhead of generating natural-language output. For example, a candidate surrounded by words like “example,” “dummy,” “fake,” or “replace-me” may be lower risk than a candidate assigned to an environment variable in a deployment file. The classifier learns these contextual differences from labeled examples rather than requiring engineers to encode every exception as another rule. We chose ModernBERT because it provides a strong fit for this workload: it is compact, efficient, and well suited for classification over code-adjacent text. For secret scanning, the model needs to make a calibrated decision over short candidate windows with high throughput and low latency. That makes a fine-tuned encoder classifier a better serving shape than a general-purpose LLM for this stage of the pipeline. The model can be trained directly on our decision boundary, and evaluated using the same recall and precision metrics that matter to secret scanning quality. How we trained The challenge was finding representative data for cases in which syntax alone is misleading. The training and evaluation strategy combines three sources: Product-derived examples from GitHub repositories, using feedback from the secret detection experience and additional human validation. High-recall regex candidates from GitHub repository data, including human-annotated examples and weak labels used to expand coverage. Synthetic examples targeted at sparse and difficult cases, such as documentation secrets, test fixtures, public keys, checksums, structured lookalikes, and near-miss pairs. Synthetic rows pass deterministic validation and model-based quality review before inclusion. Training emphasizes real secrets, realistic false positives, and hard negatives that frequently confuse regex-only approaches. Each model input includes the candidate value, file path, secret type, and a bounded amount of surrounding context. We select thresholds through offline evaluation and then validate behavior on production-like traffic before broader rollout. Before splitting the data, we grouped rows that share a repository, candidate value, or example so near-duplicates could not appear in both training and evaluation. Training focused on the cases that matter most for alert quality: real secrets, realistic false positives, and hard negatives that regexes frequently match. The input representation includes the candidate secret string, file path, type and a bounded amount of surrounding context. The output is a classification score used by the scanner to decide whether the candidate should proceed. We optimized for the product tradeoff: preserve or improve recall while reducing false positives and serving cost. In practice, that means the classifier threshold is selected using offline evaluation, then validated against production-like traffic before rollout. The most important training principle was to make the model see the confusing examples. We included candidates that are syntactically similar to real secrets but semantically different, so the model learns not only what a secret looks like, but also what a false positive looks like in context. The trained model is exported to ONNX format with 8-bit quantization, enabling fast inference. Evaluation results Online result Change vs. previous LLM pipeline What it measures Repository owner-weighted acceptance 11.5% higher Customer acceptance of detected findings Median post-push scan execution time 86% lower Production scan execution time Serving cost More than 98% lower Cost of the compact classifier compared with remote LLM calls Customer A/B evaluation results for our fine-tuned model compared to the previous LLM pipeline. We evaluated the new pipeline’s impact in production in two stages: first against the production baseline on the same public traffic and then through a customer A/B experiment. Offline evaluation: On post-push scanning, the new pipeline achieved 13% higher precision and 19% higher recall than the previous LLM pipeline when evaluated on the same public traffic. Online evaluation: We then validated whether these quality improvements translated to customer outcomes through an A/B experiment against the previous LLM pipeline. Repository owner-weighted acceptance increased by 11.5%. We weight acceptance by repository owner to reduce the influence of customers who bulk-resolve large volumes of alerts at once, which can otherwise distort a simple average. Take this with you The best security tool is the one developers trust enough to use. Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught. Lower cost means we can keep improving coverage without making scanning prohibitively expensive. For developers and security teams, this should translate into fewer false positives, and faster feedback in the places where secrets are most likely to be introduced: commits, pull requests, and everyday development workflows. Hybrid systems are powerful. Regex gives us speed, transparency, and broad candidate coverage. ModernBERT gives us learned context understanding and false-positive reduction. The combination outperforms either approach used alone for this product constraint. We also learned that model size is not the same as model usefulness. A smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical. This is one step toward making secret scanning more accurate, scalable, and easy to use. Looking ahead, we are focused on two areas: Building a continuous improvement flywheel. We will use production feedback loops to identify new false negatives and false positives, turning real-world signals into higher-quality training and evaluation data. Exploring new model architectures. We are investigating approaches that can perform secret detection and extraction in a single step, reducing pipeline complexity while
Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state changes. Existing evaluations mainly rely on real-world data or procedurally generated tasks. Real-world data preserve natural complexity, while procedural games provide controllable generation and verifiable solutions. However, it remains unclear whether abilities measured in toy games correspond to real-world visual reasoning and whether game examples can support transfer. We introduce Game2Real-Bench, a benchmark that studies these connections across four capability groups, including fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. We construct 1,000 procedurally generated game samples with executable verification and curate 400 real-world samples aligned by shared reasoning requirements rather than visual similarity. We evaluate 25 representative VLMs and observe strong game-to-real consistency, with Pearson and Spearman correlations of 0.935. We further study one-shot game-to-real transfer on four models. Constraint reasoning improves for several models, while fine-grained perception declines with game demonstrations. We also construct Game2Real-Dataset with nearly 24K verified game samples across 18 tasks and 65 QA families. Our benchmark, dataset, and code are available at https://github.com/QingAnLiu/Game2Real-Bench.
## What's Changed * fix(preview): redirect static directories to trailing-slash URLs by @jsdhwfmax in https://github.com/WhiteTowerAI/ohmygame/pull/210 * fix(canvas): remove embeds of deleted images with parentheses in their name by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/167 * fix(daemon): keep truncated UTF-8 files as text in the Code view by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/162 * fix(files): allow renames that only change letter case by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/166 * fix(editor): preserve quoted attributes when changing Scene backgrounds by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/160 * fix(story): place Scenes without a position clear of other cards by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/169 * docs(playable): match Playable Nodes docs to the implementation by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/165 * fix(agent): recover interrupted Responses WebSocket streams by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/212 * feat(publish): add automatic cover switch by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/214 * docs(readme): add Simplified Chinese README by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/215 * feat(i18n): add Simplified Chinese UI by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/223 * fix(desktop): keep playtest session callbacks safe after the window closes by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/218 * fix(desktop): ignore broken stdout and stderr pipes on Windows by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/219 * fix(desktop): stop dropped update downloads from crashing the app by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/220 * test(desktop): cover closing destroyed playtest windows by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/224 * feat(plugins): add preset and custom plugin icons by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/216 * fix(playtest): move controls outside the game surface by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/222 * feat(composer): separate model and reasoning controls by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/225 * feat(providers): support independent OpenAI image connections by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/226 * feat(onboarding): add first-run welcome and provider setup by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/227 * fix(desktop): use bundled Windows title bar icon by @82228 in https://github.com/WhiteTowerAI/ohmygame/pull/217 * chore(release): prepare v0.0.0-beta.4.7 by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/233 * chore(release): prepare v0.0.0-beta.4.7 by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/234 ## New Contributors * @jsdhwfmax made their first contribution in https://github.com/WhiteTowerAI/ohmygame/pull/210 **Full Changelog**: https://github.com/WhiteTowerAI/ohmygame/compare/v0.0.0-beta.4.6...v0.0.0-beta.4.7
An embodied agent must prioritise among competing bodily needs. In active inference, it can respond to an urgent need by increasing the precision of information about that need or by increasing its preference for outcomes that relieve it. We compare these two interventions, separately and together, in a foraging gridworld with three lethal needs, and ask how each changes survival, perception and the body model the agent learns. Precision sharpens interoceptive posteriors and improves survival. Preference reweighting keeps bodily states further from their lethal boundaries and changes the states from which the agent learns, so its learned model covers fewer need levels, with lower quality across all levels but slightly higher quality on the levels it visits. An experiment on 72 further layouts does not establish the predicted preference benefit on difficult layouts, but both interventions improve average survival there and their interaction is positive. Survival alone therefore gives an incomplete account of how a model prioritises needs and learns about its body.
Spatial understanding is essential for embodied agents navigating and interacting with the physical world. As observations arrive incrementally, agents must reason in a streaming manner. Yet streaming understanding must also be closed-loop: task-relevant evidence may require both recalling past observations and actively acquiring new views. Existing streaming approaches largely emphasize processing incoming observations, leaving task-driven active exploration underexplored. We introduce MemAct, a framework for streaming spatial understanding with memory and action. A shared vision--language model incrementally writes structured historical memory, retrieves records and past frames on demand, and acquires additional evidence through movement and turning. Retrieved evidence and action observations inform subsequent decisions, enabling the agent to continue exploring or answer within a unified reasoning loop. We develop the MemAct suite, a data generation pipeline and collection across real and simulated environments, with approximately 1M training examples for spatial reasoning, memory writing, retrieval, and exploration. Its MemAct-bench comprises 20,000 main QA pairs evaluating current-observation understanding, historical evidence use, and active exploration, plus 2,000 separate Timing questions. MemAct combines supervised fine-tuning for memory and interaction with counterfactual action optimization using cumulative evidence gains, action costs, and terminal answer quality to improve exploration and stopping decisions. Experiments show that MemAct outperforms existing spatial understanding and streaming baselines and approaches frontier multimodal models on MemAct-bench while retaining streaming inference efficiency.
Context matters when you’re working with AI. Every codebase has its own house rules: how components are structured, which scripts to run, and which conventions apply to different frameworks and file types. Copilot can help you make changes across a project, but it works best when it knows those rules, too. Much of that knowledge can live inside the repository, so Copilot can pick it up automatically instead of requiring you to explain the same standards, scripts, and workflows every time you start a new task. In this installment of our GitHub Copilot app for Beginners series, we’ll look at four ways to tailor how Copilot works with your project: Custom instructions communicate the conventions Copilot should follow. Agent skills package repeatable procedures. Custom agents bring specialized guidance to larger tasks. Model Context Protocol (MCP) servers connect Copilot to external tools and data sources, expanding what it can do. Let’s look at how each one works and how they can work together. Start with custom instructions Suppose you ask Copilot to add a custom page size to an application. Making the change is only part of the job. The new code should also follow the patterns your team already uses. Repository custom instructions give Copilot that baseline context. Add a .github/copilot-instructions.md file to define guidance that applies across the repository, such as your preferred architecture, naming conventions, or expectations for testing and validation. You can also create path-specific instruction files inside .github/instructions . These files use an applyTo field to tell Copilot when the guidance is relevant. When Copilot works on a matching file, it can use both the repository-wide instructions and the more specific guidance. That lets you keep the broad rules in one place without forcing every file type into the same mold. Package repeatable procedures as agent skills Custom instructions are useful for standards that should shape many tasks. But some guidance only matters when Copilot is performing a particular procedure, such as running quality checks, preparing a release, or debugging a failed workflow. That’s where agent skills come in. A skill is a folder containing instructions and, optionally, scripts and other resources that Copilot can load when they’re relevant. Project-level skills can live in .github/skills , with each skill in its own subfolder. For example, a .github/skills/quality-checks folder could contain a SKILL.md file that explains: When Copilot should use the skill. Which scripts run the project’s tests and linting. What each script checks. How to troubleshoot common failures. The description in SKILL.md helps Copilot determine when the skill applies. After updating the interface, Copilot can load the quality-checks skill and follow the project’s established process for testing and linting instead of guessing which commands to run. Use a custom agent for broader, specialized work Some tasks are larger than a single procedure. An accessibility review, for example, may require Copilot to inspect components, templates, styles, and interactions across multiple files. Custom agents let you create specialized versions of Copilot for work like this. A repository-level agent profile lives in .github/agents and can define the agent’s purpose, instructions, tools, and relevant project context. Use a custom agent when a recurring type of work benefits from a dedicated role or perspective, especially when the task may span multiple files and draw on several tools or sources of context. Connect Copilot to external tools with MCP servers Instructions, skills, and agents help Copilot understand how to work within your project. MCP servers expand what it can access and do beyond the repository. MCP servers connect Copilot to external tools, data sources, and services. A documentation server can give it access to framework guidance, while a server such as Playwright can let it open and interact with your application in a browser. You can browse available MCP servers in the Copilot app or add a custom one. Once Playwright is configured, you can ask: Use the Playwright MCP server to test the functionality manually. Copilot can then use Playwright to open the application in a browser and interact with it much as you would. That gives it a way to verify not only that the code passes automated checks, but also that the updated interface behaves as expected. Choose the right type of customizations These customizations solve different problems, but they become more useful when you combine them: Custom instructions define the conventions Copilot should follow across the repository or for particular files. Agent skills package repeatable procedures that Copilot can load when they’re relevant. Custom agents provide focused expertise and guidance for broader tasks. MCP servers connect Copilot to external tools, services, and data sources, expanding what Copilot can do. Take this with you Giving Copilot useful project context starts with four questions: What project convention do I keep explaining? What multi-step procedure do I repeat? What complex task could benefit from a dedicated custom agent? What outside tool or source does Copilot need? Not sure where to begin? The community has shared ready-made instructions, skills, agents, and more through Awesome Copilot . Find one that addresses a workflow you already repeat, review what it contains, and adapt it to your repository’s tools and standards. Start small: choose one convention you keep explaining, add it to .github/copilot-instructions.md , and see how Copilot applies it to your next task. Try using the GitHub Copilot app > The post GitHub Copilot app for Beginners: How to teach Copilot about your codebase appeared first on The GitHub Blog .
At a glance StudentSim is a research system for evaluating and improving AI tutors before they are tested with real learners; it is not a product, is not deployed in classrooms, and is not intended to replace teachers AI tutoring tools need feedback from students to improve, but collecting that feedback through human studies is slow, costly, and difficult to scale StudentSim trains personalized AI student simulators from publicly available, de-identified learner records, with one simulator adapted to each student The work evaluates simulators along two education-grounded dimensions: whether they respond like a specific student, and whether they update after tutor guidance Across chess, second-language English writing, and mathematics, StudentSim outperforms leading baselines and provides a useful feedback signal for training an AI tutor In research settings, teams are exploring how AI-powered tutoring systems can help explain a math problem, suggest an edit to a writing assignment, or guide a learner through a chess position. Yet the quality of teaching cannot be measured only from the tutor’s answer. A tutor needs to know what happens after the guidance reaches a student. Did the student correct the original mistake? Did the hint help them reason toward the answer, or did it give away too much? Would a different student need a different kind of support? In today’s AI tutor research, answering these questions often requires real learner studies. Those studies are valuable, but they are slow and costly to run, which makes it hard to use student feedback during fast model iteration. StudentSim is an AI system that creates personalized simulators of individual students, modeling how they answer questions and how their answers change after receiving a tutor’s guidance. The goal is to make student feedback more available during AI tutor development. This research also introduces StudentSimEval, a benchmark and evaluation protocol spanning chess, second-language English writing, and mathematics. Importantly, the goal is not to replace teachers or classroom learning. Instead, StudentSim helps researchers evaluate and improve AI tutoring systems during the research and development process before testing them with real learners. The research uses only publicly available learner datasets released for research use with de-identified records; no personally identifying information is used at any stage. Figure 1: AI tutors need feedback from students to improve. Student simulators can approximate part of that feedback during research-stage training and evaluation. They provide an early-stage research signal, not a replacement for testing with real learners. Measuring what a student simulator needs to do A simple way to build a student simulator is to ask a large language model to role-play a learner. For example, a prompt might say, “You are an elementary-school student who struggles with math.” The model can change its tone and express uncertainty. The harder part is preserving the learner’s actual ability boundary. A model may speak in a young student’s voice while still relying on knowledge and reasoning far beyond that student’s level. For training an AI tutor, this can produce misleading feedback: the tutor may appear effective because the simulated student is too capable. Traditional student modeling approaches face a different limitation. Knowledge tracing and behavior prediction models are grounded in real learning records and can estimate a student’s current knowledge, skills, and learning habits. Many of these models, however, were not designed to take natural-language tutor guidance as input. They can help describe what a student has done, but they do not always tell us how a student would respond to a particular explanation, hint, or Socratic question. StudentSim defines the target through two dimensions inspired by education research. Educators often distinguish what a learner can do independently from what they can do with support. Vygotsky’s zone of proximal development captures this gap between current independent performance and performance with guidance. StudentSim turns this idea into two evaluation goals. Behavioral fidelity measures whether a simulator matches a specific student’s recorded responses on held-out problems. Guidance responsiveness measures whether the simulator updates its response after reading tutor guidance. Figure 2: StudentSimEval evaluates whether a simulator matches a student’s independent behavior and whether it responds to guidance. These two goals are complementary. A simulator that only matches past student behavior may be useful for prediction, but it does not show whether a teaching intervention helped. A simulator that always follows the tutor may respond fluently, but it may fail to represent a particular student’s strengths, weaknesses, and likely mistakes. AI tutor training needs both signals: the learner’s starting point and how guidance affects the learner’s performance. Training personalized simulators from sparse student records StudentSimEval is built entirely from publicly available learner datasets that are released for research use with de-identified records. The records carry no personally identifying information, and each personalized simulator is trained only on the de-identified records of the student it represents. Real student data is often sparse at the individual level. A learner may contribute only a small number of essays, answers, or interactions. Training a separate simulator from scratch for each student would risk overfitting and would require each model to relearn common patterns in the domain. StudentSim uses a two-stage training pipeline. In the first stage, the model trains on records pooled across many students in the same domain. This stage learns common response formats, common errors, and common ways students revise after guidance. In the second stage, the model is specialized on one student’s own records, producing a personalized simulator for that student. Figure 3: StudentSim first learns shared student behavior from pooled records, then adapts to each student’s own data. The evaluation covers three domains with different response types. In chess, the simulator predicts a player’s next move and how that player may revise after coaching. In second-language English writing, it generates essays with error patterns matching a learner and predicts corrections after teacher feedback. In mathematics, it predicts a student’s multiple-choice answer and whether the student updates after an explanation. Across these domains, StudentSimEval uses the same principle: compare all methods on the same per-student training records and the same held-out evaluation records. StudentSim outperforms leading baselines across the three domains. In chess, for example, StudentSim reaches behavioral fidelity of 0.5150 and guidance responsiveness of 0.9067. GPT-5.4 reaches 0.2316 and 0.7186, respectively. Maia2, a chess-specific human move prediction model, reaches 0.4535 and 0.2721. The comparison illustrates the difference between the baseline families: GPT-5.4 can respond to guidance but does not reliably match a specific student’s behavior, while Maia2 predicts human chess moves more closely but lacks a natural-language guidance pathway. Figure 4: In chess, StudentSim improves both behavioral fidelity and guidance responsiveness compared with leading baselines. Using simulated feedback to improve AI tutors The research also tests whether a trained student simulator can provide feedback for tutor optimization. In a chess tutor reinforcement learning study, the tutor receives a position and a real student’s previous wrong move, then generates guidance. The student simulator’s own parameters are held fixed during this process, and it reads the guidance and produces a revised move. The reward is based on the improvement in move quality from the student’s previous move to the simulator’s revised move, using Stockfish as an independent chess engine signal. Figure 5: StudentSim can provide feedback during AI tutor reinforcement learning by estimating how a student response changes after guidance. The study compares three conditions: a supervised fine-tuned tutor with no reinforcement learning, a tutor trained with GPT-5.4 as the simulated student reward source, and a tutor trained with StudentSim as the reward source. All conditions use the same tutor policy, supervised starting point, and reinforcement learning setup. Human chess experts then evaluate the tutor responses blind to condition. The StudentSim reward produces the highest-rated tutor on all reported dimensions. The tutor trained with StudentSim reaches 90.5% accuracy, compared with 75.7% for the no-RL baseline and 71.6% for the GPT-5.4 reward condition. It also receives higher ratings for guidance quality and personalization. The result is a proof of concept that student simulators trained on real learner data can provide useful feedback for improving AI tutors. Looking ahead StudentSim does not remove the need for real learners in education research. Human studies remain essential for measuring learning outcomes, safety, usability, and classroom fit. The role of student simulators is earlier in the development process: they can help researchers test tutor strategies, compare models, and refine training signals before moving to more expensive studies with real students. The broader direction is to build AI tutor systems that learn from student reactions, not only from expert-written explanations or static correctness labels. StudentSim shows one path: use real learning records to train personalized simulators, evaluate those simulators on both independent behavior and response to guidance, and use them as scalable feedback sources for tutor training. Future work can extend this approach to longer learning trajectories, additional subjects, richer student profiles, and studies that connect simulated feedback with real learning outcomes. The paper and the StudentSim training and evaluation code, including the StudentSimEval protocol, are publicly available. Sources Paper Code Opens in a new tab The post Simulating student learning for better AI tutoring appeared first on Microsoft Research .
## What's Changed * fix(canvas): only let resizable nodes store a fixed size by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/197 * feat(canvas): size media nodes from recorded dimensions by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/198 * fix(updater): prevent Windows installer conflicts during updates by @82228 in https://github.com/WhiteTowerAI/ohmygame/pull/200 * fix(media): name generated images by their actual format by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/201 * fix(agent): bound image history and optimize OpenAI requests by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/203 * fix(models): use model-specific image reference defaults by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/204 * feat(plugins): add persistent AI plugin builder workspace by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/205 * fix(canvas): center reference removal icons by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/206 * fix(projects): show activity on recent project cards by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/207 * fix(composer): number repeated attachment names instead of refusing them by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/199 **Full Changelog**: https://github.com/WhiteTowerAI/ohmygame/compare/v0.0.0-beta.4.5...v0.0.0-beta.4.6
LLMs are used as high-level controllers for embodied agents, but it remains unclear whether they support collaborative reasoning, such as anticipating a collaborator's actions in real-time teamwork without dialogue. We ask if observing the partner's state, exchanging object knowledge, and predicting the partner's next action, shape task outcomes and coordination when agents can share percepts but never intentions, plans, or task-level messages, and whether the two agents occupy symmetric roles. Using the Habitat-based PARTNR benchmark, we introduce a Decoupled Partial Observability regime that enforces strictly individual perception, and we augment decentralized zero-shot ReAct planners with Other Agent's State (OAS) observation, object Knowledge Exchange (KE) between co-located agents, and Other Agent's Next Action (OANA) prediction, evaluated on the full 2 x 2 x 2 grid with four planners on 1,000 validation tasks. OAS observation improves task quality, reducing extraneous effort for every planner, whereas KE primarily improves execution efficiency, cutting simulation steps by up to 25% with little change in success. The two are largely redundant for strong planners but complementary for the weakest, for which KE without the OAS is the worst configuration. OANA prediction lowers the success rate in 15 of 16 model x context pairs while raising explicit partner mentions to 55-100%, and its accuracy gains lie mainly in the coarse action type, partly because agents recognize the action the partner is already performing. The robot retrieves objects sooner and completes its tasks earlier than the simulated human. Shared context reduces total task time by shortening the remaining duration that the human works alone. Also, coordination benefits arise from responding to shared context, especially the OAS, rather than explicit anticipation, yielding shallow, planner-dependent partner modeling.
Added Compositions (nested sequences). project.comps holds timelines of their own (size, length, tracks, clips); a comp clip places one, drawn like a picture of the comp's size so transform, masks, mattes, blend and grade apply to it as a whole, while the video and audio inside play and export as if on the timeline. Comps nest 4 deep. In the editor: comps in the bin's Project tab (drag to place, double-click to open), an open comp takes over the timeline with a breadcrumb and Esc to go back, Precompose in the clip menu, New composition in the Project tab menu, and a Composition section in the Inspector. Agents: addComp , updateComp , removeComp , precompose and inComp patch ops; comps are validated and listed in the compact view. Animation presets. 33 named animations (In, Out, At playhead, Loop) that key a clip relative to where it sits and how big it is: scale and opacity multiply, the rest add. Apply them from the Inspector's Motion section, and save a clip's own keyframes as a preset in library/presets/ . Templates. Lower thirds (Bar, Underline, Boxes), logo animations (Ring reveal, Slam, Lines) and social layouts (9:16 story, 1:1 quote, 16:9 end card) in the bin's new Templates tab. Each placement is its own comp whose text, colour and logo fields are edited in the Inspector; changing its length keeps the exit at the end. Right-click a comp → Save as template keeps your own in library/templates/ . A placeholder logo ships in library/elements/ . Agents: applyPreset , addTemplate , setTemplate , savePreset and saveTemplate patch ops; GET / POST /api/presets and /api/templates ; the compact view shows a placed template's fields. Hovering a comp clip on the timeline shows "Double-click to open" (and a tooltip naming the comp or template). Motion paths. Position keys (x and y keyed together) take bezier handles ( pin / pout on the keys), so a clip travels along a curve, paced along the curve by its eases. The path shows on the monitor with a dot per frame; drag keys and handles to shape it. Smooth / Straight in the Inspector's Motion section, and Auto-orient turns the clip along the path. Motion blur. motionBlur (shutter angle) and motionBlurSamples average sub-frame samples of a moving clip, in preview and export alike; playback uses as many samples as the frame time allows. Text on a path. A title's textPath (a line, arc, circle, wave, any shape outline or a pen path) carries the text, each letter turned with it; further lines run on parallel curves below. pathOffset (keyframable) slides it along, and every text animation plays along the path. Agents: setMotionPath and setTextPath patch ops; the new props are validated and shown in the compact view. Shape layers. A new clip kind, shape , draws vector items in props.shapes : rectangles (with rounded corners), ellipses, polygons, stars, lines and bezier paths (open or closed), nested in groups. Each has a fill and a stroke (width, caps, joins, dashes) and trim paths ( trimStart , trimEnd , trimOffset ) for draw-ons; the layer's own trim props trim every path at once. Every item value keys on its own, colours and path outlines included. In the editor: a Shapes tab in the bin (or the Pen) and a Shape layer section, with items moved, turned, resized and point-edited on the monitor. Null objects and parenting. kind: "null" is an invisible clip that carries a transform. Any visual clip can take a parent and then moves, turns and scales with it, through chains of parents. Choosing a parent keeps the clip where it is; right-click → Parent to a new null groups clips. Anchor point. anchorX / anchorY on every visual clip set the point position places and rotation and scale turn around. ✥ Move drags it on the monitor while the picture stays put. Agents: setShape , removeShape and setParent patch ops; props.shapes and parent are validated on addClip / updateClip ; the compact view lists shapes and parents. A motion-graphics skill covers the recipes. Changed The Keyboard shortcuts dialog is grouped into sections (Playback, Editing, IN / OUT and markers, Bin and compositions, Masks, …) and has a filter box. Fixed Exports of clips with slow-motion speed ramps (or very busy frames: grades, masks, blur) no longer fail with "presented frame timeout". A seek that lands inside the source frame already on screen is accepted, and later tries wait longer before giving up. 官方 Release：v1.14.0。请查看 Release notes、依赖和许可证变化。
## What's Changed * fix(canvas): save pending edits and viewport before closing by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/186 * fix(playtest): send game screenshots at CSS pixel size as JPEG by @dihuangdh in https://github.com/WhiteTowerAI/ohmygame/pull/188 * fix(desktop): preserve game state between Preview and Play by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/189 * fix(daemon): isolate project loading failures by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/195 * fix(editor): simplify preview controls and studio navigation by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/196 **Full Changelog**: https://github.com/WhiteTowerAI/ohmygame/compare/v0.0.0-beta.4.4...v0.0.0-beta.4.5
Vision-language models (VLMs) can answer many geometric and physical reasoning questions correctly, yet still behave inconsistently in interaction, for example, abandoning a correct answer when confronted with a confident but incorrect user assertion. Standard single-turn accuracy cannot distinguish such capitulations from beneficial corrections because it scores each response in isolation. We introduce GRIP, a controlled benchmark with 34 procedurally generated reasoning domains, 100,000 images, and 600,000 questions spanning geometric, spatial, visual, and physical reasoning. Each scene is generated by deterministic code, and every answer is derived directly from the underlying scene state, providing exact ground truth without human annotation. On GRIP, we propose Dual-Loop Evaluation. A closed loop probes each image with a five-step compositional question ladder to localize where reasoning fails. An open loop then re-elicits the model's answer under controlled challenges, including peer-model responses and unsupported user assertions. Because ground truth is known exactly, answer changes can be signed as corrections (wrong-to-right) or capitulations (right-to-wrong), separating two failure modes: rule absence and rule fragility. The dataset is available at https://huggingface.co/datasets/bilaljawaid980/GRIP-Benchmark, with further details at https://bilaljawaid980.github.io/grip-benchmark/.
Unravel 是一个无需训练的框架，能把单张图像拆解并重建为每个物体都可独立移动的交互式 3D 场景。它用视觉语言模型规划可见物体、由图像编辑器移除并修复遮挡区域，再按移除的逆序重建场景，并以点云作为空间关系恢复的主要模态。在多种室内场景实验中，Unravel 恢复的物体更多、摆放更忠实，物理合理性也优于现有方法。
Dataset distillation (DD) aims to retain the training utility of a large dataset in a much smaller set of examples. For skeleton-based action recognition, this requires capturing discriminative motion variation without disrupting the geometry and temporal continuity of articulated bodies. Directly optimizing joint coordinates can violate these constraints, while plausible motion generation alone does not ensure useful training examples. We propose Prototype-Conditioned Motion Dataset Distillation (ProMoDD), which conditions a motion diffusion model on class-wise prototypes of temporally resolved recognition features. We further tailor diffusion training to motion data, enabling the generator to model both skeletal configurations and their temporal evolution. We then construct each compact training set through budget-aware hybrid selection: representative real sequences cover densely populated regions of the pooled recognition space, while selected generated candidates complement them in less-populated regions. This couples prototype-guided generation with feature-space selection, balancing the reliability of observed motions with the variation of synthesis under a fixed per-class sequence budget. Across five difficulty-stratified subsets of the single-person classes in NTU RGB+D 60, ProMoDD achieves its largest gains under tight distillation budgets, reaching 40.32% top-1 accuracy on Full49 with three sequences per class, 4.57 percentage points above class-wise k-means. These gains reflect the complete framework. The same distilled sets train ST-GCN, MS-G3D, and CTR-GCN without regeneration.
Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision. Hand reconstruction and motion retargeting provide a direct route to such supervision, yet recovering hand motion alone does not ensure physically consistent hand-object interactions. To address this issue, we introduce Hand2Robot, an object-centric 4D hand reconstruction framework for robot policy post-training. Given videos of humans performing the target task, Hand2Robot extracts interaction clips, reconstructs object geometry, and generates robot action supervision through hand-object interaction optimization and motion retargeting. For object tracking, we combine observation confidence with temporal motion constraints to select a globally consistent object pose trajectory from per-frame pose candidates. For hand refinement, we use object geometry and motion as a reference and combine staged 2D alignment with object-local contact constraints to refine hand trajectories and finger joint configurations. Experiments on DexYCB and HO3D-v3 show improved object pose accuracy and hand-object interaction quality. Experiments on three real-robot dexterous manipulation tasks further show that reconstructed and retargeted human demonstrations can effectively supplement limited robot data for target-task policy post-training.
## What's Changed * fix(canvas): stabilize marquee selection and panel activation by @LZWcc in https://github.com/WhiteTowerAI/ohmygame/pull/177 * fix(daemon): keep the access token out of the environment by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/180 * fix(daemon): fail preview and publish on error events instead of exiting by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/170 * fix(canvas): bind copied generation history to duplicated projects by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/171 * fix(publish): store plugin archive entries uncompressed by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/172 * fix(story): keep the saved game across navigation Exits and reset on Start by @xuhongbo in https://github.com/WhiteTowerAI/ohmygame/pull/173 * fix(chat): compact errors and restore reconnect feedback by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/181 * feat(3d): unify provider generation and processing workflows by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/183 * fix(canvas): save before desktop close and restore error recovery by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/184 * fix(preview): isolate failures and manage preview lifecycles by @dogxii in https://github.com/WhiteTowerAI/ohmygame/pull/185 * fix(app): stabilize playable migration, thumbnails, and updates by @82228 in https://github.com/WhiteTowerAI/ohmygame/pull/182 **Full Changelog**: https://github.com/WhiteTowerAI/ohmygame/compare/v0.0.0-beta.4.3...v0.0.0-beta.4.4
Added Masks. Any video, image, SVG, text or adjustment clip can carry a stack of masks ( props.masks , up to 8): rectangle, ellipse, bezier (pen tool with curve handles) and free-hand (a drawn outline fitted into an editable bezier). Each has a mode (add, subtract, intersect, difference), invert, opacity, feather and expansion, and every shape parameter can be keyframed, bezier outlines included. Masks follow the clip's transform and render the same in preview and export. The Inspector's Masks section draws and edits them on the monitor and has a matte view. Agents: patch ops setMask , removeMask and setMaskKeys ; props.masks is validated on addClip / updateClip ; fablecut_scopes {mask:{clip}} reports how much of the clip its masks keep; the compact project view lists each clip's masks. Color layers' masks can be bezier or free-hand too ( Draw in the Color tab, or shape:"bezier" / "freehand" in setGradeLayer ). Track mattes. A clip can be cut by the picture on another track (the video track above by default): props.matte = alpha , alpha-inverted , luma or luma-inverted , props.matteTrack to pick the track. The matte track stops drawing while it is in use; the timeline marks both sides. In the Inspector under Blend ; agents use the setMatte patch op, and fablecut_scopes lists each clip's matte and blend. Motion tracking. Follow a point or a region of a video clip frame by frame (template matching, sub-pixel, optionally with scale and rotation); the track is stored on the clip ( props.tracks ). A mask's ◀ Back / On ▶ buttons track what it covers and key its position (and scale / rotation); the Inspector's Tracking section tracks points and attaches any visual clip to a track as editable keyframes. Esc stops a running track. Agents: the fablecut_track tool (track a point, a region or a mask, then apply it to a mask or a clip) and the removeTrack patch op; tracks are validated on addClip / updateClip and summarized in the compact view. Blend modes: color-burn , exclusion , hue , saturation , color and luminosity join the list (17 in all). blend , matte and matteTrack are validated on addClip / updateClip . 官方 Release：v1.13.0。请查看 Release notes、依赖和许可证变化。