生成媒体的下一阶段：Google DeepMind 谈多模态世界模型、评估与真实工作流

简要描述Brief Description

本场讨论汇聚了 Google DeepMind 中从事 Gemini Omni、Veo、Nano Banana、强化学习和多模态系统研究的研究人员与产品负责人。他们探讨了新近推出的图像和视频 API、生成式媒体的实际用途、语言与视听世界模型之间的关系、音视频联合生成、评估与审美默认设置、改进这些系统所需的数据，以及为什么真实的客户工作流程和专业工艺对于将前沿模型转化为实用产品至关重要。This panel brings together Google DeepMind researchers and product leaders working on Gemini Omni, Veo, Nano Banana, reinforcement learning, and multimodal systems. They discuss newly launched image and video APIs, practical uses for generative media, the relationship between language and audiovisual world models, joint audio-video generation, evaluation and aesthetic defaults, the data needed to improve these systems, and why real customer workflows and expert craft are essential to turning frontier models into useful products.

目录Table of Contents

• 新模型和 API
  New Models and APIs
• 生成式媒体的实际用途
  Practical Uses for Generative Media
• 从单一模型到视频智能体
  From Single Models to Video Agents
• 一切都会收敛为一个模型吗？
  Will Everything Converge Into One Model?
• 语言、代码与中间表示
  Language, Code, and Intermediate Representations
• 世界模型：从理解迈向生成之路
  World Models and the Path From Understanding to Generation
• 为什么音频和视频应该结合在一起
  Why Audio and Video Belong Together
• 感官细节、真实感与人类偏好
  Sensory Detail, Realism, and Human Preference
• 审美默认设置、提示词与专家品味
  Aesthetic Defaults, Prompting, and Expert Taste
• 评估生成式媒体
  Evaluating Generative Media
• 数据和工作流仍然需要的模型
  The Data and Workflows Models Still Need
• 从客户反馈到更好的模型
  From Customer Feedback to Better Models

新模型和 APINew Models and APIs

主持人：欢迎回来，无论是正在观看直播的朋友，还是来到现场的朋友。我们会利用主舞台主题演讲之间较长的时段，回顾那些不一定会集中在某个时刻发布的重要进展。今天，我们很荣幸邀请到正在参与 Gemini Omni、Veo、Nano Banana，以及一些全球领先生成式模型研发的各位。Moderator: Welcome back, both to those watching the stream and those here in person. We use these longer sessions between the main-stage keynotes to reflect on important developments that do not necessarily have a single launch moment. Today we are fortunate to have people working on Gemini Omni, Veo, Nano Banana, and some of the world's leading generative models.

我们请到了负责视频模型的 Dumitru Erhan、负责 Gemini 强化学习和 Omni 的 Shane Gu，以及参与多种生成式媒体模型研发的 Nicole Brichtova，包括 Nano Banana 和本周发布的产品。Nicole 最了解这些新品的情况：本周推出了什么？大家可以试试哪些产品？We have Dumitru Erhan, who works on video models; Shane Gu, who works on Gemini reinforcement learning and Omni; and Nicole Brichtova, who works across the generative-media models, including Nano Banana and the products launched this week. Since you are closest to those launches, what came out this week, and what should people try?

Nicole Brichtova：昨天我们有两次发布。首先，我们推出了 Nano Banana 2 Lite，这是 Nano Banana 系列中速度最快、价格最低的图像模型。它比最初的 Nano Banana 更好，因此对大多数人来说，它可以取代他们过去用于生成和编辑图像的初代模型。它的质量已经非常接近更大型模型的前沿水平。Nicole Brichtova: We had two launch moments yesterday. First, we launched Nano Banana 2 Lite, our fastest and cheapest image model in the Nano Banana family. It is better than the original Nano Banana, so for most people it replaces the original model they used for generation and editing. It gets very close to the frontier quality of the larger models.

大约三秒的延迟为创意构思和反复迭代带来了许多可能。它用起来很有趣，而且质量已经达到这样的水平：不仅适合迭代，也能产出可直接投入生产的成果。The roughly three-second latency unlocks many things in ideation and iteration. It is fun to use, and the quality has reached the point where it is useful not only for iteration but also for production-ready outputs.

其次，我们推出了在 I/O 上预先公布的 Gemini Omni Flash API。这是我们首次向开发者开放这些 API。它们支持高质量的视频生成与编辑，定价与 Veo 3.1 Fast 相同。我们的目标是以实惠的价格提供非常出色的品质。Second, we launched the Gemini Omni Flash APIs that we had pre-announced at I/O. This is the first time we are making those APIs available to developers. They support high-quality video generation and editing, and we are pricing them the same as Veo 3.1 Fast. The goal is to provide very strong quality at an accessible price.

主持人：你们刚推出 Omni 时，和 Logan 做了一期播客，并在视频素材中加入了树懒、拉面等元素。我也想给我们的视频加上这些效果，但需要一个 API 来自动化这个流程。感谢你们终于让这一切成为可能。Moderator: When you first launched Omni, you did a podcast with Logan and added a sloth, ramen, and other elements to the footage. I wanted to do that to our videos, but I needed an API so I could automate the process. Thank you for finally making that possible.

妮科尔·布里赫托娃：这是我最喜欢的应用场景。大家都应该试试。我生成的版本是一只猫，它可能是最不稀奇的动物了。如果你还没看过那些示例，可以搜来看看。它们非常有趣。我们的一位同事做了这项工作，而且特别擅长用富有创意的方式展示模型的能力。Nicole Brichtova: That is my favorite use case. Everyone should try it. My version got a cat, which may be the least exotic animal. If you have not seen the examples, look them up. They are very funny. One of our colleagues did that work and is exceptionally good at finding creative ways to show what the model can do.

生成式媒体的实际用途Practical Uses for Generative Media

主持人：猫和树懒是很有趣的演示，但除了演示之外，真正实用的应用场景有哪些？Moderator: Cats and sloths are fun demonstrations, but what are the workhorse use cases that go beyond a demo?

Nicole Brichtova：我们有两项核心能力。第一项是几乎可以接受任何内容作为输入，并生成视频。未来我们也希望支持其他输出模态；但目前，你可以提供一组分镜图像、一段作为语音参考的音轨或其他素材，并获得一段视频。这为短片制作和创作者工作流程打开了新的可能。我们还在 YouTube Shorts 中推出了工具，帮助创作者更轻松地制作内容。Nicole Brichtova: There are two hero capabilities. The first is the ability to take almost anything as input and produce video. In the future we also want to expose other output modalities, but today you might provide a set of storyboard images, an audio track as a voice reference, or other materials and receive a video in return. That unlocks short-film production and creator workflows. We have also launched tools in YouTube Shorts to help creators make content more easily.

第二项主要能力是视频编辑。可以使用自然语言在视频中添加或移除元素。添加一只树懒是个有趣的例子，但还有许多实际应用。消费者无需知道该使用哪些传统编辑工具，就能清除海滩度假视频中的杂音。企业已经在使用这些模型来制作营销和广告宣传活动。教育也是一个令人振奋的领域：媒体内容可以根据学习者的知识水平和偏好风格进行定制。与我们在自有产品中所能观察到的相比，这些 API 应该能展现出更多的工作流程。The second major capability is video editing. Natural language can be used to add or remove an element from a video. Adding a sloth is a playful example, but there are many practical applications. A consumer might clean up the noise in a beach-vacation video without knowing which traditional editing tools to use. Businesses are already using the models to create marketing and advertising campaigns. Education is another exciting area: media can be customized to a learner's knowledge level and preferred style. The APIs should reveal many more workflows than we can observe in first-party products.

小组成员：我最近看到一个很小但确实很实用的图像处理案例。我父母来访时，我给他们买的一台设备附带的说明书只有英文版，还配有几张示意图。我把那一页拍下来，请 Gemini 翻译成罗马尼亚语，同时保持其他内容不变。结果看起来几乎一模一样，示意图和版式都保留了下来，文字翻译得也相当不错。Panelist: I recently saw a small but genuinely useful image example. My parents were visiting, and a device I bought for them had instructions only in English, with several diagrams. I photographed the page and asked Gemini to translate it into Romanian while keeping everything else unchanged. The result looked almost identical, with the diagrams and layout preserved and the text translated reasonably well.

同样的理念也适用于视频。Omni 的文本渲染能力格外出色，为翻译、国际化、重新配音和扩大受众范围创造了机会。这些并非只是有趣的效果，还能让原本无法获取信息的人也能从中受益。The same idea extends to video. Omni's text rendering is unusually strong, which creates opportunities for translation, internationalization, redubbing, and broader access. These are not merely entertaining effects; they can make information useful to people who otherwise could not access it.

从单一模型到视频智能体From Single Models to Video Agents

主持人：我与处于研究前沿的研究人员交流时，总会听到这样的说法：下一步并不是打造一个更好的单体模型，而是开发视频智能体。这个想法是，不再指望一次处理就能完成所有事情，而是让模型通过智能体式流程协同工作。这个说法符合你的理解吗？还是说，这一方向仍然是一个开放的研究问题？Moderator: In conversations with researchers at the frontier, I keep hearing that the next step is not simply a better single model but a video agent. The idea is to stop expecting one pass to do everything and instead let models work together through an agentic process. Does that terminology resonate, or is the direction still an open research question?

Shane Gu：最让我兴奋的是符号基础模型与视频基础模型协同工作。生成图像和视频领域的早期进展，很大程度上依赖于语言模型能力足够强，能够提供详尽的描述。语言是一种强大的表征，因为它具有普适性，但它也可能有助于解决机器学习中的一个根本问题：虚假相关性。Shane Gu: What excites me most is symbolic foundation models and video foundation models working together. The early progress in generative images and video depended heavily on language models becoming good enough to provide detailed captions. Language is a powerful representation because it is universal, but it may also help with a fundamental machine-learning problem: spurious correlation.

当某个特征具有预测性时，模型并不会自动知道它是否具有因果性。解决这一问题的一种方法，是使用包含因果图中大量干预的极其多样化的数据进行训练。另一种方法是以因果信息为条件。语言可以发挥这种因果条件的作用。对视频或图像如何生成的描述，往往与对其背后因果结构的描述相似。这为更丰富的泛化创造了可能。When a feature is predictive, the model does not automatically know whether it is causal. One way to address that is to train on extremely diverse data containing many interventions in the causal graph. Another is to condition on causal information. Language can act as that causal conditioning. A description of how a video or image came to be often resembles a description of the causal structure behind it. That creates the possibility of richer generalization.

我们发表了一篇评测论文，题为《视频模型是零样本学习者和推理者》。之后，Nano Banana 团队也开展了相关研究，使用的是图像模型。核心观点是，视频模型可以成为处理空间与时间信息的强大基础模型。它能够以零样本方式完成许多传统计算机视觉任务、尝试解答视觉谜题，并展现出与机器人技术相关的物理直觉。虽然仍有很大的改进空间，但视觉推理与文本推理相结合，威力不容小觑。We published an evaluation paper titled "Video Models Are Zero-Shot Learners and Reasoners." The Nano Banana team later followed with related work using an image model. The central idea is that a video model can be a strong foundation model for spatial and temporal information. It can zero-shot many traditional computer-vision tasks, attempt visual puzzles, and display physical intuition relevant to robotics. There is still much to improve, but the combination of visual reasoning and text reasoning is powerful.

这最终会在统一模型内部实现，还是通过智能体协同来实现，很可能会逐步演进。我预计，最终会有越来越多的功能转移到单个模型中。不过，现阶段，通过在智能体系统中将强大的视频和图像理解能力与 Gemini 结合，并将 Omni 用作生成组件，已经可以实现很多事情。我们的团队正在积极探索这一领域。Whether this ultimately happens inside a unified model or through agentic coordination is likely to evolve incrementally. My expectation is that more of it will eventually move into one model. Today, however, there is already a great deal you can do by combining strong video and image understanding with Gemini in an agentic system and using Omni as a generative component. Our teams are actively exploring that space.

一切都会融合为一个模型吗？Will Everything Converge Into One Model?

主持人：这一切最终都会趋向于一个产品吗？你们现在有好几种模型，但“Omni”这个名字似乎暗示着，最终其他一切都可能整合进它之中。Moderator: Does this all trend toward one product? You have several models today, but the name "Omni" suggests that eventually everything else could disappear into it.

Dumitru Erhan：也许最终会是这样，但其中存在工程、研究和产品方面的权衡。Nano Banana 2 Lite 面向一个特定的细分需求。它不一定需要立即与能够生成高分辨率长视频的系统放在同一个字面意义上的检查点中。这些能力未必能通过完全相同的方式得到最有效的训练。Dumitru Erhan: Perhaps eventually, but there are engineering, research, and product trade-offs. Nano Banana 2 Lite serves a specific niche. It does not necessarily belong immediately in the same literal checkpoint as a system that can generate a high-resolution, long video. Those capabilities may not be trained most effectively in exactly the same way.

也许五年后，它们就会全都变成同一种模型。六个月后，我们可能仍会针对不同任务使用多种模型，因为实际应用中的权衡更有利于专业化。Five years from now, perhaps they will all be the same model. Six months from now, we will probably still have multiple models for different tasks because the practical trade-offs favor specialization.

Nicole Brichtova：我们称之为 Gemini Omni，是想暗示这样一种未来：Gemini 将在输入和输出两个方向上都实现完全的多模态能力。我们可能会逐步迈向 Omni 生成和编辑图像与视频的阶段。不过，Dumitru 说得没错，在此过程中，专用模型有许多实用场景。它们能满足当下的具体需求，即便这些需求或界限在一年后有所变化。Nicole Brichtova: We called it Gemini Omni because we wanted to hint at a future in which Gemini is fully multimodal in both directions. We will probably move toward Omni generating and editing images as well as video. But Dumitru is right that specialized models have many useful applications along the way. They meet concrete needs today, even if those needs or boundaries change a year from now.

关于模态之间的迁移，我们还有一个研究问题。图像和视频显然共享某些结构。联合生成音频和视频也合乎情理，因为现实世界就是以这种方式呈现它们的。但编程与视频生成之间的迁移就没那么显而易见了。将它们结合起来，可能会带来有益的共享学习，也可能会浪费模型容量。三维表示与代码等交叉领域也存在同样的不确定性。我们应该探索这些交叉领域，同时专注于人们实际希望用这些模型完成的事情。There is also a research question about transfer between modalities. Image and video clearly share some structure. Generating audio and video jointly also makes sense because that is how the world presents them. But transfer between coding and video generation is less obvious. Combining them might produce useful shared learning, or it might waste capacity. The same uncertainty applies to intersections such as three-dimensional representations and code. We should explore those corners while staying focused on what people actually want to do with the models.

语言、代码与中间表示形式Language, Code, and Intermediate Representations

主持人：有一个问题似乎还没有得到充分解答，那就是正确的中间表示形式是什么。图像描述很适合图像，随着时间推移也可以扩展到视频，但可能效率不高。也许代码更好。视频已经可以通过 FFmpeg 和 Manim 之类的工具生成和处理。我们只需要英语吗？还是说另一种表示形式更有可能胜出？Moderator: One question that feels insufficiently answered is the correct intermediate representation. Captioning works naturally for images and can be extended over time for video, but it can feel inefficient. Perhaps code is better. Video is already generated and manipulated through tools such as FFmpeg and Manim. Is English all we need, or is a different representation likely to win?

Shane Gu：我们在 Gemini 内部大量开展强化学习、智能体和编程方面的工作，因此当然也在探索将代码作为一种表征形式。但关于语言本身，人们也一直有类似的问题：为什么思维链推理必须用自然语言表达？它能否改用任意的连续 token，只需分配更多计算量？Shane Gu: We work heavily on reinforcement learning, agents, and coding inside Gemini, so we are certainly exploring code as a representation. But a similar question has existed around language itself: why must chain-of-thought reasoning be expressed in natural language? Could it instead use arbitrary continuous tokens and simply allocate additional computation?

自适应的测试时计算可以提升效果。然而，自然语言推理之所以如此有效，一个原因在于当前的训练方案。预训练通过扩大规模，吸收了海量知识和能力。强化学习可以将其提取出来并加以完善，但这一过程仍然需要大量计算资源。当推理以自然语言为依托时，就能直接复用预训练过程中获得的智能。去掉这一限制，系统就无法再如此直接地依赖同一种表征。近期文本和多模态系统的大部分进展，都得益于文本发挥了强大基础支架的作用。Adaptive test-time computation can improve results. Yet one reason natural-language reasoning works so well is the current training recipe. Pre-training scales and absorbs enormous amounts of knowledge and capability. Reinforcement learning can extract and refine it, but that process remains computationally intensive. When reasoning is tied to natural language, it can directly reuse the intelligence acquired during pre-training. Remove that constraint, and the system can no longer rely on the same representation as directly. Much of the recent progress in text and multimodal systems has been driven by text functioning as a strong backbone.

Nicole Brichtova：还有一个更简单的产品层面的原因：文字是人类交流的方式。如果你在构建供人们使用的产品，文字自然会出现在界面的某个地方，即使它并不适合表示每一种内部操作。Nicole Brichtova: There is also a simpler product reason: text is how humans communicate. If you are building products that people will use, text will naturally appear somewhere in the interface, even if it is not the right representation for every internal operation.

Shane Gu：一些强化学习至上主义者认为，思维链和语言并非必需，重要的只有额外的计算。我认为这个问题尚无定论。但事实证明，语言很有价值，因为它将计算与预训练期间习得的智能，以及人们能够交流的概念联系起来。Shane Gu: Some reinforcement-learning maximalists argue that chain of thought and language are unnecessary, that only additional computation matters. I do not think the issue is settled. But language has proved valuable because it connects computation to the intelligence learned during pre-training and to concepts people can communicate.

世界模型：从理解到生成之路World Models and the Path From Understanding to Generation

主持人：“世界模型”是我们经常使用的另一个说法，但它的定义并不十分明确。有时感觉现在什么东西都被称为世界模型了。Moderator: "World model" is another phrase we use frequently, but it is not very well defined. It sometimes seems as though everything is called a world model now.

Shane Gu：我最近在一个世界模型研讨会上做了主题演讲，并鼓励大家参考计算机视觉和基于模型的强化学习领域中的经典定义。对我来说，世界模型本质上就是基于模型的强化学习中使用的模型：它能够表示世界如何变化，并支持规划或预测。这个定义已经足够，尽管研究人员也提出了一些有用的分类体系和细化定义。Shane Gu: I recently gave a keynote at a world-model workshop and encouraged people to look at classic definitions from computer vision and model-based reinforcement learning. For me, a world model is essentially the model used in model-based RL: a model that can represent how the world changes and support planning or prediction. That definition is sufficient, even though researchers have proposed useful taxonomies and refinements.

主持人：我仍然担心语言会成为一条狭窄的管道。如果一切都必须经过它，语言就会变成一种有损压缩层。Moderator: I still worry about language as a narrow pipe. If everything must pass through it, language becomes a lossy compression layer.

Shane Gu：我们提出的并非只有语言模型，而是将视频模型和语言模型结合起来。语言本身并不足够，这就是为什么视频是一种互补的基础模型。人们常把 Veo 或 Omni 看作制作精美视频的系统，但更宏大的愿景是：视频是一种缺失的基础模型。要实现与人类能力广度相匹配、而非只在狭窄的符号领域表现出色的通用人工智能，视频模型或许是必不可少的。Shane Gu: We are not proposing language alone. We are proposing the video model and the language model together. Language by itself is not sufficient; that is why video is a complementary foundation model. People often see Veo or Omni as systems for making beautiful videos, but the larger vision is that video is a missing foundation model. It may be required for an artificial general intelligence that matches the breadth of human capability rather than excelling only in a narrow symbolic domain.

主持人：许多最初从事计算机视觉的人，后来转向了基础模型、图像生成或视频生成。这个转型是否就像把图像到文本反过来变成文本到图像那么简单？Moderator: Many people who began in computer vision have moved toward foundation models, image generation, or video generation. Is that transition as simple as reversing image-to-text into text-to-image?

Dumitru Erhan：我就是沿着这条路走来的。我最初从事计算机视觉，研究目标检测和识别。生成并不只是逆向网络；它是一种更困难、也更模糊的映射。从图像得到“猫”这个标签，与从“猫”这个词生成无数种可能图像之一，有着天壤之别。与此同时，对理解能力的进步可以产生合成标签和信号，帮助生成，因此两者之间存在一个循环。Dumitru Erhan: I followed that path. I started in computer vision with object detection and recognition. Generation is not just the inverse network; it is a harder and more ambiguous mapping. Going from an image to the label "cat" is very different from going from the word "cat" to one of countless possible images. At the same time, progress in understanding can create synthetic labels and signals that help generation, so there is a loop between the two.

妮可·布里赫托娃：我们一再发现，理解得越深入，生成效果就越好。许多图像理解领域的研究人员后来转向了图像生成，其中一些人又进一步涉足视频领域，因为视频需要建模的维度更多。Nicole Brichtova: We have repeatedly seen that better understanding improves generation. Many image-understanding researchers became image-generation researchers, and some then moved into video, where there are many more dimensions to model.

Shane Gu：我建议从理解和识别入手，因为这些判别能力能够提升生成能力，而强化学习可以在其中发挥桥梁作用。我自己的研究历程先后涉及生成式算法、强化学习、机器人学、灵巧操作，之后转向语言模型。几年前，我得出结论：符号 AGI 的发展会快于物理 AGI，因此我转向了语言模型。近来，我很享受与 Omni 团队合作。Shane Gu: I recommend starting with understanding and recognition because those discriminative capabilities can lead to better generation, with reinforcement learning acting as a bridge. My own path moved through generative algorithms, reinforcement learning, robotics, dexterity, and then language models. Several years ago I concluded that symbolic AGI would advance faster than physical AGI, so I shifted toward language models. More recently I have enjoyed collaborating with the Omni team.

研究人员应该让自己接触不同社群中最优秀的人才和思维方式。如今的视频模型让我想起早期的语言模型。起初，语言模型只是能生成故事的创意演示。指令微调让它们能够作为聊天机器人使用，但它们仍会产生幻觉，而且不太擅长遵循指令。随着预训练和后训练逐渐强大到足以支持可靠推理，测试时扩展和强化学习催生了许多性能最优的系统。Researchers should expose themselves to the strongest people and ways of thinking in different communities. Today's video models remind me of early language models. At first, language models were creative demonstrations that could produce a story. Instruction tuning made them usable as chatbots, but they still hallucinated and followed instructions poorly. Once pre-training and post-training became strong enough for reliable reasoning, test-time scaling and reinforcement learning produced many of the best-performing systems.

视频模型可能也会沿着类似的路径发展。它们会提升遵循指令的能力，并减少虚假相关性，最终成为更可靠的世界模型。随后，可以将基于视频的时空模拟与文本模拟交替结合起来，以解决一系列广泛的智能问题。Video models may follow a similar path. They will improve in instruction following and reduce spurious correlations until they become more reliable world models. Then spatial-temporal simulation from video can be interleaved with textual simulation to solve a broad set of intelligence problems.

Dumitru Erhan：一个尚未解决的差异是，多媒体领域的理解与生成还没有完全统一起来。目前，真正同时擅长这两方面的前沿视频模型并不多。理解和生成似乎是一体两面，因此有理由将它们纳入同一个系统，即使我们如今并不总是这么做。Dumitru Erhan: One unresolved difference is that understanding and generation have not yet been fully unified in multimedia. There are not many frontier video models that are genuinely strong at both. Understanding and generation appear to be two sides of the same coin, so there is a case for putting them in the same system, even if we do not always build them that way today.

为什么音频与视频密不可分Why Audio and Video Belong Together

主持人：音频和视频一样难处理吗，还是说它们在本质上有所不同？几年前，研究人员曾将类似扩散的方法用于音频。人类对音频和视频的体验各不相同，但机器或许会把它们视为相似的信号。Moderator: Is audio as hard as video, or is it qualitatively different? Several years ago researchers used diffusion-like methods for audio. Humans experience audio and video differently, but perhaps a machine sees them as similar signals.

Dumitru Erhan：技术上存在差异，但相对较小。我们推出 Veo 3 时，音频开始成为我们工作的核心。它是首批能够真正进行视听联合生成的系统之一。其他系统可以在底层将不同组件组合起来，但这个模型能够同时生成画面和声音。Dumitru Erhan: There are technical differences, but they are relatively minor. Audio became central to our work when we shipped Veo 3, which was among the first systems to perform true joint audiovisual generation. Other systems could assemble separate components under the hood, but this model generated the picture and sound together.

我们之所以做出这个选择，是因为视听事件共享同一个潜在因果过程。有人说话时，显示人脸的像素和声音并非彼此独立。嘴唇必须与音频同步运动。早期的流程通常先生成像素，再添加单独的唇形同步层，效果很差。用户体验过具备原生音频的模型后，静音视频模型就不再让人觉得完整了。由同一个模型同时生成两种模态，才是正确的选择。We made that choice because audiovisual events share one latent causal process. When someone speaks, the pixels showing the face and the sound of the voice are not independent. The lips must move in synchrony with the audio. Earlier pipelines often generated pixels first and then added a separate lip-sync layer, with poor results. Once users experienced a model with native audio, a silent video model no longer felt complete. Generating both modalities in one model was the right choice.

Shane Gu：一个区别是，语音之外的音频很难用语言充分表达。相比之下，文本转语音比较容易描述，但音乐、一个人细微的语气或音高该如何描述呢？语言可能难以胜任。味觉、嗅觉、触觉以及肤色外观的细微变化也存在类似的局限。我们的感官系统对那些关系到生存的差异极为敏感，但日常词汇并未涵盖所有这些差异。Shane Gu: One difference is that audio outside of speech is less fully verbalized. Text-to-speech is comparatively straightforward to describe, but how do you describe music, a person's precise tone, or pitch? Language may be inadequate. Similar limitations exist for taste, smell, touch, and subtle variations in skin appearance. Our sensory systems are highly sensitive to distinctions that mattered for survival, but our everyday vocabulary does not encode all of them.

一位专业品酒师曾告诉我，他借用了描述人物和关系的语言，因为普通词汇不足以表达味道。声音也可能面临同样的问题。A professional wine taster once told me that he borrowed language used to describe people and relationships because ordinary vocabulary was not sufficient to express taste. Sound can have the same problem.

妮可·布里赫托娃：在某种程度上，视觉信息也有同样的局限。拥有成熟视觉品味或审美敏感度的人，能够感知那些难以用语言表达的风格。语言会限制我们描述丰富感官体验的能力。Nicole Brichtova: To some extent, visual information has the same limitation. People with a developed visual palette or aesthetic sensitivity can perceive styles that are difficult to verbalize. Language constrains any attempt to describe rich sensory experience.

这就是投资世界模型以及感知和生成的一个原因。感官信息在很大程度上决定了人类如何在世界中行动，也将决定具身人工智能如何在其中行动。语言已经带我们走了很远，也能带我们走得更远，但在其中许多领域，语言仍然存在局限。That is one reason to invest in world models and in perception and generation. Sensory information is a large part of how humans navigate the world and how embodied AI will navigate it. Language has taken us far and can take us farther, but it remains limiting in many of these areas.

Dumitru Erhan：我认为我们还没有在音频领域探索得足够深入，尚未发现根本性的上限。目前大多数弱点看起来仍是可解决的问题，可以通过增加数据、改进数据质量或改进建模来解决。眼下的限制可能在于音频描述，而不是音频生成本身。Dumitru Erhan: I do not think we have yet pushed audio far enough to discover a fundamental ceiling. Most current weaknesses still look like solvable problems involving more data, better data, or better modeling. The immediate limit may be captioning rather than audio generation itself.

Nicole Brichtova：参考素材很有帮助，因为很难准确描述一个人的说话方式：包括语气、韵律、不流畅之处以及房间的声学环境。存在整套专业词汇，而大多数用户并不了解。语言模型在训练时关注较少的领域也可能存在知识盲区；由于媒体系统依赖语言理解作为中介，这些盲区会在生成结果中显现出来。在声称已经知道这些领域的真正上限之前，我们需要继续推动它们的发展。Nicole Brichtova: References help because it is hard to describe exactly how somebody speaks: their tone, prosody, disfluencies, and room acoustics. Entire technical vocabularies exist that most users do not know. Language models may also have gaps in domains that received less attention during training, and those gaps become visible in generation because the media system relies on language understanding as an intermediary. We need to keep pushing these areas before claiming to know their true ceilings.

感官细节、真实感与人类偏好Sensory Detail, Realism, and Human Preference

主持人：作为一名播客主播，我会从三个大类来思考声音：音乐、人声和音效。即便是人声，房间的声音也会改变听感。大房间、小房间、汽车里、电话里，以及面对面交谈，听起来都不一样。这些特性可以被标注出来，但我们的感知要丰富得多。Moderator: As a podcaster, I think about sound in three broad domains: music, voice, and effects. Even within voice, room sound changes the experience. A large room, a small room, a car, a phone, and an in-person conversation all sound different. These properties can be labeled, but we perceive them in much richer ways.

AI 视频的一个常见破绽是音频过于录音棚级的完美，因为训练素材是在录音棚里录制的。即使是声音，也需要世界模型：如果我移得更远，我的声音就应该变得更轻、更分散。沉浸式视频需要对这种空间关系进行建模。One common tell of an AI video is audio that is too studio-perfect because the training material was recorded in studios. A world model is needed even for sound: if I move farther away, my voice should become softer and more diffuse. Immersive video needs to model that spatial relationship.

Shane Gu：这是一个很好的例子，说明语言不足以精确表达某些信息。如果字幕未能体现回声或环境噪声，那么以该语言为条件进行生成，仍会具有很强的多模态性，且难以预测。理想情况下，潜在表示应捕捉大部分随机变化，这样基于它进行最终生成时，结果就会更加确定。Shane Gu: That is a good example of information for which language is not precise enough. If a caption fails to represent the echo or environmental noise, generation conditioned on that language remains highly multimodal and unpredictable. Ideally the latent representation should capture most of the stochastic variation, so the final generation conditioned on it becomes much more determinate.

Nicole Brichtova：面部表情是另一个例子。人们对微表情、目光、皮肤纹理，以及听者点头或回应的方式极其敏感。一年来，我们的表现有了巨大提升，但视频还没有完全跨越那道鸿沟。图像可能更接近这一目标：如今许多生成图像乍看之下已与照片难以区分。Nicole Brichtova: Facial expressions are another case. People are extremely sensitive to micro-expressions, gaze, skin texture, and the way a listener nods or reacts. We have improved dramatically in a year, but video has not fully crossed that chasm. Images may be closer: many generated images now look indistinguishable from photographs at first glance.

杜米特鲁·埃尔汗：我们曾做过一项实验，想看看生成的视频能否比真实视频看起来更好。我们取真实视频片段的描述，生成对应的视频片段，并在人工评估中将两者并排展示。人们往往更喜欢生成的视频。Dumitru Erhan: We once ran an experiment asking whether generated videos could look better than real videos. We took descriptions of real clips, generated equivalent clips, and showed them side by side in a human evaluation. People often preferred the generated versions.

这并不意味着生成的片段更有用或更逼真。它可能只是更锐利、色彩更饱和、更像 HDR，或者更讨人喜欢。我们优化的是那些能迅速引发积极反应的特质。深入观察后，真实视频可能仍然更加连贯或更有信息量。That does not mean the generated clip was more useful or more realistic. It may simply have been sharper, more saturated, more HDR-like, or more flattering. We optimized qualities that trigger a quick positive reaction. On deeper inspection, the real video might still be more coherent or informative.

Shane Gu：敏感度也因人和文化而异。我曾和一位漫画家聊过，他觉得生成图像令人不安，因为人物的眼神方向有些微妙的不对劲。大多数人都会忽略的细微偏差，在专家眼中却可能显得极不自然。Shane Gu: Sensitivity also varies by person and culture. I spoke with a manga artist who found generative imagery unsettling because the eye gaze was subtly wrong. A small misalignment that most people ignore can feel deeply unnatural to an expert.

Dumitru Erhan：这就是为什么单靠人类偏好并不是一个可靠的目标。询问人们是否更喜欢某张图片，可能会让模型优化表面的吸引力，却无法带来你真正想要的结果。Dumitru Erhan: That is why human preference alone is not a reliable target. Asking whether people like one image more than another may optimize superficial attraction without producing the outcome you actually want.

美学默认设置、提示词与专家品味Aesthetic Defaults, Prompting, and Expert Taste

Shane Gu：几年前，人们曾争论提示词工程是否会消失。我认为它不应该消失，因为明确说明你的需求，是控制模型输出的方式。如今，模型可以更自动地生成提示，往往也能产出足够好的结果，但用户仍应不断提升自己的判断力，描述尚存的差异，而不是接受第一次生成的结果。Shane Gu: Several years ago, people debated whether prompt engineering would disappear. I argued that it should not, because specifying what you want is how you control a model's output. Models can now perform much more automatic prompting and often produce something sufficient, but users should keep refining their sensitivity and describing the remaining differences rather than accepting the first generated result.

妮可·布里赫托娃：未经训练的眼睛与数十年来精进技艺的人之间，存在着巨大的差异。设计师、建筑师、艺术家以及其他专家能看到普通人看不到的东西。如果你在街头随机采访人们，他们可能会更喜欢非常平滑、色彩高度饱和的图像——相当于套用了 Instagram 滤镜。但这未必适合作为每项任务的默认选择。Nicole Brichtova: There is a large difference between the untrained eye and somebody who has honed a craft for decades. Designers, architects, artists, and other experts see things the average person does not. If you poll people on the street, they may prefer a very smooth, highly saturated image—the equivalent of an Instagram filter. That is not necessarily the right default for every task.

我们的模型需要具备强大的指令遵循能力，让用户能够引导模型偏离默认表现。有时语言就足够了；有时视觉或音频参考更有效，因为语言无法表达期望的差异。Our models need strong instruction following so users can steer away from the default. Sometimes language is enough; sometimes visual or audio references are more effective because language cannot express the desired difference.

主持人：作为一名产品负责人，你是否会因必须为全世界设定一种审美默认选项而感到压力？每个系统都有默认设置，无论有没有人打算做出文化层面的选择。Moderator: As a product leader, do you feel pressure from having to set an aesthetic default for the world? Every system has a default, whether or not anyone intended to make a cultural decision.

Nicole Brichtova：我们考虑过这个问题。Nano Banana Pro 发布时，生成的信息图迎来了爆发式增长。我们的默认美学有时显得过于杂乱，就像一个求知心切的学生刚学到了大量有关某个主题的知识，便试图把每条事实都塞进一张图里。效果也会因语言和文化风格而异；例如，日文信息图往往倾向于更高的信息密度。Nicole Brichtova: We have thought about it. When Nano Banana Pro came out, there was an explosion of generated infographics. Our default aesthetic was sometimes too cluttered, as if an overeager student had learned a great deal about a topic and tried to put every fact into one image. The effect also varies by language and cultural style; Japanese infographics, for example, often favor much higher information density.

Dumitru Erhan：我们在 Omni 上也面临过类似的选择。在训练和调优接近尾声时，我们必须比较不同风格，并决定更喜欢较柔和还是更饱和的色彩。这些都是实实在在的取舍。Dumitru Erhan: We faced similar decisions with Omni. Near the end of training and tuning, we had to compare styles and decide whether we preferred more muted or more saturated colors. Those are real trade-offs.

妮可·布里赫托娃：这些决策往往由建模和产品团队做出。我们有理由反思，自己是否是做这些决策的最佳人选，还是应该与艺术总监以及拥有鲜明创意视角的人更紧密地合作。我们会邀请值得信赖的测试人员参与测试，并每周开展反馈交流，也会认真对待他们的意见。内部创意专家也会测试各个检查点，并发现核心团队忽略的问题。Nicole Brichtova: The decisions often fall to modeling and product teams. It is fair to ask whether we are the best people to set them or whether we should work more closely with art directors and people with a distinctive creative perspective. We use trusted testers and weekly feedback sessions, and we take their input seriously. Internal creative experts also test checkpoints and notice things the core team misses.

测试人员可能会报告说，某项优化损害了草地细节；也可能注意到一种奇怪的重复伪影，比如太多人的手上都出现了婚戒。基于偏好的训练可能会以意想不到的方式造成虚假相关性和奖励投机。这些问题在模型开发过程中很容易被忽略，却会被高频使用模型的人一眼看出。A tester may report that an optimization ruined the detail in grass, or notice a strange repeated artifact such as wedding rings appearing on too many hands. Preference-based training can create spurious correlations and reward hacking in unexpected ways. These issues are easy to miss when developing a model and obvious to somebody who uses it intensely.

评估生成式媒体Evaluating Generative Media

主持人：除了依靠少数出色的测试人员，你们如何评估视频？想必也有定量方法吧。Moderator: How do you evaluate video beyond relying on a few exceptional testers? Surely there must also be quantitative methods.

Dumitru Erhan：提升 Gemini 的能力有助于改进视频自动评估，但这仍然是一个非常困难的问题。Dumitru Erhan: Improving Gemini improves automatic evaluation for video, but the problem remains very hard.

妮可·布里赫托娃：有些属性相对客观。对于信息图，光学字符识别可以识别出字母变形。如果文字有误，这项素材可能就无法使用。这些情况更容易通过自动化方式评估。Nicole Brichtova: Some properties are relatively objective. For an infographic, optical character recognition can reveal a malformed letter. If the text is wrong, the asset may be unusable. Those cases are easier to evaluate automatically.

美学和开放式视频质量要难得多。我们在很大程度上依赖人工评估。每个新模型都会增加新的能力，也带来更多需要测试的维度。当两个检查点表现接近时，我们中有时会有一群人真的坐在同一个房间里，将视频并排对比，讨论更喜欢哪一个输出。Aesthetics and open-ended video quality are much harder. We rely extensively on human evaluations. Every new model adds capabilities, which creates more dimensions to test. When two checkpoints are close, a group of us may literally sit in a room and compare videos side by side, asking which output we prefer.

自由形式的视频编辑尤其困难，因为可能的请求几乎没有穷尽。即使是添加一个物体这样看似简单的功能，也需要从视觉一致性、音频、语义和指令遵循等方面进行评估。没有单一的“添加一只树懒”基准能够涵盖整个问题。Free-form video editing is especially difficult because the possible requests are nearly unlimited. Even a seemingly simple capability such as adding an object requires evaluation across visual consistency, audio, semantics, and instruction following. There is no single "add a sloth" benchmark that covers the entire problem.

评估规模远超一屋子评审人员所能应对的范围。我们对数千个示例开展人工评估，通过线上实验以更大规模检测细微差异，并开发自动评分器。自动评估在语言模型领域已相当成熟，但在媒体领域仍处于起步阶段。因此，人工判断依然至关重要，尤其是那些审美观点鲜明的人，以及在日常工作流程中使用这些模型的专业人士所提供的反馈。The evaluation scale is larger than a roomful of reviewers. We run human evaluations over thousands of examples, use live experiments to detect small differences at greater scale, and develop automatic raters. Automatic evaluation is mature for language models but much more nascent for media. Human judgment therefore remains important, especially feedback from people with a strong aesthetic point of view and from professionals who use the models in daily workflows.

模型在某个基准测试子集上表现出色，仍可能导致实际工作流程出错。通过早期访问计划，我们可以观察这些问题，并在大范围发布前吸纳反馈。A model can perform well on a benchmark slice and still break a real workflow. Early-access programs allow us to observe those failures and incorporate feedback before a broad release.

Shane Gu：人工评估工作应逐步转化为更完善的自动化系统。视频理解是一项重要的智能任务，尤其是在检测生成媒体中的错误方面。有些问题关乎美学，但另一些则关乎语义。对某个著名电影场景的生成式重建可能看起来合情合理，却包含与上下文格格不入、毫无意义的矛盾之处。AI 评估器最终应该能够检测出这一点。Shane Gu: Human evaluation work should gradually be amortized into better automatic systems. Video understanding is an important intelligence task, especially for detecting failures in generated media. Some problems are aesthetic, but others are semantic. A generated reconstruction of a famous movie scene may look plausible while containing an inconsistency that makes no sense in context. An AI evaluator should eventually detect that.

如今评估生成的视频时，我会觉得自己很聪明，因为我仍能发现模型漏掉的错误。我们应该努力缩小这一差距。强大的视频理解能力不只是关乎简单的偏好或美感，最终还应能减少对重复性人工标注的需求。When I assess generated videos today, I can feel clever because I still notice errors the model misses. We should work to close that gap. Strong video understanding matters beyond simple preference or beauty, and it should eventually reduce the need for repetitive human labeling.

数据与工作流仍需的模型The Data and Workflows Models Still Need

主持人：许多公司都想向你提供视频、游戏、真实世界的影像、图片或标注服务。你们实际需要什么数据？从海量信息中筛选出有用信号并不容易，因此，如果生态系统能了解什么样的提案才有价值，将会有所帮助。Moderator: Many companies want to offer you video, gaming, real-world footage, images, or labeling services. What data do you actually need? Sorting signal from noise is difficult, so it would help the ecosystem to know what makes a useful proposal.

Dumitru Erhan：如果不透露我们项目的细节，这个问题很难回答。总体而言，我们已经拥有大量数据，因此质量比盲目追求数量更重要。具身数据与我们的机器人研究相关。对于 Omni 来说，专业制作的高质量素材可能比又一批杂乱收集的网络视频更有用。我们一直在寻找这类数据来源。Dumitru Erhan: It is hard to answer without revealing details of our projects. In general, we already have a great deal of data, so quality matters more than indiscriminate volume. Embodied data is relevant to our robotics work. For Omni, professionally produced, high-quality material can be more useful than another random collection of internet video. We are always looking for such sources.

Nicole Brichtova：对于智能体系统，真实任务的执行轨迹很有价值，也难以人为编造。以营销活动为例，一个人可能从一张新产品照片开始，将其制作成视频广告，然后再改编成适用于不同平台的多种素材。决策、迭代和转换的过程都很重要。Nicole Brichtova: For agentic systems, the trajectory of a real task is valuable and difficult to manufacture. Consider a marketing campaign. A person might start with a photograph of a new product, turn it into a video advertisement, and then adapt it into multiple assets for different platforms. The sequence of decisions, iterations, and transformations is important.

这些信息通常存在于人们实际开展工作的产品中。通用型供应商往往无法获取完整的工作流程。对于某些专业任务，我们也可能缺少自有产品入口。如果有人能够很好地捕捉这些真实的工作轨迹，我们愿意听听他们的想法。That information usually lives inside the products where people actually do the work. A generic vendor often does not have access to the full workflow. We may also lack a first-party surface for a particular professional task. If someone can capture those authentic trajectories well, we would like to hear from them.

主持人：媒体和营销团队有做不完的工作。其中一些工作需要精雕细琢，值得好莱坞级别的关注，但很多工作都重复性很强，应该可以通过建模来处理。Moderator: Media and marketing teams have endless amounts of work. Some of it is artisanal and deserves Hollywood-level attention, but much of it is repetitive and should be modelable.

小组成员：我们两方面都需要：高质量的匠心之作，以及大量常规生产工作。Panelist: We need both sides: high-quality craft and large volumes of ordinary production work.

Nicole Brichtova：即便是一场普通的营销活动，也不可能在五分钟内完成。人们会反复修改、比较各种方案，并基于某些可能永远不会被记录下来的理由，选定其中一个。也许是因为目光传递的感觉恰到好处，构图符合品牌调性，或是某个小细节体现了品质。模型构建者可能并不了解这些，因为我们不是营销总监。Nicole Brichtova: Even an ordinary marketing campaign is not created in five minutes. People iterate, compare alternatives, and choose one option for reasons that may never be written down. Perhaps the gaze felt correct, the composition matched the brand, or a small detail communicated quality. Model builders may not know these things because we are not marketing directors.

Shane Gu：自然语言也是如此。大多数信息仍然存在于人们头脑中，只有通过主动对话和建立信任才能提取出来。互联网上往往只有最终成果，比如论文或文章，却没有完整的发展轨迹：最初的起点、催生灵感的对话、失败的尝试，或灵光乍现的时刻。创作工作中蕴含着大量这类隐性或“暗知识”。Shane Gu: The same is true in natural language. Most information is still inside people and can be extracted only through active dialogue and trust. The internet often contains the final outcome, such as a paper or an article, but not the trajectory: the starting point, the conversations that sparked an idea, the failed approaches, or the moment of inspiration. Creative work contains a great deal of this tacit or "dark" knowledge.

妮可·布里赫托娃：小说常常能打动读者，是因为作者与故事、发展脉络或人物有着个人联系。如今，许多语言模型生成的文字都落入了默认模式：措辞、结构和描写大同小异，读者很快就会察觉，其中没有什么能让人产生个人共鸣。这种专业知识和经验的缺失，在生成式媒体领域同样重要。Nicole Brichtova: A novel often speaks to a reader because the author has a personal connection to its story, trajectory, or characters. Much of today's language-model writing falls into default patterns: the wording, structure, and descriptions feel similar, and readers quickly sense that there is nothing personal to connect with. That missing expertise and experience also matters in generative media.

从客户反馈到更优模型From Customer Feedback to Better Models

Shane Gu：Google Cloud 和 Google DeepMind 一直在加大对前沿部署工程师的投入，以支持产品开发和创意生成媒体工作。仅靠公开数据是有局限的。与领域专家合作，能够打造更出色的模型和产品。Shane Gu: Google Cloud and Google DeepMind have been investing more in forward-deployed engineers for product and creative generative-media work. Public data alone has limits. Partnership with domain experts can create better models and products.

我广义地将后训练定义为预训练与最终用户体验之间的一切。前沿部署工程不应被简化为销售。这一角色可以帮助客户使用模型，但也应从中提炼出能让双方受益的洞见。这些工程师可以围绕客户使用系统的方式构建配套工具，同时将真实反馈传递给上游的建模团队。I define post-training broadly as everything between pre-training and the final user experience. Forward-deployed engineering should not be reduced to sales. The role can help a customer use a model, but it should also derive insights that benefit both sides. Those engineers can build the harness around how a customer uses the system while carrying real feedback upstream to the modeling teams.

妮可·布里赫托娃：直接与使用这些模型的人交流并了解他们在哪些方面遇到困难，对我们非常有帮助。室内设计师可能需要在十种地毯尺寸（包括定制尺寸）上始终如一地复现同一种图案，但模型却会改变图案。为耳环做虚拟试戴的人，则需要珠宝和佩戴者的头部保持符合物理实际的比例。即使建模团队从未想到要测试这些问题，这些缺陷在实际工作中仍然很重要。Nicole Brichtova: It is extremely useful for us to speak directly with people using the models and understand where they struggle. An interior designer might need to reproduce a pattern consistently across ten rug sizes, including a custom size, but the model changes the pattern. Someone doing virtual try-on for earrings needs the jewelry and the person's head to remain at physically meaningful scales. These failures matter in real work even if the modeling team never thought to test them.

广告引入了品牌语言这一概念。公司可能会提供介绍这种语言的图片和 PDF，但这一概念仍然模糊不清。宜家的品牌形象不仅仅是“蓝色和黄色”；具体的色调、比例、材质、构图和视觉风格都很重要。随意搭配蓝色和黄色，并不一定就能呈现宜家的风格。仅靠前沿模型研究，并不能自动满足这些要求。Advertising introduces the idea of a brand language. A company may provide images and PDFs describing that language, but the concept remains ambiguous. IKEA's identity is not merely "blue and yellow"; the exact shades, proportions, materials, composition, and visual tone matter. A random blue and yellow result is not necessarily IKEA. Frontier-model research alone does not automatically capture such requirements.

我们希望打造能够解决具体任务的产品，而不是让成果停留在研究阶段。为此，我们需要了解用户和专家真正关注什么，将这些需求纳入评估，并建立一条直达模型的反馈渠道。We want to build products that solve concrete tasks rather than remain research artifacts. To do that, we need to understand what users and experts actually care about, incorporate those requirements into evaluation, and create a feedback path all the way to the model.

主持人：短短几年间，Nano Banana、Veo 和 Omni 取得了令人瞩目的进展，而显然，除了编程之外，还有许多领域值得探索。感谢您拨冗参与。我迫不及待想看看接下来会有什么新进展。Moderator: The progress in Nano Banana, Veo, and Omni over only a few years has been remarkable, and there is clearly much more to explore beyond coding. Thank you for your time. I cannot wait to see what comes next.

各位嘉宾：感谢邀请我们，也感谢大家。Panelists: Thank you for having us, and thank you, everyone.