和 AIE NYC 一起观看，下个月来旧金山参加 AIE CODE 吧！Watch along AIE NYC and join us at AIE CODE in SF next month!

本集包含 Taku 机器人的现场演示，在 YouTube 上观看效果最佳。This episode includes a live demo of the Taku robot that is best enjoyed on YouTube.

上个月，Dyna 宣布推出 Dyna-2，引发广泛关注。这是一款在一百万小时人类视频上预训练的世界动作模型，并声称在跨越四个数量级的人类数据上发现了几条新的缩放定律。如今，他们又推出了一款实体机器人智能体 Dyna-2.1，其表现令人难以置信：Last month, Dyna made waves by announcing Dyna-2, a world-action model pre-trained on one million hours of human video, claiming several new scaling laws on human data across four orders of magnitude. They have now followed up the model with a physical robot agent, Dyna-2.1, that has to be seen to be believed:

Dyna Robotics@DynaRobotics

We are releasing Dyna-2.1, the first Physical Agent that achieves reliable super long-horizon whole-body autonomy. It combines our brand-new semi-humanoid hardware with an agentic system built around Dyna-2 to handle ultra-long real-world workflows. Here is an uncut footage of …

4:24 PM · Sep 29, 2026 · 3.12M Views

122 Replies · 386 Reposts · 4.72K Likes

在今年的 AIE 演讲中，Jason 谈到了现实世界中的机器人技术面临的真正挑战：实现可靠的长时程自主能力；他还介绍了 Dyna 如何让模型连续 24 小时折叠餐厅餐巾，成功率达到 99.4%，甚至在机器人把整摞餐巾都弄倒后，也能恢复作业。In his talk at AIE this year, Jason talked about how the real challenge for robotics in the real world is reliable, long-horizon autonomy, and how Dyna got its models to a 99.4% success rate folding restaurant napkins for 24 hours straight, including recovering after it pulled over the whole stack.

从使用超过一百万小时的人类视频训练世界动作模型，到部署机器人每天在鼎泰丰折叠数千张餐巾，Dyna Robotics 正在推动具身 AI 走出令人惊叹的演示，进入真实生产环境。本期节目中，Dyna Robotics 联合创始人兼首席科学家 Jason Ma 到访现场，与 swyx 一起探讨：机器人如何仅凭 45 分钟的演示就学会新任务；预测物理世界的未来为何可能是实现通用机器人技术的关键；以及要打造能为真实客户可靠工作的机器人，究竟需要做到什么。From training a world-action model on over one million hours of human video to deploying robots that fold thousands of napkins daily at Din Tai Fung, Dyna Robotics is pushing physical AI beyond impressive demos and into real-world production. In this episode, Dyna Robotics co-founder and Chief Scientist Jason Ma joins swyx on-site to unpack how robots can learn new tasks from as little as 45 minutes of demonstrations, why predicting the future of the physical world may be the key to general-purpose robotics, and what it actually takes to build robots that work reliably for real customers.

我们深入探讨 Dyna-2——Dyna 的百万小时世界动作模型，它将未来视频预测与机器人动作相结合；还讨论了支撑人类技能向机器人迁移的缩放定律、扩散 Transformer 与流匹配，以及合成环境和自生成训练数据为何可能开启机器人技术进步的新一波浪潮。Jason 还解释了 Dyna 为何选择自研硬件，而不依赖现成的人形机器人；智能体框架如何让机器人执行长达数小时的工作流程；灵巧的机械手和缺失的感官数据为何仍是主要瓶颈；以及真正的商业化部署与机器人创业公司的“坟场”有何不同。我们还会探讨世界动作模型与视觉-语言-动作模型的区别、GPT-6 和机器人领域的编码智能体，以及 Jason 从 AlphaGo 和宾夕法尼亚大学到 NVIDIA、Google DeepMind，最终创立 Dyna 的历程。We go deep on Dyna-2, Dyna’s million-hour world-action model that combines future video prediction with robot actions, the scaling laws behind human-to-robot transfer, diffusion transformers and flow matching, and why synthetic environments and self-generated training data could unlock the next wave of robotics progress. Jason also explains why Dyna builds its own hardware instead of relying on off-the-shelf humanoids, how agentic harnesses enable robots to execute hour-long workflows, why dexterous hands and missing sensory data remain major bottlenecks, and what separates real commercial deployments from the graveyard of robotics startups. We also cover world-action models versus vision-language-action models, GPT-6 and coding agents in robotics, and Jason’s journey from AlphaGo and Penn to NVIDIA, Google DeepMind, and founding Dyna.

我们将讨论：We discuss:

• 为什么 Dyna-2 使用超过一百万小时的人类视频进行训练

  Why Dyna-2 was trained on over one million hours of human video

• 世界动作模型：预测未来视频以指导物理机器人的动作

  World-action models: predicting future video to guide physical robot actions

• 扩散 Transformer、流匹配与实时机器人控制

  Diffusion transformers, flow matching, and real-time robotic control

• 机器人如何仅通过 45 分钟的演示学会新任务

  How robots learn new tasks with as little as 45 minutes of demonstrations

• 为什么机器人领域没有类似互联网的东西，而数据仍然是其规模化发展面临的最大挑战

  Why robotics has no equivalent of the internet and data remains its biggest scaling challenge

• 人类第一视角视频、机器人演示以及跨具身迁移

  Human egocentric video, robot demonstrations, and cross-embodiment transfer

• 合成环境、仿真，以及生成自身训练数据的模型

  Synthetic environments, simulation, and models that generate their own training data

• 为什么机器人需要设备端推理、模型蒸馏和定制硬件

  Why robots need on-device inference, model distillation, and custom hardware

• Dyna为何选择轮式半人形机器人，而非传统双足机器人

  Why Dyna chose wheeled semi-humanoids instead of traditional bipedal robots

• 硬件瓶颈：为什么灵巧机器人手仍会过热

  The hardware bottleneck: why dexterous robot hands still overheat

• 贝斯特韦斯特、鼎泰丰，以及机器人试点与规模化部署之间的差异

  Best Western, Din Tai Fung, and the difference between robotics pilots and scaled deployments

• 机器人领域的苦涩教训：扩展数据规模，而非设计专用解决方案

  The Bitter Lesson of Robotics: scaling data instead of engineering specialized solutions

• 用于执行多步骤、持续数小时的物理工作流的智能体工具框架和机器人

  Agentic harnesses and robots executing multi-step, hour-long physical workflows

• GPT-6、编码智能体，以及用于机器人研究的 AI 生成仿真环境

  GPT-6, coding agents, and AI-generated simulation environments for robotics research

• 世界动作模型与 VLA 的比较，以及为什么仅靠视觉可能还不够

  World-action models vs. VLAs, and why vision alone may not be enough

• Jason 在 AlphaGo、宾夕法尼亚大学、NVIDIA、Google DeepMind 和 Eureka 项目中的历程

  Jason’s journey through AlphaGo, Penn, NVIDIA, Google DeepMind, and the Eureka project

• Dyna 的研究文化、部署优先理念，以及让数百万台机器人投入使用的愿景

  Dyna’s research culture, deployment-first philosophy, and vision for millions of robots

• LinkedIn: https://www.linkedin.com/in/jason-ma-742224a2/

• X: https://x.com/JasonMa2020

Dyna RoboticsDyna Robotics

• Website: https://www.dyna.co/

• X: https://x.com/DynaRobotics

时间戳Timestamps

00:00:00 介绍 Dyna-2 的世界动作模型00:00:00 Introduction and Dyna-2’s World-Action Model

00:03:08 扩散 Transformer、流匹配与通用机器人00:03:08 Diffusion Transformers, Flow Matching, and Generalist Robots

00:08:19 机器人数据瓶颈00:08:19 The Robotics Data Bottleneck

00:13:50 合成数据与自我改进的世界模型00:13:50 Synthetic Data and Self-Improving World Models

00:16:51 实时推理与模型蒸馏00:16:51 Real-Time Inference and Model Distillation

00:19:45 全栈机器人：硬件与供应链00:19:45 Full-Stack Robotics: Hardware and Supply Chains

00:25:02 Dyna为何打造轮式半人形机器人00:25:02 Why Dyna Built a Wheeled Semi-Humanoid

00:29:35 教授机器人新任务并评估其表现00:29:35 Teaching Robots New Tasks and Evaluating Performance

00:31:59 自主工作流与硬件限制00:31:59 Autonomous Workflows and Hardware Limitations

00:36:06 贝斯特韦斯特和鼎泰丰的实际部署案例00:36:06 Real-World Deployments at Best Western and Din Tai Fung

00:41:07 机器人领域的苦涩教训00:41:07 The Bitter Lesson of Robotics

00:46:13 面向物理 AI 的智能体运行框架00:46:13 Agentic Harnesses for Physical AI

00:49:07 GPT-6、编程智能体与 AI 原生机器人技术00:49:07 GPT-6, Coding Agents, and AI-Native Robotics

00:52:02 世界动作模型与 VLA 的对比00:52:02 World-Action Models vs. VLAs

00:59:17 From AlphaGo to Founding Dyna Robotics

01:04:20 Dyna 的研究文化与机器人技术的未来01:04:20 Dyna’s Research Culture and the Future of Robotics

Transcript

Introduction: Jason Ma and Dyna Robotics

Swyx [00:00:00]：好，我们现在来到 Dyna Robotics 的联合创始人兼首席科学家 Jason Ma 的办公室。感谢你邀请我们来这里。Swyx [00:00:00]: All right, we’re here on site with Jason Ma, the co-founder, chief scientist of Dyna Robotics. Thanks for having us in your office.

Jason Ma [00:00:06]：当然。谢谢你们前来。Jason Ma [00:00:06]: Of course. Thank you for coming.

Vibhu [00:00:07]：这次真是太尽兴了。我们看到了——你们也会看到很多幕后花絮——饮料被调制出来，做了很多评测，还有机器人第一次拿着刀。所以，支撑这一切的底层是一个世界动作模型。这是什么？Vibhu [00:00:07]: It’s been a blast. We got to see—you’ll see plenty of B-roll—drinks being pulled, a lot of evals, robots with knives for the first time. So under the hood of all this you have a world action model. What is that?

Dyna-2：预测未来的世界动作模型Dyna-2: A World Action Model That Predicts the Future

Jason Ma [00:00:22]：上个月，我们发布了名为 Dyna-2 的世界动作模型。这是首个使用超过一百万小时数据训练的世界动作模型。先来回答你的第一个问题：世界动作模型能够预测未来的视频，也就是世界未来的状态，因此得名“世界模型”。此外，它还可以利用生成的未来画面，生成机器人能够在物理世界中执行的动作，从而控制机器人。我们为什么要这样做？因为这是一种非常强大的架构。想想人类在物理世界中的行动方式：我们的大脑会下意识地思考，世界会如何随着我们的行动而演变。现在，我们拥有了一种能够做到这一点、并据此引导机器人行动的模型架构。我们发现，这样一来，模型的灵活性和稳健性都大大提升，也更能泛化到从未见过的环境和场景中。Jason Ma [00:00:22]: So last month, we released our world action model called Dyna-2. So it’s the first world action model trained on more than one million hours of data. So going back to your question first, world action model is a model that’s able to predict the future video, the future state of the world, so hence the name world model. But in addition, you can also use that future generation to also produce actions robots can execute in the physical world to control the robot. So why do we do that? It’s a very powerful architecture, because if you think about how humans act in the physical world, in our brain, we are implicitly thinking about how the world will evolve based on our actions. So now we have a model architecture that’s able to do that to guide robot actions. So what we find is that by doing so, the model is much more dexterous, much more robust, and much more capable of generalizing to environments, to scenarios it’s never seen before.

世界动作模型：输入、输出与视觉推理World Action Models: Inputs, Outputs, and Visual Reasoning

Vibhu [00:01:17]：我想稍微深入追问一下这个问题。我觉得很多人一听到“世界模型”，就会以为你们输出的是视频、世界状态或者 3D 状态。其实你们也会输出动作。我觉得大多数人都不了解机器人技术背后的模型是什么样的。那么输入和输出分别是什么？训练数据是什么？架构又是什么？请详细给我们讲讲。Vibhu [00:01:17]: I wanna prompt this question a bit. I think a lot of people hear world model, and they think you’re outputting video or world or 3D state. You’re also doing action. I think most people don’t understand what models are under the hood of robotics. So what are the inputs and outputs? What’s the training data? What is the architecture? Give us all the details.

Jason Ma [00:01:39]：对于输入来说，最重要的输入是视频，对吧？你可以把它想成机器人用摄像头拍摄的视频，对吧？也就是过去很多秒的视频或图像，这就是输入。但对机器人来说，真正有意思的是，机器人自身还有很多执行器和关节编码器。所以机器人也知道自己各个关节的角度。这些也可以作为输入，输入到这个世界模型中。这和纯粹只看视频的视频模型非常不同，对吧？这就是输入。此外，对于机器人，你还希望能够和它对话，指示它执行不同的任务，对吧？所以也可以输入语言指令。我认为，这些就是任何机器人模型（包括世界动作模型）的基本输入。至于输出，最基本的是需要输出机器人能够在硬件上执行的动作。但对于世界动作模型来说，真正有意思的是，除了动作之外，它还可以输出未来的视频，对吧？所以基本上，你可以先输出未来的视频，或者它的某种表示，再用它来引导机器人的动作。我认为这有点类似于思维链。基本上，模型是在视觉和物理空间中进行推理，然后输出动作。这就是世界动作模型的输出与其他类型的模型，也就是机器人模型的输出之间的关键区别。Jason Ma [00:01:39]: So, for inputs, the most important input is video, right? You can think about videos that the robot is capturing with its cameras, right? So video or images of the past many seconds, that’s the input. But for robots, what’s really interesting is that robot also has a lot of actuators and joint encoders on itself. So the robot also knows, the angles of its joints. So those can also be inputs to this world model. That’s something that’s very different from just a pure video model that’s only looking at videos, right? So that’s the input. And additionally, for robots, you want to be able to talk to it, to instruct it to do different tasks, right? So you can also feed it language instruction. So those are what I consider the basic set of inputs that go into any robotics models, including world action models. And once we have that the output, at the basic level, it needs to output actions the robots can execute on the hardware. But in the case of world action model, what’s really interesting is that in addition to the actions, it can also output future video, right? So basically, you can almost output the future video or some representation of it first, and then use that to guide robot actions. So the way I think about it is this is kind of analogous to chain of thought, you know. And basically, the models are reasoning in the visual, the physics space, and then outputting actions. So that’s the key difference between world action models outputs versus other kind of models, robot models outputs.

扩散 Transformer、流匹配与平滑动作Diffusion Transformers, Flow Matching, and Smooth Actions

Vibhu [00:03:08]：你围绕自己的模型发布了一篇很不错的博客文章。我觉得很多人都了解语言模型，或者说 Transformer、下一个 token 预测器。在这个案例中，你分享了相当多的内容：有扩散、流匹配，你们预测的是下一个世界状态以及动作。你提到的一点是“一百万小时”。你们是如何确定架构的？比如说，有哪些选择？架构是什么？我觉得在语言模型领域，很多时候就是拿一个现成的——用数万亿个 token 做预测、SFT、RL，然后就有了一个模型。那你们这个是怎样的？你们是如何从零开始做出自己的模型的？有没有现成的基础架构可以直接用？这些都是你们自己确定的吗？目前的进展又如何？Vibhu [00:03:08]: You put out a good blog post around your model. I think a lot of people understand language models or transformers, next-token predictors. In this case, you shared quite a bit. There’s diffusion, flow matching, you’re predicting next world state plus action. One thing you say is a million hours. How do you determine the architecture? Like, what are the choices there? What is the architecture? I feel like in language models, there’s a lot of take something off the shelf- Take trillions of tokens, do prediction, SFT, RL, and you have a model. What about this? How do you go from nothing to your model? Is there vanilla architecture off the shelf? Do you determine all this and what’s, what is the current state?

Jason Ma [00:03:49]：是的，我觉得，是的。所以，思考如何为机器人技术构建模型的思维框架是：本质上有几个方面值得关注。一个是数据，第二个是如何让模型从这些数据中尽可能多地学习。那我们先来谈谈模型本身，好吗？我们发布的那篇博客介绍的是我们的 Dyna-2 世界动作模型。它底层采用的是扩散 Transformer。该模型使用仅编码器的 Transformer 架构。Jason Ma [00:03:49]: Yeah, I think, yeah. So the mental framework to think about how do we build models for robotics is that there are essentially a couple things to look at. One is data, and the second is how do we make a model learn as much as possible from that data? So let’s first talk about the model actually, right? So the blog post we released is our Dyna-2 world action model. So under the hood is a diffusion transformer. The model uses an encoder-only transformer architecture.

Vibhu [00:04:20]: Right, yeah.

Jason Ma [00:04:20]：但真正有意思的是，在视觉领域，比如生成图像、生成视频，以及这里提到的机器人观察图像并据此生成动作，人们发现扩散模型架构效果非常好。那么，什么是扩散模型？扩散模型会接收一些随机噪声，通过迭代计算，将噪声转化为高维、定义清晰且特征非常精细的内容。以生成图像为例，你需要清晰的边缘、锐利的拐角和鲜明的色彩对比。而对于机器人来说，尽管你生成的不是图像，而是动作，但这些动作也必须非常精确、非常平滑，对吧？所以，它与图像生成有着类似的考量和要求。这就是为什么从第一性原理出发，整个领域——不只是我们——都在使用这种扩散 Transformer 架构来训练机器人基础模型。你需要 Transformer 这一部分，因为 Transformer 能够接收最多的数据并从中学习。而扩散模型则能帮助你不断优化结果，使其达到机器人实际执行任务所需的平滑、精确、细粒度的运动状态。这就是许多近期突破背后的架构。Jason Ma [00:04:20]: But what’s really interesting is that in visual domains, right, if you think about generating images, generating videos, and in this case, robots looking at images to generate actions, people have found the diffusion kind of architecture to work really well. So what is a diffusion model? A diffusion model is a model that takes some random noise and through iterative computation can go from noise to something that’s really high-dimensional, that’s something really highly defined and have very fine-grained feature. If you think about generating images, right, you want sharp edges, sharp corners, color contrast. And for robotics, even though you’re not generating image, you’re generating actions that needs to be very precise, very smooth, right? So it has kind of the same kind of consideration and desiderata as generating image. So that’s why from first principles, the whole field, not just us, have been using this kind of diffusion transformer architecture to train robotics foundation models. You need the transformer aspect because transformer is the thing that can just ingest the most amount of data, can learn. Then diffusion helps you to refine the outcome to get to the smooth, precise, fine-grained motion you need to actually execute on the robot. So that’s the architecture that goes behind a lot of our recent breakthroughs.

Swyx [00:05:37]：我们之前在播客里也聊过流匹配。它主要是提升效率、加快速度。比如，通常你需要进行四十到一百步扩散，现在可以缩减到四步。Swyx [00:05:37]: And we’ve covered flow matching before on the podcast. It is mostly an efficiency speed-up. Like, let’s say you normally would take forty to a hundred diffusion steps. You can reduce it to four.

Jason Ma [00:05:48]：是的，没错。扩散模型在实际应用中有一个很难处理的问题：你需要调整调度器，也就是调整初始噪声输入转换为最终输出的幅度。因此，流匹配有点像扩散模型的一种特殊情况，它让这一转换变得简单得多。实际使用时，你不需要做那么多调整，效果也一样好。这就是为什么我认为，大约两三年前，图像和视频生成建模大多从扩散模型转向了流匹配模型。不过，两者的概念是一样的，流匹配只是扩散模型的一种特殊变体，更容易调节。对。Jason Ma [00:05:48]: Yeah, that’s right. So with the diffusion model, one thing that’s very hard in practice is that you need to tune the scheduler, like by how much you’re tuning or converting the initial noise input into the final output. So flow matching is kind of like a special case of diffusion that makes the transformation much simpler. In practice, you don’t have to tune as much, and it works just as well. That is why I think about two to three years ago, most generative modeling for image and video has shifted from a diffusion model to a flow matching model. But the concept is the same, and flow matching is just a special, simpler to tune, variant of diffusion. Yeah.

Swyx [00:06:26]：这确实让我想到，流匹配在质量方面显然存在权衡。但实际上，它非常适合用于动作扩散和机器人技术，因为你可以实时重新评估。而在图像生成中，你只能——基本上就是一次性生成。Swyx [00:06:26]: It actually makes me think that. So flow matching obviously has trade-offs in quality. But actually, it’s very suited for action diffusion and robotics because you get to reevaluate in real time. Whereas in images, you are— it’s basically one shot.

Jason Ma [00:06:40]: That’s right.

预训练通用型机器人并适应新任务Pretraining a Generalist Robot and Adapting to New Tasks

Vibhu [00:06:41]：挺有意思的。我们可以摆弄一下叠衣服机器人……随便丢一条毛巾进去。模型会适应，对吧？我觉得你们模型的一个关键特点是，它们是——它们被设计成通才，所以只用一个基础模型，然后再对它们进行大量适配。这是怎么实现的？基本上，有了 45 分钟的映射数据，你就能朝着……进行泛化。我们看过一个从冰箱里拿饮料的例子。一个通用模型的思路是什么？然后你们又是如何针对不同任务进行适配的？Vibhu [00:06:41]: Which is pretty fun. We got to mess with the laundry folding robot … throw a random towel in. The models adapt, right? I think one key thing about your models is that they’re tr— they’re made to be generalists, so one foundation model and you adapt them a lot. How does that work? Basically, in 45 minutes of mapped data, you can generalize towards. We saw an example of pulling drinks out of a fridge. What’s the what’s the mindset there of one general model, and then how do you adapt to different tasks?

Jason Ma [00:07:14]：我们的思路是，对于机器人来说，我们需要构建一个非常通用、能力很强的基础模型，对吧？因为如果你想想人类如何在物理世界中互动和行动，就会发现我们拥有非常扎实的基础知识，可以轻松完成很多任务。但如果你想成为某项特定任务的专家，就需要经过一定的训练。我认为，要训练出像我们正在这里构建的这种通用机器人，也必须遵循同样的原则。 所以，我们的做法是先训练一个通用模型，尽可能使用我们能找到的海量数据进行训练。这就是为什么在 Dyna-2 报告中，我们的模型是以一百万小时的人类数据作为起点进行训练的。有了这个坚实的基础后，我们发现，只需进行极少量的微调，就能让机器人——或者说模型——快速学会执行许多不同的任务。我们只要告诉模型：“这是我希望它完成的任务，这里有少量示范”，它就能轻松完成你在办公室里看到的许多任务。是的。Jason Ma [00:07:14]: So the way we think about it is that for robotics, we need to build a very general, strong base model, right? And then— Because if you think about how a human interacts and behaves in the physical world, right, we have very strong base knowledge. We can just readily do a lot of tasks. But if you want to become an expert at any given task, there is some amount of training that needs to happen. And I think to train a general-purpose robot like the one we’re building here, the same principle has to apply. So our approach is to first train a general model that’s training on as much data as we can find. So that is why in our Dyna-2 report, our model is trained over a million hours of human data as a starting point. And then given that strong foundation, what we found is we can quickly make the robot or make the model do many different tasks with a very minimal amount of fine-tuning to just tell the model, “Here’s the task I want it to do, and here’s very few amount of demonstrations,” and the model can readily do bunch of the tasks that you have seen in office. Yeah.

百万小时的数据：第一视角视频与机器人演示A Million Hours of Data: Egocentric Video and Robot Demonstrations

Swyx [00:08:19]：实际上，我们这里有不少人从事数据交易，也有人出售数据。什么是好数据？什么是坏数据？你想买什么？我认为这些问题都还没有定论。显然，我认为最重要的是人类执行任务时的第一人称视角数据。对于这个基本观点，你还会做哪些补充？Swyx [00:08:19]: We actually have quite a few people who deal in data, who sell data. What is good data? What is bad data? What do you want to buy? I think that these are all open questions. Like, obviously, I think the top, important thing is egocentric data of humans doing tasks. What else would you modify on that basic statement?

Jason Ma [00:08:42]：所以我认为，数据是当今机器人领域最重要的东西，因为和语言模型不同，机器人领域没有互联网，对吧？因此，每家开发模型、制造硬件的公司都必须考虑数据。大致来说，这里有几种不同的做法。一种是自己收集大量数据，我们也在这么做。Jason Ma [00:08:42]: So I think data is the most important thing in robotics today, because unlike language model, there is no internet, right? So every company that’s building model, building hardware has to think about data. So there’s, roughly couple of different ways of operating here. One is that you can collect a lot of data yourself, which is what we are doing as well.

Swyx [00:09:02]：你们这里有员工在做这件事。Swyx [00:09:02]: You have employees here doing that.

Jason Ma [00:09:04]：办公室里到处都是机器人，对吧？但如果想想机器人基础模型需要多少数据，我认为任何一家公司单独能收集到的数据量都远远不够，对吧？这有点像是你只靠 Reddit 或者只靠 Facebook 就想建成整个互联网。它们都是很棒的网站，用户也很多，但这还不够，对吧？所以我们在考虑数据时，必然要通过某种方式获取数据，对吧？比如购买数据，或者查看已有的数据。而且数据有很多不同的层级，对吧？你提到了数据质量。我认为，机器人领域的数据质量是个非平稳问题，因为对你的模型来说，什么样的数据算高质量，会随着时间而变化。比如，当你的模型什么都不懂、还处于零起点时，任何数据都有用，对吧？你可以使用大量视频数据。就第一人称视角数据而言，我认为它确实是预训练机器人的一种非常好的方式，因为人类数据极其多样。它涵盖了你能想象到的每一种身体活动。这能让机器人模型对具身行动者（也就是人类）周围的事物如何变化、世界如何演变，形成非常强的先验认知。但人类数据的问题在于，除非你的机器人看起来和人类一模一样，否则人类的动作和机器人的动作之间总会存在差距，对吧？所以，仅靠人类数据训练，如今还不太可能让机器人直接执行动作。因此，你仍然需要用机器人自己的数据进行一定程度的适配，对吧？但在机器人上采集机器人自身的数据会贵得多，也难得多。所以我们考虑的方案是：如何利用当下最大的数据来源——比如第一人称视角数据——让模型对物理操作所涉及的各种情形形成强大的先验，再结合少量机器人数据，让机器人掌握我们感兴趣的任务。因此，我认为，机器人领域的数据质量和数据获取，确实是个引人入胜、涉及多方面的问题。但现实是，并不存在一种必胜策略，比如说：“哦，就这一类数据，我只采集这一种。”Jason Ma [00:09:04]: There are robots everywhere in the office, right? But if you think about how much data you need for robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient, right? It’s kind of like, you’re trying to build an internet by just having Reddit or just having Facebook. They’re great websites, a lot of users, but it’s not enough, right? So when we think about data, there has to be some sort of data acquisition happening, right? You buy data, you look at data that’s already out there. And then there’s many different tiers, right? You mentioned data quality. I think data quality in robotics is a non-stationary problem in the sense that what is high-quality data for your model changes over time. Like, when your model doesn’t know anything, like when it’s zero, any data is useful, right? You can use a lot of video data. In the case of egocentric data, I think it’s actually a very good way of pre-training robots because human data is so diverse. Like, it covers every single physical activity you can possibly imagine. And that gives robot, the model such a strong prior on how things, how the world can evolve around a embodied actor, human. But the issue with human data is that until your robot looks exactly like a human, there’s gonna be some gap between human actions and the robot actions, right? So just by training on human data, today, it’s not very likely you can just execute robot actions directly. So you still need some amount of adaptation using the robot’s own data, right? But the robot’s own data collected on a robot is gonna be much more expensive and much harder to collect. So the kind of recipe we’re thinking about is like, how do we take the largest source of data we have today, like egocentric data, to give the model a strong prior over everything physical manipulation can cover, and then combine that with a small amount of robotics data to make the robot master whatever task we’re interested in. So data quality and getting data in robotics is really, I think it’s a fascinating, multifaceted problem. But the reality is that there’s not a single winning strategy like, “Oh, here’s one kind of data. I’m only gonna collect on that,”

Jason Ma [00:11:25]：这样就能为你打下坚实的基础模型。我认为你确实需要考虑：“这些人类数据涵盖了机器人认知中我所需要的这部分内容；机器人数据则涵盖了机器人也需要的低层级反射和灵巧操作能力”，然后把它们融入一个模型中。我认为，这最终才能造就一个适应性很强、性能也非常出色的模型。是的。Jason Ma [00:11:25]: Which will give you a strong foundation model. You really need to think about, “This human data covers this part of robot cognition I need. Robot data covers low-level reflex dexterity the robot also needs,” and put them into one model. And I think that’s what gives a really adaptable, really strong model at the end. Yeah.

数据合作伙伴关系与专项训练示例Data Partnerships and Specialized Training Examples

Swyx [00:11:44]：我再提示你一下，这是说给那些想和你合作的听众听的。你——你想要哪些数据供应商？这是你征集数据的呼吁。什——Swyx [00:11:44]: I’ll prompt you, just for again, for the people listening who want to work with you. What, d— what data vendors do you want. This is your call for data. What-

Jason Ma [00:11:55]：是的，没错，这是个很好的问题。我们在 Dyna-2 博客文章的末尾留了一个邮箱地址：[email protected]。如果你是数据供应商，手头有大量高质量数据——比如第一人称视角数据、机器人数据、视频数据，或者任何你认为能帮助机器人的数据——请发送到这个邮箱。我们的团队会查看，并给你回复。Jason Ma [00:11:55]: Yeah, no, that’s a great question. And yeah, so in our Dyna-2 blog, at the end, we actually have an email address called [email protected]. So if you’re a data vendor that has a lot of high-quality, let it be egocentric data, robot data, video data, or anything you think could help robots, send it to that email. Our team will take a look, and we’ll get back to you.

Swyx [00:12:16]：不过以你现在的阶段，你应该不会再买现成的第一视角数据了，对吧？你大概会想说：“这是我需要的。你去——”Swyx [00:12:16]: But at your stage, are you. You’re probably not buying any more egocentric data off the shelf, right? That you probably want to be like, “Here’s what I need. You go-”

Vibhu [00:12:26]：有没有哪个细分领域还存在空缺？Vibhu [00:12:26]: Is there a niche that’s lacking?

Swyx [00:12:28]：你需要 Scale AI 这样的公司，需要 Mercor 这样的公司来做这件事。我明白了。定制数据——Swyx [00:12:28]: You need a Scale AI, you need a Mercor for doing this. I see. The custom data-

Vibhu [00:12:32]：那些不从事机器人行业的人可能会想：“好吧，就是某个小众工厂。”比如……你展示了把管子放进去的操作。关于这种操作，你能找到的数据也就那么多，但这些数据能帮上你很多吗？Vibhu [00:12:32]: The folks that don’t work in robotics are probably thinking like, “Okay, some niche factory.” Like … you showed putting tubes in. There’s only so much data you’ll find about that but like would a lot of that help you?

Jason Ma [00:12:43]：是的，没错。如果你真的想把一项任务做好，拥有这项任务的数据肯定会对你大有帮助，对吧？所以对我们来说，我们仍然认为，仅仅一百万小时的数据还不够，这只是个开始。比如，我们展示了首个百万小时规模的数据模型，但我们需要的不止这些，对吧？所以我们仍在寻找大量我们能获取的原始、多样化数据。但我们也在思考，如何获得更多定制数据——这些数据可以由我们内部采集，也可以由合作伙伴采集，用于真正特定的细分任务？因为我觉得，要让任何模型在物理世界中真正发挥作用、达到所需的出色水平，你仍然需要这些数据，对吧？这有点像人类，你知道。如果肖恩或我明天去 Rosewood Hotel 工作，我们肯定没法按照他们的要求铺床，对吧？我们需要大量培训。所以这类数据，也就是特殊数据，我认为就像 Mercor 今天为语言模型所做的那样，对机器人领域也会非常有价值。Jason Ma [00:12:43]: Yeah, no. If you really want to nail a task, having data from that task will certainly help you a lot, right? So for us, it’s like we still think, just a million hours of data is not enough. It’s just the beginning. Like, we showed the first model on a million-hour scale, but we need more than that right? So we are still looking at a lot of just raw, diverse data, that we can get. But we’re also thinking about how do we get more custom data either collected in-house or by our partner for really, specific niche tasks? Because I think you still need that to get to excellence that any model needs to actually be useful in the physical world, right? It’s kind of like humans, you know. If, Sean or me, we go work at Rosewood Hotel tomorrow, we are not gonna be able to make the bed the way they want, right? We need a lot of training. So that kind of data, like special data, that’s kind of like what Mercor is doing today for language model, I think will be very valuable for robotics as well.

所以，酒店行业，把 GoPro 绑在员工头上，然后……Vibhu [00:13:42]: So hotels, strap a GoPro on your employees’ heads and.

然后可能还有更多类似的内容。Jason Ma [00:13:48]: And then there’s probably more of that.

Vibhu [00:13:49]: There’s the

合成数据、仿真与 Eureka 项目Synthetic Data, Simulation, and the Eureka Project

Swyx [00:13:50]：让他们自己挡自己的路。是啊。我们之前讨论过怎么结束这次采访，有件事特别有意思：现在你基本上可以即时生成各种环境。你用前沿模型试过吗？体验怎么样？它们够好用吗？Swyx [00:13:50]: Put themselves in their way. Yeah. So one thing that we discussed is the closing this interview, and what’s very interesting is now you can basically fabricate environments on the fly. Have you tried that with the frontier models? What’s your experience? Are they good enough?

你们有没有在使用任何世界模型机器人技术？比如说，利用世界模型公司来生成这些数据，并基于这些数据进行训练？Vibhu [00:14:06]: Are you using any world model robotic. Like, world model companies to generate these and train on them?

Jason Ma [00:14:13]：所以我觉得，生成合成环境、合成数据，以及用于机器人技术的仿真数据，将会成为下一个前沿领域之一，对吧？因为如今人们已经在研究第一人称视角数据和机器人数据。这些领域都在不断扩展，但我认为合成数据将成为扩展规模的一个新维度。回到你的问题，是的，我们一直在探索如何利用前沿模型，根据我们的数据和我们关注的环境，为我们生成合成环境。令人惊讶的是，我觉得现在这项技术已经发展到了一个阶段，确实非常值得深入研究。简单介绍一下背景：我读博时主要研究基于仿真的机器人训练。那时候仿真还不够成熟，但我们发表过一篇名为 Eureka 的论文，它是首篇利用大型语言模型（当时用的是 GPT-4）自动生成环境、为强化学习生成奖励函数，并在仿真中完成各种任务的论文。最后，我觉得我们做了一个非常棒的演示：让机器人旋转铅笔，对吧？当时那真的很酷。但那时候这类方法仍然非常有限，因为当时的模型视觉能力不够好，也没有内在的三维理解能力，对吧？所以它基本上只能生成代码，再利用非常有限的文本反馈，尝试创造新的仿真环境。但如今，这些前沿模型的视觉能力简直不可思议。它们可以完成从真实到仿真的转换、CAD 建模、三维重建，以及所有这些任务。所以我觉得现在已经到了一个转折点，……Jason Ma [00:14:13]: So, I think generating synthetic environments, synthetic data, simulation data for robotics, it is going to be one of the next frontiers, right? ‘Cause today people have looked at egocentric data, robot data. Those are scaling, but I think synthetic data will be a new dimension of scaling. So going back to your question yeah, we have been exploring using frontier models to generate synthetic, environments for us on our data, on our environments that we care about. And surprisingly, I think now it’s at a point where this technology is actually very much worth going really deep into. So for some context, so, in my PhD, I primarily focused on simulation-based training for robotics. Back then, simulation wasn’t ready, but we had this paper called Eureka, which was the first paper taking a large language model. At the time, it’s GPT-4, which is how to self-generate environment, generate reward functions or reinforcement learning, and just do things in simulation. And at the end, I think, we had this really nice demo of the robot spinning pencils, right? That was really cool at the time. But that kind of thing was still very limited at the time, because back then the model didn’t really have good vision capability, didn’t have a innate 3D understanding, right? So it was basically just generating code and using very limited feedback from text to try to come up with new simulation environments. But today the vision capability of these frontier models are crazy. They can do real to sim, do CADing, there’s 3D reconstruction and all of that. So I think now it’s at a tipping point where

Jason Ma [00:15:52]：这项技术确实能加速机器人技术的发展。对我们来说，我们会把它看作加速内部进展的一种方式，对吧？因为我们既在构建模型，也在制造硬件。所以，如果前沿模型能以这种方式为我们提供更多数据，比如生成真正为我们量身定制的数据，那对我们来说就非常有用。我们还在考虑另一件事：除了用这些模型生成 3D 环境，我们也用过自己的世界模型，也就是我刚才跟你说的世界动作模型。因为这个模型不仅能生成动作，还能生成视频。我们——这个模型还可以生成自己的训练数据。这就形成了一个自我强化的飞轮，我觉得它也会很强大。总的来说，我对这个方向真的非常兴奋，因为尽管我们已经把数据规模扩展到一百万小时，也部署了一大批机器人，机器人正——已经开始做一些非常酷的事情，但我们才刚刚起步，只看到了未来前景的冰山一角。是的。Jason Ma [00:15:52]: This technology could really accelerate robotics. And for us, right, we think about that as a way to accelerate our internal progress, right? Because we’re building models, we’re building hardware. So if a frontier model can give us more data that way, like generate really tailored data for ourselves, that’s extremely useful for us. And another thing we’ve been thinking about is instead of using these models to generate 3D environments, we have also used our own world model, right, the world action model I was telling you about, because the model not only generates action, it can also generate videos. We— The model can also generate its own training data. So that becomes a self-fulfilling flywheel that I think will also be powerful. So overall, I’m just really excited about this direction because I think even though we have scaled to a million hours of data, we are deploying a bunch of robots. The robots are act— already starting to do very cool things. We’re just at the beginning, just the tip of iceberg for what is there to come. Yeah.

设备端推理、延迟与模型蒸馏On-Device Inference, Latency, and Model Distillation

Vibhu [00:16:51]：对于做研究的人来说，我们最近请 Runway 上了播客。他们正在做的一件事是机器人世界模型。他们正稍微朝这个方向转，所以……想引导大家关注一下。我想问问你们自己开发模型时受到哪些限制？如果拿语言模型来类比，你会发现 GPT-OSS 20B 非常适合在普通 MacBook 上运行。然后还有一个更大的模型，适合在单块 GPU 上运行。你们是在机器人端侧模型、云端模型，还是混合模式这些限制下工作？这又会如何影响你们的数据缩放定律？一百万小时的数据很棒，但我们是不是只要扩大参数规模和数据量就行了？Vibhu [00:16:51]: For those on the research side, we recently had Runway on the podcast. One of their things is robotic world models. They’re, shifting a little towards there, so … nudge people to look at it. One of the questions I have about you making your own models is what is the constraints you operate in? So if you draw a parallel to language models, you see GPT-OSS 20B is very fit for your average MacBook. Then there’s a larger one that’s fit for a single GPU. Do you work in the constraints of robot, of model on on device, in cloud, a hybrid? And then how does that affect your scaling law of data? A million hours is great, but like do we just scale up parameters, data?

Jason Ma [00:17:38]：不，这是个很好的问题。我觉得机器人和语言模型及其他应用真正不同的地方在于，对于编程智能体来说，语言模型有延迟也没关系，对吧？比如现在我每次问 ChatGPT，它都要过几秒钟才会输出内容。但对机器人来说，推理延迟必须非常短，对吧？机器人得行动起来。比如，它不能拿着一把刀，三秒钟什么也不做。那会把人吓坏的，对吧？Jason Ma [00:17:38]: No, that’s a great question. So I think what’s really different about robots versus language model and other applications is that for coding agents, language model, it’s okay if it’s— it has latency, right? Well today, whenever I ask ChatGPT, it takes like couple of seconds before it spits out anything. But for robots the inference latency has to be very short, right? The robot needs to act. Like, it can’t just hold a knife and not do anything for three seconds. That’s gonna freak people out, right?

Vibhu [00:18:07]：是啊，我们看到有人把自己的手真的放到了刀底下。Vibhu [00:18:07]: Yeah, we saw someone put their real hand under a knife.

Jason Ma [00:18:10]：我们的模型挺聪明的，所以没问题。但总的来说，如果你想想那些非常动态的任务，比如说，你希望机器人拿着——我不知道——一支网球拍颠球，对吧？这类任务变化非常快，需要实时反应。你不能只是……毕竟物理规律不会等你，对吧？所以，机器人模型必须能够非常快速地进行推理。做到这一点最快的方法，就是在本地部署，比如直接在设备上运行，对吧？因为如果你调用一个模型——Jason Ma [00:18:10]: Our model’s, pretty smart, so that’s okay. But in general if you think about very dynamic tasks, like for example, you want your robot to hold a I don’t know, a tennis racket and juggle the ball, right? It’s just very dynamic, real reactive. You can’t just like. The physics, gravity doesn’t wait for you, right? So for that reason, robotics model needs to be able to inference very fast. And the fastest way to do that is by serving on-prem, like on device, right? Because if you call a model-

Swyx [00:18:39]: Network latency.

Jason Ma [00:18:40]: There’s no guarantee-

Vibhu [00:18:41]: Self-drive a car.

Jason Ma [00:18:42]：这并没有保证，对吧？所以，最终部署的模型必须大小适中，能够在本地运行。但这并不意味着你最初训练的模型也必须很小，对吧？比如说，在语言模型——Jason Ma [00:18:42]: There’s no guarantee, right? Like, so for that reason, the model that gets deployed eventually has to be of reasonable size that can be served locally. But that doesn’t mean that the initial model you train also has to be small, right? Like in language model-

Vibhu [00:18:57]: You can distill.

Jason Ma [00:18:58]：人们会做蒸馏，对吧？比如说，要让一个小模型变得很聪明，最好的办法就是做蒸馏，对吧？先训练最大的模型，然后再蒸馏，对吧？所以我觉得，机器人领域也正在朝这个方向发展，我想这最终会成为大家的共识。对。Jason Ma [00:18:58]: People do distillation, right? Like, the best way to make a small model very smart is by doing distillation, right? You train the biggest model, then you distill, right? So I think in robotics, we’re getting there, but I think that’s what people are going to converge on. Yeah.

Vibhu [00:19:14]：推理方面呢？你们有自己的推理技术栈，还是……很多工作都在架构上。你们引入了扩散模型之类的东西。我猜你们可以借助某些现成的东西。有什么想分享的吗？Vibhu [00:19:14]: How about inference? You have your own inference stack or. A lot of this is architecture. You’ve pulled diffusion and whatnot. I’m guessing you can pull off of something. Anything you wanna share?

Jason Ma [00:19:26]：所以，无论我们部署什么模型，都会构建自己的推理技术栈，因为——就像我刚才说的，延迟非常重要。因此，掌控整套技术栈对机器人来说极其重要，对吧？你需要针对正在运行的模型类型优化推理速度，而这需要投入大量工作。这就是我们所做的。Jason Ma [00:19:26]: So we build our own inference stack for whatever model we deploy, because that’s— like I was saying latency is so important. So owning the whole stack is extremely important for robotics, right? You want to optimize the inference speed for the kind of model you’re running, and there’s a lot of work in there. So that’s what we do.

构建完整的机器人技术栈：传感器与执行器Building the Full Robotics Stack: Sensors and Actuators

你真的是全栈，涉及从执行器一直到工业设计（ID）——我之前还真不知道 ID 是个缩写。当然，也包括模型和数据。Swyx [00:19:45]: You’re really full stack, like- Everything from the actuators all the way to in the industrial design ID, which I- Didn’t really know as a acronym. Obviously to the models and data as well.

Vibhu [00:19:57]：我很好奇，这些执行器是你们自己做的吗？还是……这里面有多少是你们自己做的？比如，当然，设计方面，不是说人形机器人——执行器之类的，你们在机器人技术栈上深入到了什么程度？Vibhu [00:19:57]: I’m curious are these your own actuators, or are they. How much of this is your own? Like, of course, the design, it’s not like humanoid- Actuators, like how far down the robotics stack?

Jason Ma [00:20:08]：所以，我们看待哪些东西需要自己掌握、哪些东西可以买现成的，方式是这样的：如果某个组件的输入和输出非常明确，而且已经有非常成熟、优秀的解决方案，那就没必要自己开发。比如说，在这台机器人上，你能看到激光雷达，对吧？那些黑色圆形的部件就是激光雷达，用来帮助机器人探测障碍物、了解周围环境并进行导航。这是一项非常成熟的技术，已经应用了几十年，也用在自动驾驶汽车上。所以我们没必要从头开始研发激光雷达，对吧？摄像头也是一样。但对于那些我们认为对整个系统最终性能至关重要的东西，我们的愿景是能够自己掌握并自主研发。比如你提到的执行器，这就是我们正在研发、并且打算自己掌握的东西。原因是执行器的质量和设计方式，对机器人能否得到准确建模影响很大，对吧？执行器驱动着机器人。要在仿真中很好地对机器人建模，或者进行任何形式的建模，就需要选用非常适合建模的执行器。这才是我们最终真正看重的，所以我们会自己掌握这项技术。机器人的身材比例也是一样。这台机器人的身材比例是参照真实的人来测量的。为什么要这么做？因为我们要用人类数据进行预训练。Jason Ma [00:20:08]: So the way we think about, what we need to own versus what we can buy is like if the Input-output and of a component is very clear, and like there’s already extremely good mature solution, then there’s no point of building in-house. For example, on this robot, you see the LiDARs, right? So those, black circle, those are LiDARs to help the robot detect obstacle, navigate surroundings. So that’s a very mature technology that people have been using for now decades. It’s on self-driving car. So we don’t need to build LiDAR from scratch, right? Same goes with some cameras. But for anything that we think is extremely crucial for the whole system’s performance at the end, our vision is to be able to own that and build that ourselves. So one thing is, you mentioned actuator. So that’s something we’re working on and we’re going to own ourselves. The reason is that like, the quality and how you design the actuator really matters for how well you can model the robot, right? Actuator is what drives the robot. To be able to model your robot very well in simulation or to do any kind of modeling, you need to select actuators very good that can actually be easily modelable. So that’s what we care about eventually. So we’re gonna own that. Same thing with the robot’s proportion. This robot’s proportion is we measure it against real people. Why do we do that? ‘Cause we’re pre-training on human data.

Swyx [00:21:36]: Oh, yeah.

最好的迁移方式，对吧？所以，硬件方面会涉及很多这类决策，而这些决策又和……密切相关。因为最终承载智能的是硬件。智能是极其重要的东西。我们希望确保硬件能够服务于智能。Jason Ma [00:21:36]: Best transfer, right? So there’s a lot of these kind of decision that goes into hardware that’s very much tied to like. Because eventually, it’s hardware that carries intelligence. Intelligence is the extremely important thing. We want to make sure the hardware serves for that.

Swyx [00:21:53]：我只想特别提一下，不确定这对你是否相关，但我聊过的最有意思的机器人话题之一，是有人对执行器，尤其是磁体，感到非常焦虑，因为据说美国不生产磁体。Swyx [00:21:53]: I would just highlight, I don’t know if this is relevant to you, but one of the most interesting robotics discussions I’ve had is someone who was very stressed out about actuators and specifically magnets, ‘cause apparently we don’t make magnets in the US.

制造业、稀土磁体与全球供应链Manufacturing, Rare-Earth Magnets, and Global Supply Chains

Jason Ma [00:22:06]: No, we don’t.

Swyx [00:22:07]：而且只有两个国家生产磁铁。Swyx [00:22:07]: And there’s only two countries that make magnets.

稀土材料。Jason Ma [00:22:09]: Rare earth material.

Swyx [00:22:10]：稀土，对吧？我不知道那个……那是什——是什么化学物质？是什么材料？我不知道那东西是什么，不过——所以，是的，如果有人对此感兴趣，显然这是我们遇到的一个瓶颈。Swyx [00:22:10]: Rare earth, right? I don’t know what the. Wh-what’s the chemi— what’s the material? I don’t know what the thing is, but- so yeah, if anyone’s interested in that’s, apparently a bottleneck that we have.

Jason Ma [00:22:20]: No, not bottleneck.

Swyx [00:22:21]：这是个瓶颈。那我们该怎么办？Swyx [00:22:21]: It’s a bottleneck. So what do we do?

Jason Ma [00:22:23]：这是个好问题，对吧？我觉得，关于把制造业带回美国，大家有很多令人振奋的讨论，但尤其对机器人来说，实际情况比看起来要难得多。可以说，即使是现在，美国也没有一家公司真的能从头到尾都在美国制造所有东西，对吧？比如，即使他们说自己在组装机器人、在美国制造实际负责生产的部件，他们的原材料仍然来自世界其他地方。我觉得，作为机器人制造商，确实要意识到这一点，对吧？我们作为一家全栈机器人公司，意识到供应链有多长，整个问题有多复杂。这实际上极其困难。如今，很多讨论都围绕着机器人基础模型所需的数据展开，但我觉得，硬件以及制造出优质硬件究竟有多难，这些都被严重低估了。不过，我认为这会非常重要。Jason Ma [00:22:23]: It’s a good question, right? I think, there’s a lot of exciting discussions on bringing manufacturing back to the US, but for robots especially, that’s actually much harder than what it seems. And I would say even now, there isn’t a company in the US that’s truly building everything from scratch in the US, right? Like, even when they say they assemble robot, they build whatever they’re actually doing in the US, they’re still sourcing their raw materials from other places in the world. And I think that’s something to really be aware of as a robot builder, right? Like, I think, like as a full-stack robotics company, we realized just how long the supply chain is and how long the whole problem is. It’s actually extremely hard. And I think these days, a lot of the discussion centers around a robot foundation model data, but I think hardware and just how hard it is to build good hardware is very under-emphasized. But I think that’s gonna be extremely important.

Swyx [00:23:27]：这是一个平台，对吧？Swyx [00:23:27]: This is a platform, right?

为什么 Dyna 会协同设计硬件与模型Why Dyna Co-Designs Its Hardware and Models

Jason Ma [00:23:29]: Platform, yeah.

Swyx [00:23:30]：有些公司选择成为——我们姑且称之为——无平台公司。它们只做模型，并会说：“我们可以部署在任何平台上。”我们部署在 Optimus 上。Swyx [00:23:30]: There are some companies that choose to be, let’s call it platform-less. They only do the model, and they will say, “We deploy on any platform.” We deploy on Optimus.

Vibhu [00:23:37]：现成的机器人技术。Vibhu [00:23:37]: Off-the-shelf robotics.

Swyx [00:23:38]：所以这是一种选择，对吧？显然，一切都由你掌控，所以你可以这么做。你觉得自己会更多地在美国建设，还是会打造自己的全球供应链？哪种发展方向更有可能？Swyx [00:23:38]: And so, it’s a choice, right? Like, obviously, you own everything, so you can do that. Do you see yourself building more in the US, or do you see building your own global supply chain? What’s the direction that is likely?

Jason Ma [00:23:52]：所以我们目前正在探索许多不同的选项，对吧？因为全球供应链会受到一些地缘政治因素的影响，对吧？Jason Ma [00:23:52]: So we are currently exploring many different options, right? Because with global supply chain, then there is some, geopolitical things, right?

Swyx [00:24:02]：发布。Swyx [00:24:02]: Shipping.

Jason Ma [00:24:02]：发货。还有发货延迟之类的情况。Jason Ma [00:24:02]: Shipping. Also shipping delays and all that stuff.

Vibhu [00:24:04]：给机器人用的真船。Vibhu [00:24:04]: Literal ships for robots.

Jason Ma [00:24:06]：在美国，从零到一需要一些时间，对吧？所以我们在思考一种解决方案，既能让我们在美国扩大建设，又能利用全球供应链，对吧？不过回到你刚才关于有些公司选择只做模型、而我们什么都做的问题，我认为把这件事做好非常重要。我觉得，机器人领域难的地方在于……比如说，构建整个系统。要构建一套完整的系统，需要哪些东西？之前我在 DeepMind 和 NVIDIA 工作，主要就是构建模型；我也看到了当你无法掌控硬件、无法对其进行优化时，系统的上限在哪里，对吧？因为如果你想想机器人创造价值的方式，最终机器人是来提供服务、解决某些物理任务的。所以，如果你想解决物理任务，就要能够优化一切，这一点很重要。Jason Ma [00:24:06]: And in the US, it’s just that going from zero to one will take some time, right? So we’re trying to think about a solution that can both allow us to build more in the US while leveraging global supply chain, right? But I do think, going back to your previous question on some companies choose to just build model, we build everything, I think it’s very important to get this right. I feel that for robotics, the hard thing is. Like building the whole system. What are the necessary things to build a whole system? Like, previously, I was at DeepMind, I was at NVIDIA, primarily just building models, and I saw the upper bound of the system when you don’t control the hardware, when you don’t get to optimize that right? Because if you think about the value creation for robots, eventually robots are here to deliver a service to solve some physical tasks. So if you wanna solve physical tasks, you wanna be able to optimize everything, and that’s important.

半人形机器人：轮式移动、躯干运动与灵巧操作能力The Semi-Humanoid Robot: Wheels, Torso Movement, and Dexterity

Vibhu [00:25:02]：我们有这个机器人。你得跟我们讲讲它的设计。你们自己定制的平台。这叫什么？Vibhu [00:25:02]: We have the robot. You gotta tell us about the design. Your own custom platform. What do you call this?

我们把它叫什么？Jason Ma [00:25:11]: What do we call it?

Vibhu [00:25:12]：是的，所以它不是人形的。它非常流畅，不过那个旋转臂是什么？Vibhu [00:25:12]: Yeah, so it’s not humanoid. It’s very smooth, but what’s the what’s the rotating arm?

Jason Ma [00:25:16]：对，我们一般把它看作一种半人形机器人。基本上，它的上半身就像人形机器人，有肩膀，当然也有手臂。腰部还有四个自由度。所以，基本上人类能做的所有上半身动作，它都能做。但它是半人形机器人，因为下半身是轮子。好，那我们为什么选择轮子呢？因为我认为这里存在一种差异：实用性和性能之间的最佳权衡。我觉得如果用腿，能耗——能耗会高得多，也会不稳定得多。安全性会极难保障。所以目前来说，对于足式机器人，其实无法保证它能始终保持直立和平衡。我相信你们都看过很多这类机器人跳舞的视频。就是机器人跳舞的那种。Jason Ma [00:25:16]: Yeah, so we think of this as generally like a semi-humanoid. So basically, the upper body is, like a humanoid. It’s got, of course, shoulder, got arms. It also got four degrees of freedom on the waist. So you can basically do all the upper body motion that a human can do. But it’s a semi-humanoid because the lower body is wheels. All right, so why do we choose wheels? Because I think there is a difference. There’s, optimal practicality versus performance trade-off. I think with legs, the energy con— the energy consumption would be a lot higher. It would be a lot more unstable. Safety will be extremely hard to guarantee. So right now, for legged robots, there’s actually no guarantee that the robot will stay upright, balanced the entire time. I’m sure you guys have seen a lot of those, robot dancing videos. Like the robot dancing thing.

看到它们跳舞了，是的。它们都很容易摔倒。Swyx [00:26:08]: Seeing dancing, yeah. They all, they fall over very easily.

Vibhu [00:26:10]：不是，不过你也能看到那些一看就不是量产产品的演示，比如你可以跟机器人打拳，也可以把它推倒，而它表现得很好。但那个机器人只是平衡能力好，对吧？就像——有一个……Vibhu [00:26:10]: No, but you also see the demos that you can tell are not production, where you can box the robot and you can push it over and it’s very good. But that robot is just good at balance, right? Like- there’s a.

Swyx [00:26:21]：我觉得你们创新出来的东西更有意思。我们有一个原型，就是后面这个“超梦”。Swyx [00:26:21]: I think the more interesting thing that you guys have innovated. So we have, one of the prototypes, which is the Mewtwo back here.

你可以这么称呼它。Jason Ma [00:26:29]: You can call it that.

Swyx [00:26:30]：这就像是一个上下轴，而你选择朝着这种可以倾斜的东西发展。我们还会展示洗衣演示，在那里你必须倾身靠近洗衣机，对吧？那又是一个自由度，我甚至不知道该怎么称呼它。它不是俯仰，也不是偏航，而是别的什么。Swyx [00:26:30]: This is just like an up-down, axis, and you’ve chosen to then move towards something like this which, has the ability to lean. And we’re gonna show the laundry demo where you have to lean into machines. Right? That’s another degree of freedom that I don’t even have the words for it. It’s not pitch, it’s not yaw. It’s something else.

Jason Ma [00:26:47]：所以，是的，我们是怎么考虑这个问题的呢？实际上，这种设计对锻炼很有用。我觉得几年前我锻炼时，有时会不小心拉伤肌肉。对吧？我相信你们也有过这种经历。接下来几天你就动不了了；或者睡姿不对，脖子就会僵得厉害。我们都有过这种体验。每当遇到这种情况，你就会意识到，哇，我们日常生活中会用到那么多肌肉、那么多自由度，而我们早已习以为常。但身体某个部位无法活动时，做一些原本觉得很自然的事情就会变得非常别扭，对吧？我们设计机器人时也是这么想的。如果机器人采用这种设计——纯粹的升降结构，只能上下移动——我们就会发现，人体躯干的许多动作，比如日常活动中经常用到的腰部和背部动作，都无法实现。对吧？就算我想拿这个瓶子，也不只是动动手臂而已。如果还得刻意想着固定上半身，那就会非常奇怪。我总是会这样动，对吧？人类有很多下意识的身体动作，我认为如果机器人也有相应的自由度，同样会很有用。再进一步想想，我们希望从人类数据中学习。人类总是在活动躯干，你会希望把这些数据输入模型，用来控制机器人。但如果机器人的形态设计不支持这些动作，迁移效果就会打折扣，机器人也会变得不那么灵巧，动作看起来也会更别扭。Jason Ma [00:26:47]: So yeah, how did we think about that right? Like,. So actually, it’s very used for exercise. Like, I think sometimes, a couple years ago when I was exercising, I accidentally like, pulled a muscle. Right? I’m sure you guys have that experience too, right? Then the next few days you just couldn’t move, or like you pull it where you slept in the wrong position, so your neck is very stiff. We have all experienced that. And then whenever you experience that you kind of realize, wow, so many muscles, so many degrees of freedom that we use in our daily life that we take for granted. But when it’s disabled it’s very awkward for you to do some tasks that you feel are very natural, right? So that’s also how we think about when we design robots. When you have a— this kind of design, which is this purely elevator, could only go up and down, we realize a lot of the human torso movement, like we use our waist, back a lot for day-to-day physical activity, right? Even if I wanna get this bottle I’m not just moving my arms. It’s actually very strange for me if I have to intentionally think about fixing my upper body. I would always do this right? So there’s just a lot of subconscious way that humans move their body that I think will be equally useful if robot has a degree of freedom. And if you further think about, okay, we want to learn from human data. Like, humans are moving their torso all the time. You wanna be able to ingest that into the model to control the robot. But if you have a form factor that doesn’t allow for that then the transfer becomes less and the robot becomes less dexterous and it’s just more awkward.

将人类预训练迁移到新型机器人硬件上Transferring Human Pretraining to New Robot Hardware

Vibhu [00:28:22]：这方面的数据标注是怎样的？比如说，我——你见过那些人。这里指的是后训练微调的最后阶段，也就是用遥操作或远程控制机器人来映射动作的那一小时吗？还是说其中有些东西可以泛化？Vibhu [00:28:22]: What’s the data labeling side of this? Like, I— you’ve seen people,. Is this the last post-training fine-tuning side of actually the hour of using the robot tele-operated or remote-controlled mapping motion? Or does some of it generalize over?

Jason Ma [00:28:41]：你说的标注是指什么？Jason Ma [00:28:41]: When you say labeling, you mean?

比如，把你用于预测状态的模型映射到特定硬件上。Vibhu [00:28:43]: Like, taking the model you have that predicts state to mapping it to your specific hardware.

Jason Ma [00:28:50]：哦，是的。它的泛化能力非常强。我给你们展示的所有新机器人演示都是如此，因为这台新机器人是我们不久前才准备好的。所以，跟我们给你们看过的固定机械臂相比，我们手头的数据确实少得多，而那些机械臂的数据要多得多，对吧？不过，对于这台新机器人，模型在预训练阶段（至少在我们的一些预训练模型中）并没有见过这些任务。它却能非常快地学会你们看到的任务。我认为这很大程度上要归功于：甚至在硬件准备好之前，我们就已经决定用人类数据进行预训练，因为我们知道，某个时候我们的机器人会变得像人一样。硬件准备期间，训练方面可以先行一步。我觉得现在我们看到，这种迁移确实存在，而且确实很有帮助。Jason Ma [00:28:50]: Oh, yeah. It generalizes very well. So for all the demos that I have shown you on the new robot, ‘cause the new robot, we, it got ready not so long ago. So we quite literally do not have a lot of data compared to the stationary arms we showed you, which had much more data, right? But a new robot, the model has not seen this in pre-training, in some of our pre-trained model. It’s able to learn the task that you saw very quickly. And I credit a lot of that to like, even before the hardware is ready, we already committed to pre-training on humans because we knew at some point our robots will be like human. You can get ahead on the training side while hardware is getting ready. And I think now we’re seeing that the transfer is actually there and is actually very helpful.

评估新技能并从真实世界的反馈中学习Evaluating New Skills and Learning from Real-World Feedback

Vibhu [00:29:35]：聊聊这件事的评估环节。你可以让它做出预测、执行各种操作，但我们时不时会发现它漏掉一些东西。你们也会针对非常小众的场景进行测试，比如装药瓶。这个过程需要多长时间？你们如何看待将它部署到新任务上、对它进行评估并不断迭代这一环节？具体是怎样的？Vibhu [00:29:35]: Talk about the eval side of this. So you can have predictions, you can have it do stuff, and every now and then we see it miss stuff. You test on very niche things as well. It’s like filling a pill bottle. How long does that process take? How do you think about the aspect of actually deploying on a new task, evaling it, iterating? What does that look like?

Jason Ma [00:29:56]：是的，所以如果我们想学习一项新任务，基本上会采集一些机器人数据。然后模型，这些——Jason Ma [00:29:56]: Yeah, so if we wanna learn a new task, we basically do a little bit of robot data collection. Then the model, these-

有点像什么？Swyx [00:30:04]: What is a little bit like?

Jason Ma [00:30:05]：是的，大概几个小时。对，我们——有些任务只要几十分钟，对吧？对。然后——Jason Ma [00:30:05]: Yeah, like couple hours. Yeah, we’re— In some of the tasks, like just tens of minutes, right? Yeah. And then-

那个切碎的环节花了45分钟。Swyx [00:30:11]: The chopping one was 45 minutes.

打开冰箱的时候是——Vibhu [00:30:13]: The open the fridge was on-

Swyx [00:30:14]：这就是一场面试——也可以说是给你们某位工程师安排的入职任务。Swyx [00:30:14]: And that’s an interview— That’s like an onboarding task for one of your engineers.

Jason Ma [00:30:18]: So,.

Vibhu [00:30:20]：如果你想来这里工作，有件有趣的事：找到任务，让机器人去做。Vibhu [00:30:20]: Fun thing if you wanna work here, you know. Find the task, have the robot do it.

Jason Ma [00:30:24]：给我提个新任务，我就去办公室收集一些数据，训练一个模型，看看会发生什么。Jason Ma [00:30:24]: Pitch me a new task and I’ll come to the office, collect some data, train a model and see what happens.

Vibhu [00:30:29]：系好你的鞋带。Vibhu [00:30:29]: Tie your shoes.

Jason Ma [00:30:31]：确实，这很有联系。是啊，那会很酷。不过，嗯，你知道的。收集少量数据，训练一个模型，然后我们就把模型部署到现实世界中。接着观察它的表现，然后——Jason Ma [00:30:31]: Truly, it’s tying. Yeah, that would be cool. But yeah. So, you know. A little, a small amount of data collected, train a model, then we just deploy the model in the real world. And then we look at the performance, then-

填写已收集的信息，意味着你是——你有一个人实际在操作——Vibhu [00:30:41]: To fill in collected means you are— you have a person actually operating-

运营中。Jason Ma [00:30:46]: Operating.

Vibhu [00:30:48]：先做一会儿这个任务。然后再训练？效果还可以，你也有所进步。你怎么衡量？Vibhu [00:30:48]: Doing the task for a bit. And then you train? And it does okay, you improve. How do you measure?

Jason Ma [00:30:56]：所以训练模型时，有一些指标可以看，对吧？就像训练一个模型，它会很好地收敛。现在，对于当下的机器人技术，我们会直接在真实世界中运行模型，对吧？就在采集数据时使用的同一套配置下，看看模型表现如何。有时候模型开箱即用，效果就非常好，对吧？基本不需要再做什么。但如果模型大概只达到七八成的效果，那你就看看它的表现，找出它不擅长的地方。然后可以再收集一些数据，再迭代，对吧？我觉得这就像是机器人领域版的基于人类反馈的强化学习，对吧？让机器人实际运行一遍。你可以给它打分：好、不好、还算不错。然后这些数据，也就是机器人自己的经验，就可以用来训练模型。你也可以自己再收集更多数据来训练模型，对吧？最终，我们的目标是让这个机器人成为一个平台。任何人都应该能在上面训练各种任务。Jason Ma [00:30:56]: So when you’re training the model, there’s, certain metrics you can look at right? Just like training a model, it converges nicely. Now, after that for robotics today, we just run the model in the real world, right, on the same setup that the data is collected to see how the model does. Sometimes the model just works great out of the box, right? You don’t really need to do much more. But when the model’s maybe like seventy percent, eighty percent there, then you see how the model behaves. You figure out what it’s not good at. Then you can collect a bit more data, then iterate, right? I think of this as the robotics version of reinforcement learning from human feedback, right? The robot rolls out. You can score it good, bad S- somewhat good. And then that data, the robot’s own experience can be used to train the model. And you can also just collect more data yourself to train a model, right? Eventually, our goal is like, this robot is gonna be our platform. Anyone should be able to train tasks on it.

长时程任务与通用机器人技术的瓶颈Long-Horizon Tasks and the Bottlenecks to General Robotics

Vibhu [00:31:59]：我想大致了解一下这会如何发展，以及相应的时间线。目前这些任务还算比较简单，比如把洗好的衣服拿出来——那是条毛巾，不能把它压坏。切点食材。还有另一类任务，比如搭建这个拍摄工作室，对吧？挪动东西，搬张桌子，把灯架起来。我觉得——Vibhu [00:31:59]: I wanna get a bit of a prediction and timeline to where that goes. Right now, the tasks are, somewhat simple, like- Take laundry out. It’s a towel. You can’t crush it. Chop some food. There’s also the side of like, okay, set up this filming studio, right? Move stuff around, bring a table, set up lights. I think-

Swyx [00:32:21]：管理我的数据中心。还有，呃——比如说，给另一台机器人充电。Swyx [00:32:21]: Manage my data center. And also, ref— like, recharge another robot.

Vibhu [00:32:29]：我们什么时候能看到工厂建成？或者说，要实现规模化，我们还需要什么？是模型吗？还是所有方面都要提升？看起来你们的硬件已经为此做好了准备，但我们什么时候才能看到它发展到这样一个阶段：我不再只是切一样东西，而是可以说：“好了，给我一份配方。”Vibhu [00:32:29]: When do we see the factory or what do we need to scale up? Is it model? Is it everything? It seems like your hardware is set for that but where do we see it go to a point where I can go from instead of just cut one thing to, “Okay, here’s a recipe.”

Jason Ma [00:32:49]：我想说，要实现真正的通用机器人——基本上就是能做任何事的机器人或模型，比如——Jason Ma [00:32:49]: I would say to get to truly general-purpose robots like basically a robot or a model that does anything, like-

Vibhu [00:32:58]：不是那个。就是，好吧，为了让讨论在技术层面稍微落地一些，比如——我们什么时候开始看到我们把这些反馈推送出去，对吧？所以——Vibhu [00:32:58]: Not that. Like, okay, so to keep it somewhat grounded on a technical level, like- when do we start to see us pushing that feedback, right? So-

Jason Ma [00:33:06]: I see.

Vibhu [00:33:06]：不只是非常简单的切碎——比如一个五到十分钟的序列，对吧？现在评估这一步可以在更多节点上中断。不只是装满一个药瓶——而是按顺序装十种不同的药片，对吧？是什么限制了我们？我们又在努力推动什么？Vibhu [00:33:06]: Instead of very simple chopping- Maybe a five to ten-minute sequence, right? Where now the eval step, it can break at many more points. Instead of just fill one pill bottle- Ten different pills all in a sequence, right? What’s limiting us? What are we pushing on?

Jason Ma [00:33:20]：是的，我觉得……是的，我觉得有几件事。第一，你们看过的洗衣演示，是要花一个小时才能完成整个流程，对吧？所以机器人完成一项完整工作的最长时长，已经达到一个小时左右了，对吧？但我觉得，要让它能完成任何任务，瓶颈还是在数据和硬件上，对吧？有些任务，比如整理空间，你可能需要的不只是平行夹爪，对吧？你可能需要更灵巧的东西，比如一只手，像人手那样，对吧？但如果你想用这只手来训练模型，就会受限于硬件的稳定性。比如说，手——如今最好的灵巧手还是不够耐用，很快就会过热，对吧？这就成了一个瓶颈，对吧？比如说，你——我们——真的得给它敷个冰袋——Jason Ma [00:33:20]: Yeah, I think. Yeah, I think a couple of things. One is that the laundry demo that you have seen, yeah, it’s gonna be one hour to finish a whole loop, right? So the longest duration the robot can do a full job is already in the hour range. Right? But to be able to do that for any task, I think the bottleneck is coming back to data and hardware, right? So for some of the tasks, for example, setting up the space, you might need more than a the parallel jaw gripper, right? You might need something more dexterous, like a hand, like a human hand, right? But then if you wanna train a model on the hand, then you’re bottlenecked by the hardware stability. Like, hand— Today, the best hands, dexterous hands are still not, durable enough. They overheat pretty quickly, right? So that becomes a bottleneck, right? Like, you— We— you quite literally have to put a ice bag-

Swyx [00:34:15]：冰。用来降温。Swyx [00:34:15]: Ice. To cool.

人类。Jason Ma [00:34:16]: Human.

我们不只是自动进行研究，还要改进现有模型。我们有硬件限制。Vibhu [00:34:16]: We’re not just auto-researching, make better model we have. We have hardware limitations.

Jason Ma [00:34:20]：是的，所以硬件限制是个大问题。但如果硬件受限，那一切都会变慢，对吧？因为要知道任何任务是否取得了进展，最终都得在硬件上进行评估，对吧？所以，如果评估不能非常灵活、非常快速地开展，也会限制研究进度，对吧？因为那我们该怎么收集合适的数据呢？我在想，我们该怎么收集合适的数据，来完成极其复杂的任务，比如搭建这个工作室，对吧？你可以想象一下，如今大多数数据都是基于视觉的，对吧？比如视频流，以及一些对手部动作的编码，也就是机器人的关节运动。但人类拥有更多的感官。我们能听见，也能感知触觉。Jason Ma [00:34:20]: Yeah, so hardware limitation is a big thing. But then if hardware is limited, then that will slow down everything, right? Because to know that you’re making progress on any task, you eventually have to eval on the hardware, right? So then without the eval being very spontaneous, very fast, that also limits your research progress, right? Because then how do we collect the right data, right? I think, how do we collect the right data to do extremely complex, like setting up this studio, right? Because you can imagine, like today, most of the data is vision-based, right? It’s like video streams and some, encoding of your hand motion where the robot’s, joints. But Humans have a lot more sensors. We hear things. We feel things.

气味。Swyx [00:35:05]: Smell.

Jason Ma [00:35:06]：对，你会闻到气味，对吧？所以我们会在潜意识里运用这些感官，帮助我们熟练地操控物理世界。但如果今天我们完全没有采集这些信息，那么即便把数据规模扩大到数百万小时，你也可能仍然缺少任务中的关键维度，因此永远无法完成这个任务，对吧？所以我认为，这些就是拖慢进度的因素。但对于那些今天就能高保真地采集数据的任务或工作流程，我不认——我不认为其中存在根本性的瓶颈，对吧？就像我们本周向公众展示的那样，我们有一个能够完成整个洗衣流程的物理智能体，就像人类一样，对吧？因为对于这项任务，我们并没有遇到那种情况：比如，你实际上需要的不只是一个平行夹爪；你还需要某种目前尚不可用的传感方式，对吧？所以我确实认为，要实现更通用的能力，最大的瓶颈仍然是硬件和数据。Jason Ma [00:35:06]: Yeah, you smell things, right? So we subconsciously use all of those to help us be so good at manipulating the physical world. But today if we’re not capturing that at all, then even when we scale to millions of hours, you might just be missing critical dimensions of the task, so you’ll never be able to solve the task, right? So I think those are what I consider what’s dragging on the timeline. But for tasks or for workflows that you can collect data today to high fidelity, then I don’t ne— I don’t necessarily think there is a fundamental bottleneck, right? Just like what we are demonstrating, to the public this week. We have a physical agent that can do the entire laundry workflow, right, as a human would, right? Because for that task, we don’t really have like, oh, you actually need more than a parallel jaw gripper. You actually need some sensing modality that’s just not available today, right? So I do think the biggest bottleneck To have a much more general thing is still hardware and data.

实际应用部署：贝斯特韦斯特酒店和鼎泰丰Real-World Deployments: Best Western and Din Tai Fung

Swyx [00:36:06]：我想借这个机会聊聊——让你来夸夸你的客户和合作伙伴。你们已经公布了一些合作关系。我们可以先从 Best Western 说起，然后再聊聊其他一些让你们引以为豪的合作。有没有什么有意思的故事之类的？Swyx [00:36:06]: I want to take this chance to talk— let you brag about your customers and partners. You have some announced, relationships. We can start with Best Western and then maybe some of the others that you’re, that you’re proud of. Cool stories, anything like that.

Jason Ma [00:36:20]：我们网站上有一篇关于部署的博客文章，里面有更详细的介绍。我强烈建议观看视频，因为我觉得视频……Jason Ma [00:36:20]: So we have a deployment blog post, on our website that goes into much more detail. I really recommend watching the video because I think the video is-

Swyx [00:36:29]：哪一个？贝斯特韦斯特那家？Swyx [00:36:29]: Which one? The Best Western one?

Jason Ma [00:36:30]：不，我们为所有合作客户制作了一段视频。这段视频非常棒，展示了我们在做什么。我觉得很不错。我们合作的客户包括 Best Western，他们属于酒店行业。我们为他们折叠很多洗衣毛巾。每家酒店都需要，每家健身房、每个健身中心也都需要。-Jason Ma [00:36:30]: No, we have one video for all the customers we’re working with. It’s a really nice video. It shows what we’re doing. And yeah, I think it’s great. But yeah, we have customers, Best Western. That’s in the hotel space. We fold a lot of laundry towels for them. Every hotel needs it, every gym needs it, every fitness center needs it. -

Swyx [00:36:51]：目前部署得怎么样了？怎么说呢——我该去哪里——去看它的话，我该去哪家贝斯特韦斯特酒店？Swyx [00:36:51]: What’s the deployment so far? How— Like, where can I go to— Which Best Western do I go to see it?

Jason Ma [00:36:56]：所以如果你们想去什么地方，我其实推荐去我们的另一个部署点——鼎泰丰。你们去过——Jason Ma [00:36:56]: So if you wanna go somewhere, I actually recommend going to our other deployment, Din Tai Fung. Have you guys been to-

非常薄的封装层。Vibhu [00:37:04]: Very thin wrapper.

Swyx [00:37:05]：我们本该在那里拍摄的。Swyx [00:37:05]: We should have filmed there.

Jason Ma [00:37:09]：刚刚在那里拍摄了。Jason Ma [00:37:09]: Just filmed there.

Vibhu [00:37:10]: The B-roll.

Jason Ma [00:37:11]：不过在鼎泰丰，或者其他档次稍高的餐厅，桌上都会放餐巾，对吧？所以我们的机器人现在每天会为许多不同的地点折叠成千上万条餐巾，包括美国各地的很多家鼎泰丰门店。这实际上是我们规模最大的客户，对吧？所以我想在这里区分一下。如今有很多机器人公司都在谈部署、谈试点。但真正难的不是把机器人放到一个地点，或者做一次试点。真正难的是让客户相信或验证你的机器人能创造足够的价值，让他们愿意在旗下许多门店部署你的机器人。鼎泰丰就是这样，我们做到了。据我所知，其他做类似通用型机器人的公司还没有真正做到这一点。在鼎泰丰，我们的机器人每天要折叠成千上万条餐巾。它——餐巾的质量标准极高，因为他们要求，比如说，第二道折痕，或者餐巾的接缝，必须在那个……我都不知道该怎么称呼，口袋的下方几厘米处，这样就能把筷子插进去。所以这其实是一项非常精细、精度要求很高的任务。我可以给你看看。Jason Ma [00:37:11]: But in Din Tai Fung if you go to Din Tai Fung or other somewhat upscale restaurant, there’s napkins on your table, right? So our robots today fold thousands of napkins for many different locations, many different Din Tai Fung locations in the United States. So that’s actually our biggest scaled customer, right? So I wanna make a distinction here. Today, there’s a lot of robotics companies that’s talking about deployments, talking about pilots. But what’s really hard is not putting your robot in one location or doing a pilot. What’s really hard is that the customers believe or validate that your robot creates enough value that they wanna deploy your robots across many of their sites. So that’s what’s happened for us with Din Tai Fung, which, to the best of my knowledge, hasn’t really happened for other companies doing similar general purpose robots. So at Din Tai Fung, our robots fold thousands of napkins per day. It’s — The quality standard is extremely high, ‘cause they want like, okay, the second fold or the seam of the napkin is only like a couple centimeters below the I don’t even know how you call it, the pocket, so you can stick the chopsticks in. So it’s actually very fine-grained, high-precision task. I can show you.

Vibhu [00:38:26]：从某种程度上说，这正是机器人擅长做的，机器人也擅长这个，对吧？人类做事并不是最稳定的——尤其是在细节方面，而机器人多少能做到完美。很有意思，你举这个例子来说明通用机器人；但在某种程度上，我觉得人们会以为——你可以有一台专用机器人，能把餐巾纸折得完美，精确到毫米，而不是像这样的东西，不过……Vibhu [00:38:26]: At some level, that’s what robotics are good for robots are good for too, right? Humans are not the most consistent- Especially in small details, whereas a robot can somewhat do it perfectly. It’s interesting that’s the example you bring for general purpose robotics versus, at some level, I think people would assume- you can have a specialized robot that will fold napkins perfectly to the exact, millimeter as opposed to something like this but.

Jason Ma [00:38:53]：对，我相信，也许你能做到，对吧？但操控可变形物体很难，比如餐巾、毛巾，所有这些东西——都是可变形的。可能的形态配置简直有无穷无尽之多。所以这对专用机器人来说确实很难，对吧？我们发现，通过开发一个通用机器人模型，并让它适应这些应用场景，实际上可以更快达到生产级质量。特别好的一点是，现在鼎泰丰的业务规模扩大了，也和我们签下了大得多的合同。他们不只想让我们的机器人叠餐巾了，现在还想让机器人切东西。这就是我们举这个例子的原因：把食物切成——Jason Ma [00:38:53]: Yeah, I’m sure, maybe you can, right? But the hard thing about manipulating deformable object, right, like napkin, towels, everything’s- Deformable. There’s just infinite- infinitely many possible configuration. So that’s really hard for a specialized robot, right? And what we find is that by developing a general purpose model in a robot and adapting it to these use cases, you can actually get to a production quality faster. And what’s really nice is that now that Din Tai Fung has scaled, has signed a much bigger contract with us, they not only want our robots to fold napkins. Now they want our robots cut. That’s why we had the- Example, to cut food to the-

Swyx [00:39:31]：那真的是第二个用例吗？Swyx [00:39:31]: Is that actually the second use case?

这是他们正在考虑的一个使用场景。鼎泰丰每天需要切很多蔬菜。Jason Ma [00:39:33]: That’s one of the use case they are thinking. Din Tai Fung needs to cut so many vegetables per day.

Swyx [00:39:38]：他们得包饺子。Swyx [00:39:38]: They need to do the dumplings.

没错。到首屏。那就是最终 Boss。那是——Jason Ma [00:39:39]: Exactly. To the fold. That’s the final boss. That’s-

Vibhu [00:39:41]：这是最后一个。Vibhu [00:39:41]: That’s the final one.

Swyx [00:39:43]：我觉得你可以。我觉得你有足够的灵活度和精准度来完成这件事。Swyx [00:39:43]: I think you can. I think you have the dexterity and the precision to do this.

Vibhu [00:39:46]：你有一个小机器人。Vibhu [00:39:46]: You have a little robot.

Swyx [00:39:47]：不过，不要把饺子弄破了。Swyx [00:39:47]: But no, don’t break the dumplings.

Vibhu [00:39:50]：显而易见的好处是，你可以进一步调整它们。Vibhu [00:39:50]: That’s the obvious benefit is you can adapt them anymore.

通才型机器人：规模化的速度与一致性Generalist Robotics, Speed, and Consistency at Scale

Vibhu [00:39:54]：我觉得你提到的另一点其实是速度，对吧？要定制折叠纸巾的方式，还要做到精确的层数，不管是什么——你得先造出机器来。你不能直接在亚马逊上买到。但这个能做得很快，对吧？Vibhu [00:39:54]: I think the other thing you really touch on is the speed, right? To make a custom napkin folding with the exact ply, whatever- you have to make the machine. You can’t just Amazon this. But this can do it pretty fast, right?

Jason Ma [00:40:07]：是的，这能做到，而且速度非常快。现在它实际上已经达到了人类水平。而且这台机器人不会累，也不需要你知道……你可以……我觉得我很喜欢你提到的一致性，对吧？比如，鼎泰丰的一个问题是，他们在洛杉矶、纽约和西雅图的门店，叠餐巾的方式略有不同，但从他们的角度来说，他们希望保持一致。而这恰恰是这些模型擅长的，因为模型就是一组权重。你只要把这些权重部署到不同地区的不同机器人上，就行了——Jason Ma [00:40:07]: Yeah, this can do it ex— very fast. And now it’s actually al— it’s at human parity. And this robot doesn’t get tired, doesn’t need to you know. You can. And I think I really like the fact that you mentioned consistency, right? Like, one issue with Din Tai Fung is like, their LA location with New York location, Seattle location, the they fold napkins slightly differently, but from their perspective, they want consistency. But that’s exactly what these models are good at, ‘cause models are, just, it’s a set of weights. You can just put the weights in the different robots across different-

Vibhu [00:40:43]：你也可以通过 OTA 更新它们。Vibhu [00:40:43]: You can OTA update them as well.

Jason Ma [00:40:45]：对，你可以直接一次性把它们全都更新。就是……一旦成功了，我觉得真的，简直太不可思议了。Jason Ma [00:40:45]: Yeah, you can just update all of them at once. It just. Once it works, I think it’s truly, like- Incredible.

Vibhu [00:40:52]：最终大 boss 是 Cheesecake Factory。那里的东西都是从零开始做的。每样东西都非常一致，精确到克。每种酱汁都是现做的。无论你走到世界哪个地方，吃到的东西都一模一样。你要是不知道，那就不知道。但这就是最终大 boss。Vibhu [00:40:52]: Final boss is Cheesecake Factory. If the story there, they’re from scratch. Everything is very consistent, down to the measured gram. Every sauce is made. Like, anywhere you go in the world, you’ll have the exact same thing. If you don’t know, you don’t know. But that’s the that’s the final.

机器人领域的苦涩教训：扩大正确数据的规模The Bitter Lesson of Robotics: Scale the Right Data

Swyx [00:41:07]：我想到的说法有点像是机器人领域的“苦涩教训”，对吧？与其打造专用机器人——那需要大量工业工程——不如构建一个通用平台，然后可以……进行最后一公里的微调，再……Swyx [00:41:07]: The term that comes to mind is kind of like bitter lesson for robotics, right? You, instead of the specialized robots, which is a lot of industrial engineering, you build a generalized platform that can- Do a last-mile fine-tuning and-

Vibhu [00:41:18]: That’s right.

Swyx [00:41:19]：把这些事情都做好。Swyx [00:41:19]: And do all those things.

把专家附加项放在你的……之外。机器人领域还有什么更令人痛苦的教训？Vibhu [00:41:20]: Give the expert add-on outside of your. What is more bitter lesson robotics?

Jason Ma [00:41:25]：是的，我觉得机器人领域有好几条惨痛教训。其中一条是，机器人领域也会像机器学习已经解决的其他领域一样实现规模化，对吧？那这意味着什么？意味着要尽可能多地获取数据。但在机器人领域，难得多的一点是，比起其他领域，要弄清楚什么才是合适的数据要困难得多，对吧？语言就是文本。电子游戏领域就是游戏数据。但在机器人领域，你选择什么传感器，决定了你会获得什么样的数据，对吧？不过，如果你有一套合适的传感器，然后只管扩大数据规模，我觉得模型性能就会有可预期的提升。从这个角度来说，机器人领域并没有那么不同。这就是那条惨痛教训。由此推论，机器人领域有很多研究会引入非常具体的先验。比如说，我们需要让机器人像人一样看待物体，画几个框。这样能让机器人的感知能力更好，对吧？但我觉得这里的惨痛教训是：让算法保持简单，只需扩大合适类型的数据规模。这样一来，泛化能力和更好的行为就会自然涌现。所以这方面并没有那么不同。第二点，我觉得这就是主要的惨痛教训：归根结底，规模化是行得通的，但在机器人领域，要实现规模化以及如何实现规模化要困难得多，因为这里没有互联网。你得自己开发硬件，配好传感器，把这些都弄对，然后再扩大规模。我觉得，…。Jason Ma [00:41:25]: Yeah, I think there’s several bitter lessons for robotics. One is that robotics is going to scale just like any other domains that machine learning has solved, right? So what does that mean? That means having as much data as possible. But in robotics, the one thing that’s a lot harder is what is the right data is much harder to figure out than other domains, right? Language is text. In video games, just video game data. But for robotics what sensors you choose determine what kind of data you have, right? But if you have the right set of sensors and just scale that data, I think you’ll get predictable return on model performance. So in that way, robotics isn’t so different. So that’s the bitter lesson. As a corollary of that is that like there’s a lot of research in robotics that’s coming out with very specific priors. Like, for example, we need to make the robot see objects as they are, draw some boxes. That will give robot better perception, right? But I think the bitter lesson here is that just keep the algorithm simpler and scale the right kind of data. Then generalization, better behavior will emerge. So that’s not so different. And the second. I think that’s the main bitter lesson, which is at the end of the day, scaling will work, but in robotics, to scale and how to scale is just much harder ‘cause there is no internet. You have to come up with your hardware, come up with the sensors, get all of those right, and then scale. Then I think,.

人形机器人、实用迭代与通用性之路Humanoids, Practical Iteration, and the Path to Generality

Vibhu [00:42:53]：有一种说法是，世界在很大程度上是为人类打造的。它应该由人来完成，对吧？硬件方面的问题确实很棘手，但想想十年、二十年、一百年、两百年以后，我们能不能造出一个真正高效的人类？不过，你也得立足现实，考虑我们现在拥有的条件，对吧？Vibhu [00:42:53]: There’s the argument that the world is very made for humans. It should be a human, right? As hard of a hardware problem that is, think ten, twenty, a hundred, two hundred years down the line, can we make a really efficient human? But you have to be practical with what we have now, right?

Jason Ma [00:43:12]：我的想法是，没错，最终机器人应该是人形的。如果我们想打造一款通用机器人，也就是一台能够适应尽可能多种用途的机器，我认为人形形态会是最佳选择，毕竟我们这么做已经有很长时间了，对吧？但我认为，真正打造人形机器人的最佳方式是采取迭代式的方法。因为机器人技术实在太难了，整个问题、整个技术栈都非常庞大，所以很难事先就知道该做出什么样的正确决策，对吧？而且硬件方面的决策代价更高、成本也更大，因为硬件迭代要慢得多。因此，你实际上应该采取渐进式的迭代方法，而不是一步到位地追求最终形态。因为你打造出来的东西很可能根本撑不住，不够好。最后你还是得把它推倒重来，你懂吧？比如，Optimus 的情况已经是公开消息了，它已经经历了多轮迭代，他们不得不推翻原先的设计，对吧？我认为，通过采用半人形形态进行迭代，并真正把它们交给真实客户使用，或者让模型真正部署起来，你就能获得反馈闭环；如果你只是想造一台仓储人形机器人，然后花五年、十年去打造它，就得不到这样的反馈。我认为，迭代式的方法实际上能让我们更快接近最终形态的机器人。Jason Ma [00:43:12]: So the way I think about it is that yes, eventually the robot should. If we wanna build a general-purpose robot, right, like a machine that can be adapted to as many use cases as possible, I think human form factor will be the best one, just because that’s what we have been doing, for ages, right? But I think the best way to actually build a humanoid robot is to take an iterative approach. Because robotic is so hard, the whole problem, the whole stack is so long, it’s really hard to know a priori what’s the right decision to make, right? And because hardware decision is much more expensive and much more costly, just ‘cause hardware iteration is a lot slower, you actually want to take an incremental iterative approach, not going all the way to the final form factor. Because the chances are the thing you build actually doesn’t last. It’s not good enough. So you just end up scratching it anyway, you know. Like, it’s public news Optimus, already gone through many iterations, and they had to scratch their designs, right? And I think by iterating with a semi-humanoid form factor and actually putting them out to real customers or in getting models really deployed, you get that feedback loop that you wouldn’t get if you’re just trying to build like, warehouse humanoid and you spend like five, ten years building it. I think the iterative approach will actually get us closer to the final robot faster.

Vibhu [00:44:33]：我想说，对于只听音频、没看视频的人来说，它可能是我们见过动作最流畅的机器人。比如——Vibhu [00:44:33]: And I will say, for those listening and not watching, it’s probably the smoothest robot we’ve seen. Like-

连续动作空间与上下文的重要性Continuous Action Spaces and the Importance of Context

Vibhu [00:44:38]: ……你见过的每一个人形机器人动作都很生硬，甚至人类走路都没有这东西这么流畅。Vibhu [00:44:38]: … every humanoid you’ve seen is jerky and even humans don’t walk as smooth as this thing.

因为，没错，这个模型可以同时驱动所有电机。Swyx [00:44:43]: Because, yeah, the model is. Can move every single motor at once.

在两个以上的队列中，是的。Vibhu [00:44:48]: In more than two cohort, yeah.

Jason Ma [00:44:50]：是的，这实际上是个非常难的问题，因为如果你只考虑模型，它输出的是动作。这里有动作维度，对吧？维度越大，模型就越难拟合数据。但在这里，动作空间相当大，不过我们做得非常好，确保模型能够很好地拟合，并给你带来流畅的——Jason Ma [00:44:50]: Yeah, and that’s actually a very hard problem, because if you just think about the model, it’s outputting action. There’s the action, dimension, right? The bigger the dimension is, the harder to fit, the model to the data. But here, the action space is quite large, but we did very good job to make sure the model fits well and gives you a smooth-

Swyx [00:45:10]：你能大致估算一下动作空间的数量级吗？还是说……Swyx [00:45:10]: Can you bound the order of magnitude of the action space, or is that-

Jason Ma [00:45:14]：是的，大约有一百个动作维度。Jason Ma [00:45:14]: Yeah, so it’s on the order of, about a hundred action dimensions.

Swyx [00:45:19]：对，而且这些都是连续的——连续的，对吧？Swyx [00:45:19]: Yeah, and these are all continuou- continuous, right?

Jason Ma [00:45:21]：连续的，对吧。所以模型基本上就是以非常高的频率进行预测，比如每秒十次、二十次地预测一个几十维、至少五十维的向量，从而得到这种连续运动。Jason Ma [00:45:21]: Continuous, right. So the model is just basically predicting at very high frequency, like tens, like at least fifty-dimensional vector, like ten, twenty times a second to get that continuous motion.

Swyx [00:45:34]：还可以预测未来一到两秒的情况，也就是再往后推算一百帧左右。Swyx [00:45:34]: And potentially predicting one to two seconds out, so that is another hundred frames or whatever.

Jason Ma [00:45:39]: Of course.

Vibhu [00:45:40]：我想这里有一点历史记录，类似于之前状态的上下文。Vibhu [00:45:40]: There’s a bit of a history, similar to context of previous state, I guess.

Jason Ma [00:45:45]：所以，在机器人领域，上下文非常重要，对吧？因为如果只根据当前观察来做决策，有时候真的很难判断下一步该做什么，对吧？比如想想打开微波炉的过程，在中间阶段，你很难判断自己到底是不是——Jason Ma [00:45:45]: So context is a very important thing in robotics, right? Because otherwise, if you’re just making decision based on the current observation, sometimes it’s really hard to tell what to do next, right? For example if you think about opening a microwave, like in the middle part, it’s unclear whether or not you’re-

Vibhu [00:46:01]：结束Vibhu [00:46:01]: Closing

Jason Ma [00:46:01]：……是关闭它还是尝试打开它，对吧？所以，语境也非常重要。Jason Ma [00:46:01]: … closing or trying to open it, right? So context is also really important.

Swyx [00:46:05]：我还有一张冰箱演示的截图，里面展示了智能体式规划、工具调用以及其他这些功能。我们可以把它展示给听众看。Swyx [00:46:05]: And I have a screenshot of the fridge demo where it is, there’s agentic planning and tool holding and all those things. So we can show that for listeners.

Vibhu [00:46:13]：我觉得这是个有意思的方面。在模型——Vibhu [00:46:13]: I think that’s an interesting side. After the model-

智能体支撑框架与物理工作流规划Agentic Harnesses and Planning Physical Workflows

Vibhu [00:46:16]：……你怎么把它整合进智能体执行框架里？比如，什么是——Vibhu [00:46:16]: … how do you wrap it around into agentic harness? Like, what is-

Swyx [00:46:22]: Claude Code

Vibhu [00:46:22]：……Claude Code 部署？Claude Code UI。你们有很多很酷的 UI，那要如何把模型输出包装成最终产品？这是针对每种使用场景定制的吗？你们有自己的 SDK 吗？Vibhu [00:46:22]: … Claude Code deployment? Claude Code UI. You have a lot of cool UIs, but what goes into wrapping model output towards final thing? Is this custom per use case? Is it you have your own SDK?

Jason Ma [00:46:37]：所以，最终我们会有自己的 SDK，但我们的想法是，为机器人基础模型搭建一个通用的框架层。现阶段，我们用的是自己的模型，对吧？我的理解是，对于物理世界中的任何真实任务，尤其是耗时较长的任务，都需要进行大量高层次的决策，对吧？机器人要做的不只是针对物理世界进行推理并执行动作、生成连续轨迹，对吧？因为那只是非常底层的东西，是你实际在物理世界中执行的内容，但它并不是任务的意图。 比如说，我们做决策时，会考虑一系列步骤，就像菜谱里的步骤一样。我们会想：“哦，这一步做完了，接下来得做那一步。”或者：“等这部分完成的时候，我得去别的地方做点别的事。”这里面涉及大量高层次的认知和决策。我认为，机器人要想完成任务，哪怕是我们展示给你们看的洗衣任务，也必须具备这种能力。因为它需要推理：好，洗衣机正在运行，那我应该趁它运行的时候去做别的事。但现在洗衣机洗完了，我应该马上把毛巾从洗衣机里拿出来，放进烘干机里，对吧？否则，这就会成为整个工作流程中的瓶颈，对吧？ 所以，机器人需要进行大量推理。理论上，我认为你可以试着用一个输出动作的底层模型来学习这些能力，但效率会极低。这就是为什么我们需要分层系统：在世界动作模型周围搭建一个框架，比如一个智能体系统，让模型能够组合不同的能力和技能，去完成时间跨度更长的任务。 这很像语言模型，对吧？两三年前，模型只擅长自动补全。人类必须不断点击，模型才会继续生成代码。但如今，一个智能体系统就能端到端地完成任务。我认为这一点很重要，因为现在机器人领域的人仍然在非常具体的任务层面思考自主性，比如一分钟、三十秒的任务。但真正的价值在于那些能在无人监督的情况下持续数小时的任务或工作流程，而这正是我们正在努力实现的方向。 这也是为什么我认为，思考机器人问题时，不能只把它看作建模问题，而应该把它看作系统问题：你需要优秀的智能体、优秀的框架工程师、优秀的模型、优秀的硬件，样样都要好。这就是我们认为通用机器人应当如何实现的方式。Jason Ma [00:46:37]: So, eventually we will have our own SDK, but the way we think about is we want a general harness layer for robotics foundation model. Today, that is our own model, right? The way I think about is for any Real task, long task in the physical world, there is a lot of a high-level decision-making you have to do, right? It’s not just like robot, only reasoning about the physical world in the action, continuous trajectories, right? Because that’s a very low level, the thing you’re actually executing in the physical world, but that’s not the intent. But like if. When we are doing decision-making, we think about steps of a recipe. We think about, oh I finished this I need to do that. While I’m waiting for this part to finish, I need to go somewhere else to do something else. There’s a lot of high-level cognition, decision-making that’s happening. And I think for robots to handle even, the laundry task we are showing you, it has to have that capability. Because, it needs to reason that okay, the washer is running, so I should go do something else while the washer’s finishing. But now the washer is done, I should take, I should take the towels from the washer and put it in dryer immediately, right? Because if not, then that becomes a bottleneck in the whole workflow, right? So there is a lot of reasoning that needs to happen for robotics. And I think in theory, you could just, try to learn that with a low-level model that’s outputting actions, but it’ll be extremely inefficient. So that is why having a hierarchical system, like putting a harness, like an agentic system around our world action model is what allow the model to compose different capabilities, different skills to solve a much longer horizon task. It’s very much like in language model, right? Like two years ago, three years ago, the model was only good at autocomplete. Like, humans have to tap to get the model to keep spitting out code. But today, just an agentic system just carries the end-to-end task. And that’s. I, and I think that’s important because I think today in robotics, people are still thinking about autonomy at the very task level like one minute, thirty-second task. But really, the value is about tasks or workflows that go on unsupervised for hours. And that’s what we’re building towards. And that’s why I think thinking about. The robotics problem, not just as a modeling problem, but rather as a system problem, you need good agent, good harness engineer, good model, good hardware, everything. It’s like how we think general-purpose robots ought to be solved.

编码智能体如何加速机器人研究How Coding Agents Accelerate Robotics Research

Vibhu [00:49:07]：语言模型编程工具越来越好，你们觉得这会带来什么好处吗？比如新的 Opus 5.5、新的 Astra。你们的智能体框架里有用语言模型吗？语言模型领域在研发通用人工智能，这会对你们产生什么影响？Vibhu [00:49:07]: Do you guys see benefits from language model coding tools getting better? So new Opus 5.5, new Astra. Do you have language models in your agent harness? How does AGI being developed in language model space affect you guys?

Jason Ma [00:49:24]：不。是的，绝对是。它在很多方面都对我们影响很深。首先，我们是一家非常 AI 原生的公司，对吧？像编程智能体这类东西，都是我们工作中非常基础的一部分，而且我们非常欢迎它们。比如每次有新模型发布，我们总是第一个去测试，对吧？我们就是这样。GPT-6 发布时，我们那周就举办了全公司黑客马拉松，工程师们做出来的东西非常惊艳。而且他们中的很多人——我们开发的很多功能已经合并到 main 分支，投入生产了，对吧？所以我们真的很喜欢用编程智能体来加快工作进度。Jason Ma [00:49:24]: No. Yeah, no, absolutely. Like, it affects us very deeply in many ways. One is that we are extremely AI-native company, right? Like, coding agents, all of that is a very fundamental part of our work, and we’re very welcoming of that. Like, whenever there’s a new model, we’re the first one to test it, right? Like, we being. Like, when GPT-6 came out, we had a company-wide hackathon the same week, and the thing that our engineers built was incredible. And many of them are— Many of the features we built is already in main, in production, right? So we really like using coding agents to accelerate our work.

Swyx [00:50:00]：只是想问问感兴趣的朋友们，你发现 GPT-6 有什么令人意外的地方吗？Swyx [00:50:00]: Just for the people interested, anything surprising in GPT-6 that you found?

Jason Ma [00:50:05]：……关键在于它构建虚拟环境的能力，对吧？改变信息传播。我们二十分钟前谈过这个。我觉得这真的很令人兴奋。Jason Ma [00:50:05]: … it’s about the ability for it to construct virtual environments, right? Turns dissemination. We talked that twenty minutes ago. I think that’s really exciting.

挺有意思的。当时有很多 CAD、Zillow 这类偏 3D 的东西。Vibhu [00:50:17]: It’s interesting. There were so many CAD, Zillow to 3D-ish.

Swyx [00:50:21]：他在黑客马拉松比赛中输给了 CAD，-Swyx [00:50:21]: He lost the hackathon to the CAD, -

Vibhu [00:50:24]：我赢了那个。我做了 CAD 那堆破事。反过来。Vibhu [00:50:24]: I won the. I did the CAD bullshit. Other way.

Swyx [00:50:27]: No. Oh.

Vibhu [00:50:28]：是这么回事。我只是觉得好玩，就赢了一场 GPT 黑客马拉松。我当时想：“我对 CAD 一窍不通。”既然每个演示都在搞这个，那不如也玩玩。我心想：“这肯定会很烂。”结果我又想：“嗯，还挺靠谱的，也还不错。”不过这也就是黑客马拉松的东西。Vibhu [00:50:28]: That was. I won a GPT hackathon just for fun. I was like, “I know nothing CAD.” Might as well play around with it ‘cause every demo is doing it. I’m like, “It’s gonna be garbage.” And I’m like, “Well, it’s kinda grounded, it’s kinda good.” But it’s just hackathon stuff.

部署里程碑与竞争性机器人战略Deployment Milestones and Competing Robotics Strategies

Vibhu [00:50:45]:.Vibhu [00:50:45]:.

Swyx [00:50:45]：好。我本来——我本来打算再补充一些用例，然后聊聊公司的文化和你的个人背景。Swyx [00:50:45]: Cool. I wa— I was gonna round out, some of the use cases and then go into a little bit of company culture and your personal background.

Vibhu [00:50:52]：我有两个技术方面的问题。第一个是，你有没有想为自己设定什么目标，比如在六个月、一年或两年内，你觉得机器人应该能够做到什么——填空——而你正在朝这个目标努力？Vibhu [00:50:52]: I have two technical questions. One is, are there any goals you wanna set for yourself, like six months, one year, two year timeline that you see robots should be able to do, fill in the blank, that you’re kinda working towards?

Jason Ma [00:51:09]：是的，在未来六个月到一年的时间里，我们的目标肯定是扩大机器人和模型的规模，并真正让它们投入部署——Jason Ma [00:51:09]: Yeah, in the six months to one year frame, our goal is definitely scaling up our robots and our model and truly getting them to deploy-

像是一个任务。Vibhu [00:51:18]: Like a task.

Jason Ma [00:51:18]: … completely different.

Vibhu [00:51:19]：比如说，现在它是在切东西，或者打开冰箱。有没有……比如，你会不会给自己设定一个时间表，觉得到了这个时间，为了按计划推进，我们应该能够完成某件事？Vibhu [00:51:19]: Like, right now it’s cutting or opening a fridge. Is there. Like, do you set yourself to by this timeline, we should, to stay on track, be able to do whatever?

Jason Ma [00:51:29]：我认为，就部署而言，我们正在探索许多不同的行业。我们并没有认定必须攻克某一项特定任务，对吧？因为我们的目标是打造一个通用模型、一个通用平台。我们把每一项不同的任务都视为对平台的验证，也将其作为一个切入点，帮助我们摸索如何把整个系统打造得更好。因此，目前我们在洗衣和餐饮领域做了很多这类工作。我们已经部署了几十台机器人，而明年的总体目标就是部署数百台机器人。Jason Ma [00:51:29]: I think, in terms of deployments, we’re exploring many different industries. There isn’t a particular task we think we have to nail, right? Because our goal is to build a general model, a general platform, and we think of each different task as validation of our platform as a way, as a wedge for us to figure out how to build everything better. So today, we are doing a lot of that in laundry, in restaurants. We have already deployed tens of robots, but our goal next year is overall just deploying hundreds of robots.

Vibhu [00:52:02]：我关于上一个技术问题的第二部分是：你怎么看待行业里的其他人？也就是说，能不能给大家介绍一下，其他人正在采用哪些方法来构建世界模型？它们为什么不对？它们有哪些不同之处，又有哪些优点？总的来说，这个领域的机器人世界模型现状如何？以及，你是如何形成目前的看法的？Vibhu [00:52:02]: My part two on the last technical question is, how do you see the rest of the industry? So fill in for people, what other approaches are people taking to world models? Why are they wrong? What is different but good about their approaches as well? Just what is the landscape of robotic world models in this space and why— how have you come to your current?

Jason Ma [00:52:25]：不，这是个很棒的问题。我认为，机器人领域是当今最令人兴奋的领域，而且有非常多令人惊叹、让我由衷敬佩的公司。它们和我们一样，一直在做前沿探索，这非常令人振奋。我认为，我们现在看到的是，机器人领域依然存在许多不同的技术路线。比如，我们认为硬件需要自主研发，整个产业链都要掌握在自己手中。我们认为，扩大人类数据的规模至关重要。但也有公司押注于扩大机器人数据的规模，或扩大某种中间数据采集设备的规模，对吧？这些路线都在发展，我认为它们各有不同。我们就看看最终会怎样。我觉得现在判断还为时尚早，但我们对自己的做法相当有信心，因为从公司文化和使命来看，我们不是只想构建一个模型，而是真正要让机器人投入应用。所以，我们会尽可能多地部署机器人，攻克那些棘手的问题。这样一来，我们就能更早地发现问题。我认为这确实有助于我们找出真正的瓶颈并加以解决。对于只在问题某个环节上工作的人来说，这就像身处战争迷雾之中，看不到全貌。你可能做出了技术上非常酷的东西，但最终未必是人们真正需要或想要的，对吧？我认为，机器人公司有过不少失败案例，它们的机器人都非常炫酷、非常吸引人，但真正弄清楚人们到底想要什么，其实……“做出人们想要的东西”说起来很容易，人人都会这么说，但要经营好一家机器人公司，真正把这件事落到实处却非常难。Jason Ma [00:52:25]: No, that’s a great question. So I think robotics is the most exciting space today, and there are just so many, amazing companies that I truly respect. They’ve been doing great frontier work like us, and it’s very exciting. And I think what we are seeing today is that there’s still many different approaches to robotics. Like, we believe that hardware needs to be built in-house. We need to own the whole, chain. We believe that scaling human data is extremely important. But there are companies that’s betting on just scaling robots’ data or scaling some sort of intermediate data collection devices, right? All of them are in play, and I think it’s different approaches. We’ll just see what happens. I think right now, it’s still a bit early to tell, but we feel pretty good about the way we’re building things just because we as a company, culture-wise, mission-wise, we’re not here to just build a model. We’re here to truly get robots deployed. So we deploy as much as possible, work on the hard problems. So we identify problems much earlier on. So then I think that really helps us identify what the true bottlenecks are and then solve them. And I think for folks, for people who are only working on one part of the problem, it’s kind of like fog of war. You don’t really see the whole picture, and you might be building something that’s extremely cool technologically, but may not be what people need or want at the end, right? I think there’s a whole graveyard of robotics company, and I think all of them had very shiny, very cool robots, but just figuring out what people actually want is actually. It’s very easy to say “Build something people want.” everyone says that but it’s actually really hard to execute a robotics company.

世界动作模型与视觉-语言-动作模型对比World Action Models vs. Vision-Language-Action Models

Vibhu [00:54:07]：在技术方面，你们有自己的世界动作模型。这是行业通用标准吗？人们还在探索其他方向吗？Vibhu [00:54:07]: On the technical side, you have your world action model. Is that standard across industry? Is there other directions people are taking?

Jason Ma [00:54:15]：是的，我会说这也不是标准做法。所以我们是在世界模型的基础上构建世界动作模型。有很多公司都在构建世界模型。你提到了 Runway。他们也开始适配——Jason Ma [00:54:15]: Yeah, I would say that’s also not standard. So we’re building world action model on top of world model. So there is many companies building world model. You mentioned Runway. They’re also getting to adapt-

Vibhu [00:54:26]：视频世界模型。Vibhu [00:54:26]: Video world models.

也有一些公司正在构建视觉-语言-动作模型。Jason Ma [00:54:27]: There are also companies building a vision language action model.

VLA。Vibhu [00:54:31]: VLA.

Jason Ma [00:54:31]：这就像是把视觉语言模型应用到机器人技术中。我觉得有很多不同的做法。但归根结底，我们给模型起什么名字并没有那么重要，重要的是从第一性原理层面来看，我们认为模型需要具备哪些能力。我想稍微解释一下这一点。 我认为，对于机器人来说，最重要的是能够预测未来。对吧？机器人技术以及任何物理系统，根本上都是要根据当前及之前掌握的信息，判断下一步该做什么。所以关键在于预测接下来会发生什么。预测机器人的动作，就像是预测机器人下一步应该做什么。但仔细想想，机器人只是世界的一部分，对吧？机器人会影响世界，而世界也会影响机器人在那个环境中应当如何行动。所以从这个角度来看，你能做的最全面、最符合第一性原理的事，就是尽可能多地预测这个世界，对吧？这就是为什么我们认为预测未来极其重要。 如今，这一理念的具体实现形式是世界动作模型。VLA 只预测机器人动作，或者对世界的某种更抽象的表征。但每当你预测世界的抽象表征时，就会把自己的偏见编码进什么值得预测这一判断中。更深层的启示是：让我们消除人类的偏见，直接采用最原始的表征——摄像头输入，并对其进行全面预测。这就是我如今对机器人建模的看法。Jason Ma [00:54:31]: So that’s like taking vision language models and adapting them to robotics. So I think there’s many different ways to go about it. But I think at the end of the day what we call the models is less important than what we believe the model needs to be able to do at a first principle level. So I wanna explain that a bit. I think for robotics, the most important thing is that you need to be able to predict the future. Right. Like robotics, any physical system is fundamentally about based on the information you have at the present moment and before, what should you do next? So it’s really about predicting what comes next. So predicting robot actions is like predicting what the robot should do next. But if you think about it, robot is just a part of the world, right? Robot influences the world, but the world also influences how the robot should behave in that environment. So if you think about it this way, then the maximal, the most first principle thing you can do is just predict as much of the world that you can possibly predict, right? So that’s why we believe predicting the future is extremely important. And today, the instantiation is world action model. And VLA only predicts robot actions or some more abstract representation of the world. But whenever you’re predicting abstract representation of the world, you’re encoding your own bias into what’s important to predict. And the bigger lesson tells you, let’s remove human biases. Let’s just take the most raw representation, the camera inputs, and just predict all of that right? So that’s how I think about, modeling for robotics today.

Swyx [00:56:12]：那我再稍微追问一下，你基本上是在说，你们的方法比 VLA 更好，对吧？因为你们强调的是预测。我完全认同这一点。我——我是说，你已经说服我了。如果你错了，最有可能的第二个原因是——Swyx [00:56:12]: So just to press you a little bit on this you’re, basically you’re saying your approach is better than the VLA, right? Because you’re emphasizing prediction. I fully buy that. I— Like, you’ve sold me on it. If you were wrong, what is the most likely number two-

超越视觉：触觉、音频与红外感知Beyond Vision: Tactile, Audio, and Infrared Sensing

是其他人在做吗？Swyx [00:56:26]: Someone else is doing?

Vibhu [00:56:27]：有意思的问题。Vibhu [00:56:27]: Interesting question.

Jason Ma [00:56:28]：所以，是的，我之前说的话有个需要注意的地方，我觉得这也正是我可能会错的地方，对吧？我说过，你需要尽可能多地预测这个世界，对吧？但这并没有说明，你的传感器是否包含了预测世界所需的一切信息。我觉得我之前稍微提到过这一点，对吧？比如说，如果你训练的世界模型完全基于视频，我觉得它会是一个非常强大的世界模型，但它可能还不够丰富，因为你缺少了世界中的一些关键方面，比如声音、触觉，对吧？所以我觉得我们可能错在这里：好吧，我拥有最好的视觉世界模型，但明天有人推出一个视觉加触觉的世界模型，数据效率高出一百倍——仅仅是因为视觉是一种信息量很大，或者说冗余度很高的输入，因为相邻像素之间的变化并不大。你看——即便视觉不完美，你也能理解这个世界。但触觉或其他模态可能包含更密集的信息，对吧？所以，也许对这些信息进行预测，就能让你少用很多视觉信息。这样一来，你用少得多的算力和数据，就能达到相同的性能，对吧？但是——Jason Ma [00:56:28]: So yeah, there’s a caveat in what I’ve said before, and I think that’s where I could be wrong, right? I said that you need to predict the world as much as possible, right? But what that doesn’t say is that does your sensor have everything about the world that you need to predict? I think I touched on this a little bit, right? Like, if you’re training a world model that’s purely based on videos, I think it will be a very powerful world model, but it may not be a sufficiently rich enough world model because you’re missing critical aspects of the world, like sound, like touch, right? So I think where we could be wrong is like, okay, I have the best world model on vision-based world model, but tomorrow someone else comes out with a vision plus tactile world model that’s just like a hundred X more data efficient just because, vision is a very high information or a highly redundant, input because, every other pixel, they don’t change that much. You can see— you can understand the world without perfect vision. But tactile, other modality might be much denser in information, right? So maybe having that predicting that would just allow you to use much less vision. So you can get to the same performance with much less compute, with much less data, right? But-

Swyx [00:57:51]：我觉得你很容易就能适应。Swyx [00:57:51]: I think you will be able to adapt very easily.

Jason Ma [00:57:53]: No, exactly.

Vibhu [00:57:53]：数据量也有不同的层级。Vibhu [00:57:53]: There’s also a level of how much data.

Jason Ma [00:57:56]：对。我之所以提这个，是因为我认为，如果最终解决机器人问题的是视觉-触觉世界模型，或者视觉-触觉-音频世界模型，不管是哪种世界模型，那么机器人公司会比那些试图转型做机器人的视频生成公司做得更好。原因在于，理解感知实际上非常复杂。比如，我们从公司创立之初就一直在开发这方面的技术，所以我们知道瓶颈在哪里，也知道如何把这些感知方式结合起来。但如果你只是想把视频模型适配到机器人动作上，那么万一我们判断错了——视频模型并不足够——你就不太可能掌握如何让触觉发挥得很好。但我们掌握这方面的经验。Jason Ma [00:57:56]: Right. So the reason why I bring this up is because I think if a vision-tactile world model or vision-tactile-audio, whatever world model, is the final thing that solves robotics, then I think robotics companies will be able to do that better than video generation companies that’s trying to pivot to robotics. The reason is to understand sensing is actually very complicated. Like, we’ve been building that since the company started, so we know the bottleneck. We know how to combine them. But for— if you’re just trying to adapt a video model to robotics action, then if we were wrong, if, video model is not enough, then you don’t really have the know-how to get tactile to work very well. But we have that.

Swyx [00:58:39]：这是个题外话，不过我觉得很快就能说完。热量似乎非常重要。对你来说，红外线只不过是另一种颜色吗？还是说……。Swyx [00:58:39]: This is a side question, but I think it’s a quick one. Heat seems very important. Is infrared just another color to you, or is that a.

Jason Ma [00:58:47]：是啊，也可能是另一个——Jason Ma [00:58:47]: Yeah, it could be another-

只是另一种颜色。Swyx [00:58:48]: Just another color.

Jason Ma [00:58:48]：它可能是另一种颜色，也可能是模型的另一种输入。实际上，它可以用于烹饪场景。比如，我知道有些公司会用红外线来判断食物熟到什么程度。因为光靠 RGB 摄像头真的很难分辨，对吧？Jason Ma [00:58:48]: It could be another color, another input to the model. Could be used in cooking scenario, actually. Like, I know companies that’s using infrared to know how cooked food is. Because that’s really hard to tell from RGB camera, right?

Vibhu [00:59:01]：我对你最后一点的一个非常天真的反驳是，也许我们不需要理解硬件或云端代码，Codex 直接用氛围编程把它写出来就行。不了解也能做很多事情，不过当然——Vibhu [00:59:01]: My very naive pushback on your last point is maybe we don’t need to understand hardware, cloud code, or Codex can just, vibe code it in. There’s a lot you can do without understanding, but of course-

这不在发行版中。Swyx [00:59:15]: It’s just not in the distribution.

Jason Ma’s Journey: AlphaGo, Penn, NVIDIA, DeepMind, and Dyna

Swyx [00:59:16]：所以我想把话题稍微转到你的个人经历或背景上。我们都上过宾夕法尼亚大学。Swyx [00:59:16]: So I wanted to bring it a little bit to your personal story or your background. We both went to Penn.

我今天才刚知道。太疯狂了。Jason Ma [00:59:24]: I just found out today. Crazy.

Swyx [00:59:26]：是啊。我最大的遗憾就是没参加自动驾驶的 Docker 挑战赛。那样的话，我说不定就能早早加入 Waymo 之类的公司了。你之前在 Grasp 工作过，后来也在 Google 和 NVIDIA 工作过。你觉得学术界、Google 和 NVIDIA 的侧重点分别是什么？你都待过这些地方，对吧？然后显然，这些经历让你决定：“管它呢，我要自己干。”对吧？你本来还可以加入 OpenAI 的机器人实验室。所以，不如按时间顺序跟我们聊聊，也说说你从每个地方学到了什么。Swyx [00:59:26]: Yes. I, my big regret is I didn’t join the self-drive, the Docker challenge. Which, and then I could have been early Waymo or something. You were at Grasp. And then you’ve also worked at Google and NVIDIA. What are the different focuses of let’s say, academia versus Google versus NVIDIA? You’ve seen all these places, right? And then that obviously caused you to say “Screw all that. I’m gonna do my own thing,” right? You could have joined OpenAI’s robotics lab. So may-just take us through chronologically, but also one thing you learned from each.

Jason Ma [00:59:58]：按时间顺序来说，也许可以先聊聊我最初是怎么接触 AI 的，对吧？我小时候在日本和中国都生活过。我从小就下围棋，就是 AlphaGo 下的那种围棋。AlphaGo 问世时，我正读高中最后一年，简直惊呆了，因为我下围棋已经很多年了。这游戏太复杂了。你知道，我记得 AlphaGo 问世前，人们会说：“好吧，围棋是 AI 永远也攻克不了的游戏之一。”它的搜索空间比世界上原子的总数还大。你怎么可能在其中搜索出解法呢？后来强化学习（RL）就攻克了这个难题，对吧？我想这就是我最初接触 AI 的契机。我当时真的很感兴趣，想研究能够自我改进的模型、系统或智能体，对吧？所以我到宾夕法尼亚大学读博士时，就特别想研究强化学习。我是由两位导师共同指导的，他们当时都对强化学习感兴趣。其中一位导师更偏机器人方向。他曾在伯克利机器人实验室做博士后。我就是这样接触到机器人的。在宾夕法尼亚大学，我们有 GRASP 实验室，我觉得那里是研究机器人最好的地方之一。所以我博士期间就开始研究用于机器人的强化学习。后来，我做了不少研究，思考如何将强化学习与人类数据结合起来，以及如何以非常严谨的方式研究强化学习理论。之后我去了 NVIDIA。那时 GPT-4 刚刚问世。当时我们非常想解决一个转笔任务。我也不知道为什么。就像——Jason Ma [00:59:58]: So chronologically, yeah, maybe it’s useful to talk about how I got into AI in the beginning, right? So, as a kid, so I spent time in Japan and China. I grew up playing Go, the Go in AlphaGo. So when AlphaGo came out, I was a senior in high school, and I was mind-blown because I’ve been playing this game for many years. It’s so complicated. Like, I think before AlphaGo came out, people were like, “Okay, Go is one of the game that AI will never solve.” The search space is larger than the total number of atoms in the world. How could you possibly search for solutions in that? Then RL, reinforcement learning just cracked it, right? And I think that was how I got into AI initially. Like, I was just really interested in models or systems, agents that can self-improve, right? So when I got to Penn as a PhD student, I just really wanted to work on reinforcement learning. So I’m co-advised. Both my advisors were interested in RL at the time. One of my advisor was more in robotics. He did a postdoc at Berkeley Robotics Lab. So that’s how I got into robots. At Penn, we have GRASP Lab, I think one of the best places to do robotics. And so that’s how I got into doing RL for robotics in my PhD. And then, yeah, I did a bunch of research thinking about how to connect reinforcement learning, with human data, how to do reinforcement learning, theory in a very rigorous way. And then I went to NVIDIA. That’s when GPT-4 came out. At the time, we were very interested in just solving a pen spinning task. I don’t know why. Like, -

Swyx [01:01:35]：这是灵巧度的……这是灵巧度的极限，对吧？你能解魔方，然后你……Swyx [01:01:35]: It’s the dexter- it’s the dexterity limit, right? That you can solve Rubik’s cubes, and then you-

Jason Ma [01:01:39]：当时，我正在和……一起工作，-Jason Ma [01:01:39]: At the time, I was working with, -

Swyx [01:01:41]：金鹏Swyx [01:01:41]: Jinpeng

Jason Ma [01:01:41]：……UCO 的 Jinpeng，你们可能也认识。我们当时就想：“好，试试看让机器人灵巧地转笔。”最初的想法是让它从人类的视频中学习，对吧？我们想观察人类是怎么转笔的。但后来我意识到，好吧，如今我们还没有一只能够转笔的灵巧手。所以，即便从人类行为中学习确实可行，我又该如何评估算法呢？于是我想：“好，我们得先在仿真环境里试试。”我刚开始为一只拿着笔的机器人手搭建仿真环境，就立刻发现了问题所在：如果要我设计这个仿真环境，以及让强化学习解决转笔任务所需的奖励函数，那根本不可能。我没法想出所有必要的奖励项，让 RL 学会正确的行为。接着我就想：“如果有个能编写代码的模型可以做到这一点呢？”于是就有了 Eureka 项目，对吧？基本上……在我看来，我们在仿真环境中打造出了第一个递归式自我改进的机器人智能体，因为我们确实用它解决了一个此前没人演示过的新任务。这让我看到了使用语言模型、使用代码模型来真正加速机器人技术发展的潜力。 后来我去了 Google DeepMind，因为当时他们拥有 Gemini 这样的前沿模型，即使是现在也是如此，对吧？所以我开始研究如何将 Gemini 用于机器人技术。但最终我意识到，虽然我做学术研究时乐在其中，心里却总有个问题挥之不去：“这一切真的有意义吗？”你在实验室做这类研究时，会问我：“我们怎么才能想出这么多有趣又好玩的任务？”说到底，你是在自己做评估，自己给自己的作业打分。我想做些更贴近现实的事，想做出真正能被全世界数百万人使用的东西。我觉得，2024 年，也就是我们在 2024 年底创立 Dyna 的时候，正是把这些技术真正带入现实世界的好时机。所以这就是为什么。Jason Ma [01:01:41]: … Jinpeng at UCO, which you guys probably also know. Like, yeah, we were just like, “Okay, let’s try to do dexterous pen spinning.” The initial idea was learning it from human videos, right? We were trying to watch humans do it. But then I realized, okay, today we don’t have a dexterous hand that can spin pencil at all. So how would I even be able to evaluate the algorithm, even if learning from human worked at all? So I thought, “Okay, we need to first try this in simulation.” Then as soon as I tried to set up a simulated environment for a robot hand with a pen, I immediately saw what the issue is. The issue is like, if I had to come up with this simulation environment, the reward function for reinforcement learning to solve pen spinning, that’s impossible. Like, I wouldn’t be able to come up with all the terms that you need to get RL to learn the right behavior. And then I was just thinking, “What if a model that can write code can do that?” That’s what became the Eureka project, right? We basically. Like, in my opinion, we had the first recursive self-improving robotic agent in simulation, ‘cause we really used that to solve a new task that no one has demonstrated before. And that allowed me to see the potential of using language model, using coding models to really accelerate robotics. So then I went to Google DeepMind, ‘cause at the time, even today, they have one of the frontier models in Gemini, right? So I was looking at how to use Gemini for robotics. But eventually I realized like, I had a lot of fun doing academic research, but there was always a question in the back of my mind “Does any of this matter?” like, when you’re working on this kind of thing in a lab you ask me, “How do we come up with so many interesting, fun tasks?” You’re basically doing your own eval. You’re grading your own homework. Like, I wanted something more real. I wanted something that could actually be used by millions of people in the world. And I felt twenty-twenty-four was when we started Dyna, like end of twenty-twenty-four. That was the right time that we can take some of these technology to actually putting them out in the real world. So that’s why

Jason Ma [01:03:45]：我和 Lindon Gao、York Yang 一起创立了 Dyna。转眼两年过去，我们打造出了平台和模型。我们的机器人已经进驻鼎泰丰。去看看吧，然后——Jason Ma [01:03:45]: I started Dyna with Lindon Gao and York Yang. And fast-forward two years, we built our platform, our model. Our robots are in Din Tai Fung. Go check it out and, -

Vibhu [01:03:55]: Go eat.

Jason Ma [01:03:56]：是啊，去吃吧。这一路走来真不容易。而且，呃，不过，真正让我依然感到兴奋的是，比如说，我们创办 Dyna 时暂时搁置的很多想法，因为要创办一家公司，你必须非常务实。比如，我觉得 2024 年时仿真技术还没准备好。但现在，随着新一代前沿模型的出现，以及我们在内部看到的情况，这种情况可能会改变。这真的令人兴奋。Jason Ma [01:03:56]: Yeah, go eat. It’s been a journey. And,. But what still gets me really excited is, like some of— a lot of the ideas that I temporarily abandoned when we started Dyna, because, to build a company, you have to be very practical. Like, I think simulation wasn’t ready in twenty-twenty-four. But now with like the new frontier models and what we’re seeing internally this might change. And that’s very exciting.

产品驱动型研究、实验性评估与公司文化Product-Led Research, Experimental Evals, and Company Culture

Swyx [01:04:20]：那么，我们也来聊聊公司文化吧。你们秉持哪些原则来建设公司？显然，科学研究和研究议程由你负责。我感觉，每次和工程师交流时，他们都非常投入，也很有主人翁意识，动力十足。Swyx [01:04:20]: And so then also let’s talk about company culture. What principles you’re building the company with. Obviously the— you’re responsible for the science and the research, agenda. And it just seems like when we meet the engineers, they are, they’re very engaged and then they have a lot of ownership. They have a lot of motivations.

Vibhu [01:04:38]：我想说，看到这么多人做各种业余项目，确实挺有意思的。我们看过的一些演示，做这些项目的工程师就是觉得：“嗯，这看起来挺有意思的，不如就做做看。”说到你提的有趣评测，就在这儿，摄像机后面大概两英尺的地方——Vibhu [01:04:38]: I’ll say it’s interesting to see how many side projects people just work on. So some of the demos we’ve seen, it’s just the engineers that work on it were like “Yeah, this seems fun. Might as well do it.” To your point on fun evals, right here, like two feet behind the camera-

Swyx [01:04:55]：不能谈论那个。Swyx [01:04:55]: Can’t talk about that.

Vibhu [01:04:57]：……只是这个评估，就是那儿有几支荧光笔，四支荧光笔被橡皮筋捆在一起，对吧？就像，谁会想出“是的，我们得测试一下把橡皮筋套在高处”这种主意。就——Vibhu [01:04:57]: … just the e— the eval is just there’s highlighters, four highlighters being rubber banded together, right? It’s like, who comes up with, “ yes, we need to test putting rubber bands on high.” It’s just-

有点 Fissionary 的感觉Swyx [01:05:07]: Fissionary-ish

Vibhu [01:05:08]：……有趣的东西。这里的人都非常……富有创造力。这是个很美好的空间。Vibhu [01:05:08]: … fun stuff. People are very … creative here. It’s a beautiful space.

Jason Ma [01:05:12]：对，我觉得是文化。嗯，那我们先聊聊我们是如何经营公司的吧。我们认为自己最终会成为一家兼具研究与产品的公司。这有点像 OpenAI：开展前沿研究，但最终所有成果都会通过 ChatGPT 融入 Codex。我们对自己的定位也是如此。我们不是只做当下感兴趣、没有明确方向的事。我们始终以产品为导向。比如，我们想把机器人和平台部署到新的客户那里，那我们需要解决哪些技术问题呢？我们得解决泛化问题、全身控制问题、如何快速适应的问题、如何让机器人听懂语言的问题，还得解决灵巧操作的问题。把这些问题都明确下来之后，我们的工程师能力都很强，也很聪明，所以他们会围绕这些能力设计具体任务，给这些能力做压力测试。比如，抓取饮料这个任务，就是用来测试机器人是否能听从语言指令。你只要说清楚自己想要什么，如果模型能理解意图、听从指令，就应该能完成任务。这就是我们的思路：在顶层明确需要解决的问题，然后通过很多下层子项目推动执行，逐一解决这些问题。接着，招到合适的人。这里的每个人都对机器人充满热情，已经做了很久，也很热爱这件事。我们只需要营造一种文化，让这一切自然而然地发生。这里是个非常有意思的工作场所。Jason Ma [01:05:12]: Yeah, I think culture. Yeah, so let’s first talk about, how we run a company, right? Like, we think of ourself as like a like a eventually like a research plus product company. It’s kind of like OpenAI where you do frontier research, but eventually all of them make their way into Codex through ChatGPT. That’s how we think about it too. We’re not here to just do open-ended whatever thing that interests us at the time. We’re always product-driven. Like, we wanna deploy our robots, our platform to a new customer. What are the technical problems we have to solve, right? We have to solve generalization. We have to solve whole body control. We have to solve how to adapt quickly. We have to solve how to make the robots follow language. We have to solve dexterity. So once all those problems are very clear, then, like. Our engineers are very capable. They’re very smart. So they just create instances of the task. They create tasks that will stress test those capabilities, right? Like the j— the grabbing a drink task is just how we test language following, right? You can just specify what you want. If the model follows intent, follows instruction, it should be able to do it. So that’s how we think about, like. At the top level, we identify what are the things we have to solve. Then we have a lot of bottom sub-project to actually drive execution to solve them. And you hire the right people. And everyone here is super passionate about robots. They’ve been doing this for a long time. They love it. And we just create a culture where it just naturally happens. And it’s a really fun place to work.

团队：硬件、软件、研究与学习The Team: Hardware, Software, Research, and Learning

Vibhu [01:06:44]：公司在技术研究软件和硬件方面的业务占比如何？Vibhu [01:06:44]: What’s the split between technical research software versus hardware side of the company?

Jason Ma [01:06:51]：我们最近发展得非常快，所以我也说不上确切的数字。不过，硬件方面，我觉得大概有百分之三十到四十的人，因为硬件领域需要更多人手。然后大约有十个人在做模型。还有软件工程师、机器人工程师。业务团队里也有很多非常优秀的人才。创办一家机器人公司最有趣的地方，就是你什么都能接触到。任何可能有帮助的东西，都会派上用场，对吧？所以你有机会接触到很多来自不同背景的人，他们都对机器人充满热情，这让你收获特别多。作为创始人，我觉得这让我大开眼界。因为我在实验室做研究员或读博士的时候，每天都在和研究人员交流，那很棒，真的很好。我现在每天仍然能做这些事，但如今我还可以向许多来自不同人生轨迹的人学习。比如，我们有些负责采集数据的同事之前完全没有机器人领域的经验。他们原来在零售业工作，或者做其他工作，后来就来到了这里。你知道，他们也因此接受了很好的机器人培训。我觉得能看到这一点真的很了不起。我也觉得自己还在不断学习。我一直把自己看作机器人领域里一个持续学习的人，而我现在学到的东西比以往任何时候都多。尽管你想想，我现在已经不是学生了。我在这里经营一家公司，管理很多事情，但我还是觉得每天都能学到特别多。Jason Ma [01:06:51]: We recently grew so much, so I don’t have the perfect number. But I would say on the hardware side, we have, I would say probably like thirty percent, forty percent of hardware, just because hardware you need a lot more people. And then we have around ten people working on models. And then we have software engineers, robotics engineer. We have a lot of very talented people on the business side. The most fun thing about building a robotics company is you see everything. Everything under the sun that could possibly be helpful will be helpful, right? And so you just get so much exposure working with so many different people from many different backgrounds who are all very passionate about robots. And I think it’s been really eye-opening for me as a founder. ‘Cause when I was a researcher at a lab or a PhD student, I was talking to researchers every day, and that was great. That was really good. I still get to do that every day, but now I also get to learn just from so many different people from different walks of life. Like, some of our people who collect data, they had no robotics experience. They were working in retail, working in some other jobs, and they came here, you know. They also became very well-trained in robotics, and I think that’s just pretty amazing to see. And I think of myself as I’m still learning, you know. I think of myself as just always learning in the robotics space, and I haven’t learned so much as I am now. Even though, you think about me now, I’m not a student anymore. I’m here running a company, managing a lot of things, but I still feel like the day-to-day learning is just so much.

Dyna 的愿景及其名称的由来Dyna’s Vision and the Origin of Its Name

Swyx [01:08:29]：最后再问一个收尾问题，Dyna接下来会走向哪里？两年、五年后的愿景是什么？Swyx [01:08:29]: And then just to close off, where’s Dyna going? Two years, five years, what’s, the vision?

Jason Ma [01:08:37]：是的，我认为我们的愿景是切实地——我们的高层使命确实是把机器人和智能带入现实世界，对吧？也就是说，真正看到我们的机器人部署在成千上万的客户那里，甚至更多，达到数百万台机器人。这就是我们想要实现的目标。Jason Ma [01:08:37]: Yeah, I think our vision is to truly— The high-level mission is truly bring robots and intelligence, right, into the physical world. Like, actua— That means actually seeing our robots deployed thousands of customers, even more, millions of robots. That’s where we wanna get to.

Vibhu [01:08:58]：Dyna 那边的名字是什么来着？Dynamic？有什么故事吗？Vibhu [01:08:58]: What’s the what’s the name from Dyna? Dynamic? Is there a is there a story?

Jason Ma [01:09:03]：是的，有很多故事。所以我起——我给它起名叫 Dyna。内部有很多选项。Jason Ma [01:09:03]: Yeah, there’s many stories. So I na— I named Dyna. There were a lot of internal options.

Vibhu [01:09:09]: Take the credit.

Jason Ma [01:09:11]：……内部有不同的选择。不过这个名字是怎么来的，有两方面原因。第一，图灵奖得主、强化学习领域的先驱 Richard Sutton，多年前写过一篇名为《Dyna》的论文。它讲的基本上是一个能够自我改进的智能体，也就是世界模型。我觉得，即使我们创办公司时还没有涉足世界模型，我也认为机器人领域的问题最终会通过这种方式解决。现在，我们的技术方向也正逐渐朝这个方向靠拢。所以我觉得这是个很棒的灵感来源，而且我也做过强化学习，对 Richard 非常敬重。第二个原因则更私人一些。我在日本长大，我出生那年，正好有一部动画片。我不知道你们有没有看过《奥特曼》。Jason Ma [01:09:11]: … internal, different options. But how the name came to be was, there are two things. One is so, Turing Award winner, Richard Sutton, who was the pioneer of reinforcement learning, wrote a paper called Dyna many years ago. It’s about basically an agent that self-improves, world model. And I felt even when we started the company we weren’t doing world model yet, but I felt that’s how robotics will get solved. And now our technology direction is converging to that. So I felt that was a really neat inspiration, also because I worked in reinforcement learning, I have a lot of respect for Richard. The second thing is that it’s a bit more personal. Like, I grew up in Japan, and the year I was born, there was just a cartoon show. I don’t know if you guys have seen Ultraman.

Swyx [01:10:00]: Yeah, of course.

Jason Ma [01:10:01]：对，那一年那个奥特曼叫戴拿。所以，是的，这是件很微妙的事，没多少人知道，不过我想现在大家都会——Jason Ma [01:10:01]: The yeah, the Ultraman that year was called Dyna. So, yeah, it’s a very subtle thing not people know, but I guess now everyone would-

Vibhu [01:10:10]：团队也正在了解此事。Vibhu [01:10:10]: The team is finding out too.

Swyx [01:10:13]：不过我觉得理查德会很感激的。我不知道你以前有没有跟他聊过，不过——Swyx [01:10:13]: But I think Richard would appreciate it. I don’t know if you ever talked to him, but-

Jason Ma [01:10:16]：没有，我在 RL 会议上见过他。Jason Ma [01:10:16]: No, I’ve seen him at RL conferences.

Swyx [01:10:18]：你就应该直接说：“嘿，这家公司——”Swyx [01:10:18]: You should just be like, “Hey, this company-”

Jason Ma [01:10:20]: Yeah, I’ll invite him, come-

受到你的启发。Swyx [01:10:21]: Inspired by you.

Vibhu [01:10:22]：大家都知道理查德能带来更好的课程，不过你懂的。Vibhu [01:10:22]: Everyone knows Richard for a better lesson, but you know.

Jason Ma [01:10:24]：是的，这也是更好的教训。Jason Ma [01:10:24]: Yeah, also better lesson.

Swyx [01:10:27]：好的，最后还有什么想说的，或者有什么行动号召吗？Swyx [01:10:27]: Okay, any parting thoughts or calls to action?

结语：部署真实机器人，构建未来Closing: Deploying Real Robots and Building the Future

Jason Ma [01:10:31]：行动呼吁。是的，我觉得整个行业的机器人领域仍然非常注重演示。过去我在实验室里做过很多演示，我们在这里也不得不展示一些演示。但我认为，我们需要更多真正以使命为导向的人，不惜一切代价推动机器人落地部署，让机器人基础模型真正与硬件集成，并在真实客户现场投入使用。我认为我们在这方面走在前列，也希望看到更多人投身这一领域。对于正在观看的各位，如果这项使命能激励你，如果解决当今最棘手的问题让你每天早晨都充满动力，而你也想为此贡献力量，我们一直在招人。是的，期待你的加入。Jason Ma [01:10:31]: Calls to action. Yeah, I think I do feel like as an industry, robotics is still very demo-oriented. Like, I have done a lot of demos in the past in the lab, even here we had to show some demos. But I think we need more people that’s truly, mission-focused on doing whatever it takes to get robots deployed, get, robotic foundation models truly integrated with hardware and seeing them at real customer sites. I think we’re a pioneer on this regard, and I wanna see more in this field. And for the audience that’s watching, if this mission inspires you, if, solving the hardest problems today is what makes you wake up every day and you wanna contribute, we’re always hiring and yeah, looking forward to that.

Swyx [01:11:14]：湾区？还有别的地方吗？Swyx [01:11:14]: Bay Area? Anywhere else?

Jason Ma [01:11:16]：是的，今天只在湾区。Jason Ma [01:11:16]: Yes, just Bay Area today.

湾区。Swyx [01:11:18]: Bay Area.

Jason Ma [01:11:18]：不过，随着公司发展壮大，谁也不能说我们绝不会去其他地方。对，这就是我想说的，我希望世界上能有更多机器人。Jason Ma [01:11:18]: But as company grows, never say never to other locations. Yeah, but that’s what I wanna say, and I wanted to see more robots in the world.

Swyx [01:11:27]：太棒了。恭喜你们取得的一切成就。两年就有如此惊人的进展。真了不起。Swyx [01:11:27]: Cool. Congrats on everything. This is incredible progress for two years. Yeah. Amazing.

Jason Ma [01:11:34]：是的，我真的非常感谢。谢谢你来拜访我们。希望你玩得开心。Jason Ma [01:11:34]: Yeah, I really appreciate it. Thank you for visiting us. I hope it’s been fun.

Swyx [01:11:40]：我不后悔到现场来做这次采访。我们在自己的演播室里做过很多采访，但这次亲自到现场很值得。Swyx [01:11:40]: I don’t regret doing this on site. We do a lot of interviews in our studio, but this was worth it.

Jason Ma [01:11:48]：我觉得机器人这块得在现场做。Jason Ma [01:11:48]: I think for robotics, gotta do it on site.

Vibhu [01:11:51]：正在收听的朋友们，可以看看视频版。我们准备了有趣的补充镜头。Vibhu [01:11:51]: For people listening in, watch the video version. We have fun B-roll.