有什么比通过破坏系统更好的方式来了解它的极限呢？混沌测试在十多年前由 Netflix 普及，它是一种在正常运行的系统中故意引入受控故障以测试其弹性的做法。如今，一家围绕这一实践建立业务的初创公司正在利用人工智能来自动化更多的测试和故障排除流程。Gremlin 的 Foresight AI 是其现有“混沌工程”服务的一项新附加功能，它可以自主破坏软件基础设施，以便在生产环境中暴露隐藏的漏洞之前将其揪出来。
What better way to find out what your system can take than by breaking it? Popularized by Netflix more than a decade ago, chaos testing is the practice of deliberately introducing controlled failures into working systems to test their resilience. Now, a startup that built its business around this practice is using AI to automate more of the testing and troubleshooting process. Gremlin’s Foresight AI, a new add-on to its current “chaos engineering” services, autonomously breaks software infrastructure to uncover hidden bugs before they pop up in production.
对于工程师来说，混沌工程可能既有趣又危险：关闭一个随机服务器、负载均衡器，甚至整个云部署区域，然后观察系统其余部分在努力继续为客户服务方面的表现如何。如果成功了，说明工作做得不错。如果搞砸了，那就得做更多的可靠性工作。科尔顿·安德鲁斯（Kolton Andrus）是曾在该流媒体服务公司从事混沌工程的 Netflix 工程师之一，他创立了 Gremlin，旨在将故障注入引入企业，并配以对商业友好的配套设施，例如评分、高管报告和组织问责工具。
Chaos engineering can be deliciously dangerous for an engineer: Turn off a random server, load balancer or even an entire region of a cloud deployment, then observe how well the rest of the system tries to keep serving customers. If it does, job well done. If it's borked, more reliability work is afoot. Kolton Andrus, one of the Netflix engineers who did chaos engineering at the streaming service, founded Gremlin to bring fault injection to enterprises, along with business-friendly accoutrements such as scoring, executive reporting, and organizational accountability tools.
安德鲁斯告诉媒体，将人工智能融入其中大大加快了流程，它使许多过去需要手工完成的准备和测试后任务实现了自动化。
Adding AI into the mix speeds the process considerably, Andrus told The, by automating many of the preparatory and post-testing tasks that used to be done by hand.
故障即服务（Failure-as-a-Service）类的 AIOps 服务对网站可靠性工程师（SRE）来说并不新鲜，但其中大多数工具只有在系统已经崩溃后才诊断问题。安德鲁斯说：“许多 AI 解决方案都是：‘嘿，我们猜了一下。给你。祝你好运。’”相反，Gremlin 会有目的地让系统瘫痪，然后就如何防止这种情况再次发生提供专家建议。
Failure-as-a-Service AIOps services are nothing new for the site reliability engineer (SRE), but most of these tools diagnose issues only after the system has already crashed. "A lot of AI solutions are, 'Hey, we took a guess. Here you go. Good luck,'” Andrus said. Instead, Gremlin purposefully tips over the system and then offers expert advice on how to prevent that from happening again.
要使用 Gremlin，用户需要在其系统上安装代理，这些代理可以注入延迟、终止容器（pod）或耗尽内存。然后，Gremlin 云服务会观察随后的遥测警报，以寻找系统脆弱性的蛛丝马迹。
To use Gremlin, the user installs agents on their system that can inject latency, kill pods, or max out memory. A Gremlin cloud service then observes the subsequent telemetry alerts for telltale signs of systemic weakness.
Gremlin 的“秘密武器”是其“故障图谱”（Failure Atlas），这是该公司在过去十年中针对“数万个系统”开展的数百万次混沌工程实验的数据库，Andrus 表示。Gremlin 开发了一种工具，将大语言模型（LLM）与“故障图谱”相连接，使其能够生成更具技术依据的故障分析答案，而不仅仅是依赖自己从互联网上收集到的随机信息，该公司声称。
Gremlin’s secret sauce is its Failure Atlas, a repository of millions of chaos engineering experiments that the company conducted over the past decade on “tens of thousands of systems,” Andrus said. Gremlin developed a harness that yokes the LLM to the Failure Atlas, allowing it to generate more technically grounded answers about what went wrong, rather than simply drawing on its own collection of random information found on the Internet, the company asserts.
根据当时表现最佳的模型，Gremlin 会采用多种闭源和开源的 LLM 来完成这项任务。Andrus 表示，“故障图谱”能让 LLM 保持分析方向的正确性，从而最大限度地减少幻觉。不制造问题，只指出问题Gremlin 现有的防护机制围绕企业自身的访问权限和其他安全措施构建，以将“影响范围”（该公司所用术语）降至最低（如果影响范围完全无法控制，那显然就是访问控制的问题了）。
Gremlin uses a variety of closed and open LLMs for the job, depending on which one is working best at the time. The Atlas keeps the LLM on point, minimizing hallucinations, Andrus said. Not creating problems, just pointing them out Gremlin’s existing guardrails are built around the enterprise’s own access permissions and other security precautions to keep the “blast radius” (company’s words) to a minimum (if the blast isn’t contained at all, well, that’s an access control issue right there).
一旦系统故障被触发，Foresight AI 会确定根本原因，生成它认为能够修复问题的代码或配置更改，然后可以选择直接应用解决方案，或为 SRE 工程师准备一份报告。这基本上是一个“测试-替换”的智能代理循环——在关键节点保留人工干预——直到故障不再被触发为止。
Once a system failure is induced, Foresight AI determines the root cause, generates code or the configuration change it thinks will fix the issue, and then can either apply the solution, or prepare a report for an SRE to read. It’s basically an agentic loop of test-and-replace – keeping humans in the loop at critical junctures – that runs until the failure is no longer induced.
Andrus 表示，Gremlin 目前的用户群体主要是规模较大、计算密集型的企业，其中许多专注于金融服务、零售和企业级 SaaS 领域。他们使用 Gremlin 来测试灾难恢复计划，确保系统在非理想条件下仍能正常运行，并验证 Kubernetes 能否正确扩展。当这类复杂的分布式系统彻底崩溃时，诊断问题尤其困难。由于常规集成测试和单元测试难以捕捉到这些问题，通过注入网络延迟、内存耗尽等故障，暴露分布式系统的弱点，就能让一些隐藏的问题浮出水面。
Gremlin’s current user base tends to be larger compute-heavy enterprises, many specializing in financial services, retail and enterprise SaaS, Andrus said. They use Gremlin to test disaster recovery plans, ensure correct operation under less-than-ideal conditions, and make sure Kubernetes scales correctly. Such complex distributed systems are particularly difficult to diagnose when they go tits up. Difficult to catch with routine integration and unit testing, some of these bugs can be flushed out of hiding by injecting faults such as network latency, memory exhaustion, or other conditions that expose weaknesses in distributed systems.
Andrus在接受《The》采访时解释说，企业急于利用人工智能来编写应用程序和部署基础设施，但也带来了自身的一系列复杂问题。人工智能的运行速度极快，人类审核人员根本跟不上；而且人工智能在任何地方都可能犯下低级错误。Intellyx知名分析师Jason English在该技术相关的一份报告中指出，任何声称为了“更大的利益”而破坏系统的服务，都需要获得企业最高层级的批准。他写道：“为了让这项技术有效运作，我们需要改变对风险的认知。”®
The corporate rush to use AI to both write applications and deploy infrastructure brings its own set of complications, explained Andrus in an interview with The. AI works at such a speed that it is impossible for human reviewers to keep up, and AI can make dumb mistakes anywhere. Any service that promises to break systems for the greater good will require sign-off at the highest corporate levels, said noted Intellyx analyst Jason English, in a note about the technology. For it to work effectively, “we need to change our mindset about risk,” he wrote.®