看不见数据，如何造法律智能体
做法与步骤
- 因客户数据隐私，Legora 无法读取智能体运行轨迹，转而用三名工程师和三名律师搭建模拟客户工作的测试环境，类似强化学习中的“世界”。
- 评估团队将失败案例归纳为10到20种类型，并转述轨迹（paraphrasing traces）来定位问题，而非逐条查看原始日志。
- 一次发布后，引用准确率从70%跃升至99%，而 Legora 未改动任何代码，说明模型自身进步可解决部分问题。
- 产品上，Tabular Review 用百万单元格表格处理万份合同，前10行就能暴露问题；Slack 中的 intake bot 创建列表，列表可启动其他智能体。
- 团队规模从“300斯巴达人”幻灯片设想的20人永久够用，扩张到100人仍觉人手不足，反映需求增长快于预期。
- 所有服务运行在 Azure 上以满足数据隐私，不使用 ElevenLabs 做转录；曾长期抵制模型选择器，最终因客户需求添加。
对实践的启示
对做 AI 应用的团队，当无法直接查看用户数据时，可借鉴“模拟世界+失败分类+转述轨迹”的评估方法；产品设计上，表格和列表等简单抽象能有效组织智能体协作。
作者承认的反方与局限
作者承认评估并非总是值得，早期曾怀疑其价值；同时提到模型实验室可能像切断编码工具一样切断法律工具，存在平台风险。
原文摘引
How do you build and test an agent when you can't see its work?
A citation problem where the model got it right 70% of the time, then 99% after one release. Legora didn’t have to change anything.
Why drafting a long contract is a search problem
▸ 展开原文全文（71 段，抓取自 Latent Space）
Legora can’t read a single trace of what its agent does for a lawyer. The data belongs to the law firm’s clients. How do you build and test an agent when you can’t see its work?
Jacob Lauritzen is the CTO of Legora, one of the fastest-growing companies in legal tech history. The team started in a Stockholm basement with ChatGPT, Swedish law, and a search tool. Today, a lawyer can use Legora to draft and edit Word documents, review 10,000 contracts in a single table, or run due diligence. The agent also coordinates work between lawyers, clients, and other agents. When Legora made its own acquisitions, the team used it for diligence.
What we talked about
• Why Legora resisted adding a model picker for years, and why it finally added one
• Jacob’s estimate that models already do about 60% of legal work well, and why the rest takes a smarter model
• How three engineers and three lawyers build test environments modeled on customers’ work, much like the “worlds” used in reinforcement learning
• A citation problem where the model got it right 70% of the time, then 99% after one release. Legora didn’t have to change anything.
• Why Legora runs everything on Azure and why it doesn’t use ElevenLabs
• Tabular Review, the million-cell spreadsheet the industry copied down to the name
• How intake works, how agents create lists, and how a list can start another agent
• What legal engineers and forward-deployed engineers each do, and who gets to write code
• The 300 Spartans slide Jacob used to argue that 20 engineers would be enough forever. Now there are 100, and he still feels short-handed.
• Why Jacob finds Fable and Opus too chatty for drafting and likes Astra’s concise writing
• Why drafting a long contract is a search problem
• Whether model labs will cut off legal tools the way they cut off coding tools
• What happens to law firm partnerships, the billable hour, and jobs for junior lawyers
Legora is hiring engineers in New York, Stockholm, and London.
Chapters
0:00 Cold open
0:40 Meet Jacob Lauritzen, CTO of Legora
1:05 Starting out as underdogs in a Stockholm basement
1:54 How Legora grew from ChatGPT with Swedish law into a legal agent
4:12 The switch from assistant to agent
5:23 Which models Legora uses and why it added a model picker
7:48 Are today’s models good enough for lawyers?
8:20 What agents couldn’t do nine months ago
9:18 Testing the agent in simulated worlds and a 10,000-file data room
10:07 The evals team and Legora’s 100 legal engineers
11:03 When evals didn’t seem worth it and one release took citations from 70% to 99%
12:18 What legal practice could look like in five to ten years
14:18 Will the law firm partnership model survive?
16:40 What happens to paralegals and junior lawyers?
18:00 The billable hour and why earlier legal tech fell short
19:39 Where the money goes now
22:44 How Legora used its own product for acquisition diligence
23:59 How Swedish and American law firms differ
26:12 Product metrics when you can’t read the traces
27:41 Paraphrasing traces and narrowing failures down to 10 or 20 types
28:42 Why data privacy led Legora to run everything on Azure
30:04 Open-weight models and model arbitrage
30:55 Transcription models and why Legora doesn’t use ElevenLabs
33:33 The agent, Tabular Review, projects, and the Word add-in
34:24 How a lawyer uses Tabular Review
35:51 Why a spreadsheet works and what you learn from the first 10 rows
37:53 How the legal intake bot works in Slack
39:11 How agents create lists and lists start agents
40:15 Building products on those same pieces and acquiring Wexler
42:37 What forward-deployed engineers do at Legora
43:52 How legal engineers and forward-deployed engineers onboard a big client
46:43 What’s next for the forward-deployed team over the coming year
47:24 Why Legora resisted building APIs
49:26 Can the agent be the API?
50:12 The 300 Spartans slide and the plan to stay at 20 engineers
51:32 The bets nobody asked Legora to make
53:05 Post-training open-weight models for legal research
54:05 Search, retrieval, and late interaction
54:51 What makes a good legal agent harness and why drafting depends on search
56:25 Which model writes the best draft?
57:03 Hallucinations and Legora’s citation system
57:41 Do lawyers check the citations?
58:19 MapReduce or one big context window for a million-token contract?
1:00:16 Getting anonymized examples and the idea of buying a law firm for its data
1:01:53 Why Jacob advises against default embeddings
1:02:44 Will lawyers work in Legora or through Claude?
1:03:26 Will the model labs cut Legora off?
1:04:24 Law schools and scholarships for computer science majors
1:05:28 Hiring in New York, Stockholm, and London
1:06:09 Why most work eventually becomes legal work
Guest — Jacob Lauritzen, CTO of Legora
Co-host — swyx, Latent Space
Host — Basil Chatha, Forward Deployed