跟踪大语言模型应用教程

在本教程中，你将使用检索增强生成（RAG）构建一个客户支持聊天机器人，并在开发的每个阶段（从早期原型设计到生产）为其添加LangSmith可观测性。学完本教程后，你将了解如何：

跟踪单个大语言模型调用和完整的应用管道。
收集和查询用户反馈。
记录元数据并将其用于过滤和A/B测试。
使用监控仪表板跟踪生产性能。

该应用将检索相关的文档片段，并用它们来回答用户问题。本教程中的检索器是模拟的；在实际应用中，你可以将其替换为向量搜索或其他类似功能。

前提条件

开始之前，请确保你已具备：

一个LangSmith账户：在 smith.langchain.com 注册或登录。
一个LangSmith API密钥：按照创建API密钥指南操作。
一个OpenAI API密钥：从 OpenAI控制面板生成。
LangSmith CLI（可选）：安装后可从终端检查跟踪信息。安装说明请参考 LangSmith CLI。

安装所需的包：

pip install langsmith openai

npm install langsmith openai
npm install -D typescript tsx

原型设计

从一开始就设置好可观测性可以让你更快地迭代。你可以准确地看到发送给模型的内容、返回的结果以及时间花在哪里，而无需添加打印语句或运行调试器。

设置环境变量

在你的Shell中设置以下环境变量：

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

To send traces to a specific project, use the LANGSMITH_PROJECT environment variable. If this is not set, LangSmith will create a default tracing project automatically on trace ingestion.

你可能在其他地方看到这些变量被引用为 LANGCHAIN_*。两种方式都可以工作，但推荐使用 LANGSMITH_TRACING 和 LANGSMITH_API_KEY。

跟踪大语言模型调用

首先跟踪你的OpenAI调用，这是实际调用模型的地方。这让你能立即看到应用发送的提示词以及模型返回的响应。使用 wrap_openai（Python）或 wrapOpenAI（TypeScript）包装OpenAI客户端。创建一个名为 app.py（或 app.ts）的文件，包含以下代码：

from openai import OpenAI
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

async function supportBot(question: string): Promise<string> {
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
}

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

调用 support_bot("How many users can I have on the Starter plan?") 会生成一个OpenAI调用的跟踪记录。

跟踪整个管道

跟踪大语言模型调用很有用，但跟踪整个管道（包括检索）能让你全面了解应用的行为。在主函数上添加 @traceable（Python）或 traceable（TypeScript）：

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable } from "langsmith/traceable";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

const supportBot = traceable(async function supportBot(question: string): Promise<string> {
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

现在调用 support_bot("How many users can I have on the Starter plan?") 会生成整个RAG管道的跟踪记录。 LangSmith UI 显示一个跟踪记录，包含一个外部应用跨度和一个嵌套的大语言模型调用跨度。

从终端检查跟踪记录

如果你安装了 LangSmith CLI，无需打开UI即可列出项目的最近跟踪记录：

langsmith trace list --project <your-project> --limit 5

要查看特定跟踪记录的完整运行层次结构以及输入/输出：

langsmith trace get <trace-id> --full

测试

一旦你的应用在原型设计阶段运行良好，你就可以将其发布给一小部分真实用户。在这个阶段，你通常不知道用户会如何与你的应用交互，因此需要更丰富的可观测性。你不仅想知道应用做了什么，还想知道用户对其作何反应。

收集反馈

将用户反馈关联到特定的跟踪记录，可以帮助你识别哪些回答是有帮助的，哪些没有。更新上一步中的 app.py（或 app.ts），为每次调用添加一个运行ID，并在之后附加一个分数：

import os  

from openai import OpenAI
from langsmith import traceable, Client, uuid7  
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    run_id = str(uuid7())
    support_bot(
        "How many users can I have on the Starter plan?",
        langsmith_extra={"run_id": run_id},
    )
    ls_client = Client()
    # 反馈需要拥有该运行的跟踪项目的UUID  
    project_name = os.environ.get("LANGSMITH_PROJECT", "default")
    session_id = ls_client.create_project(project_name=project_name, upsert=True).id  
    ls_client.create_feedback(
        run_id, key="user-score", score=1.0, session_id=session_id  
    )

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable, getCurrentRunTree } from "langsmith/traceable";
import { Client } from "langsmith";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

let capturedRunId: string;
let capturedProjectName: string;

const supportBot = traceable(async function supportBot(question: string): Promise<string> {
    const runTree = getCurrentRunTree();
    capturedRunId = runTree.id;
    capturedProjectName = runTree.project_name;
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});

(async () => {
    await supportBot("How many users can I have on the Starter plan?");
    const lsClient = new Client();
    // 反馈需要拥有该运行的跟踪项目的UUID
    const { id: sessionId } = await lsClient.createProject({
        projectName: capturedProjectName,
        upsert: true,
    });
    await lsClient.createFeedback({
        runId: capturedRunId,
        sessionId,
        key: "user-score",
        score: 1.0,
    });
    await lsClient.flush();
})();

在生产环境中，这两部分会位于不同的位置：带有 run_id 的 support_bot 调用保留在你的应用中，而 create_feedback 则移到接收用户反馈的端点（例如，一个 /feedback API 路由）。run_id 从一个地方传递到另一个地方，以便将反馈关联到正确的跟踪记录。因为反馈也需要项目UUID，所以请将 session_id 与 run_id 一起传递。

当你在UI中检查运行时，反馈会出现在 Feedback（反馈）标签页中。然后，你可以使用 Runs（运行）表格中的过滤控件按反馈分数筛选运行。

记录元数据

元数据允许你用属性标记运行，这些属性有助于过滤和比较。例如，使用了哪个模型版本或哪个用户发起了请求。以下示例跟踪了检索器（使用 run_type="retriever"）和主函数（使用 metadata 属性记录模型名称）：

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

@traceable(run_type="retriever")
def retriever(query: str) -> list[str]:
    return docs

@traceable(metadata={"llm": "gpt-5.4-mini"})
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    support_bot("How many users can I have on the Starter plan?")

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable } from "langsmith/traceable";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

const retriever = traceable(  
    function retriever(query: string): string[] {
        return docs;
    },
    { run_type: "retriever" }
);

const supportBot = traceable(
    async function supportBot(question: string): Promise<string> {
        const context = await retriever(question);
        const systemMessage =
            "You are a helpful customer support agent. " +
            "Answer using only the information provided below:\n\n" +
            context.join("\n");
        const response = await client.chat.completions.create({
            model: "gpt-5.4-mini",
            messages: [
                { role: "system", content: systemMessage },
                { role: "user", content: question },
            ],
        });
        return response.choices[0].message?.content ?? "";
    },
    { metadata: { llm: "gpt-5.4-mini" } }
);

(async () => {
    await supportBot("How many users can I have on the Starter plan?");
})();

两个元数据值都会出现在跟踪记录上。你可以使用 Runs（运行）表格中的过滤控件按元数据筛选运行。

生产环境

有了强大的可观测性，你可以自信地将其部署到生产环境。在生产环境中，流量要大得多，你无法逐个检查每个跟踪记录。LangSmith提供监控工具，帮助你理解聚合行为，并在出现异常时深入排查。

监控

在UI侧边栏中，选择 Monitoring（监控），然后从左上角的下拉菜单中选择一个跟踪项目。图表会显示项目随时间变化的关键指标，包括跟踪数量、延迟、错误率、反馈分数和成本。有关可用指标和图表配置的更多信息，请参考仪表板。 LangSmith UI 显示监控页面，包含跟踪数量图表和可用标签页。

A/B 测试

分组功能要求给定的元数据键至少有两个不同的值。

由于你一直在记录 llm 元数据属性，你可以按该属性对监控图表进行分组，以比较模型性能随时间的变化。从UI侧边栏的 Monitoring（监控）中，单击左上角的 Group by（分组依据），从下拉菜单中选择 Metadata（元数据），然后选择 llm。图表会更新为按该属性分组显示结果。有关分组和自定义图表的更多信息，请参考仪表板。

深入排查

当监控图表显示异常时，单击一个数据点以固定工具提示，然后单击指标名称（例如，Input）跳转到该时间窗口的过滤运行表格。有关搜索和筛选运行的更多信息，请参考筛选跟踪记录。 LangSmith UI 显示监控页面，其中Input Tokens图表上的某个特定点被高亮显示。

LangSmith UI 显示监控页面，其中Input Tokens图表上的某个特定点被高亮显示。

总结

在本教程中，你为一个应用的整个开发生命周期添加了LangSmith可观测性。在原型设计阶段帮助你快速迭代的同一套跟踪设置，在生产环境中将继续提供价值。你将能够查看单个跟踪记录和聚合性能趋势。更多信息，请参见：

可观测性概念：术语和核心思想。
跟踪集成：LangChain、LangGraph、Anthropic 等提供商。
自动化规则：自动在跟踪记录上运行的规则和在线评估。

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.

Edit this page on GitHub or file an issue.

Tracing setup

Configuration & troubleshooting

前提条件

原型设计

设置环境变量

跟踪大语言模型调用

跟踪整个管道

从终端检查跟踪记录

测试

收集反馈

记录元数据

生产环境

监控

A/B 测试

深入排查

总结

​前提条件

​原型设计

​设置环境变量

​跟踪大语言模型调用

​跟踪整个管道

​从终端检查跟踪记录

​测试

​收集反馈

​记录元数据

​生产环境

​监控

​A/B 测试

​深入排查

​总结

前提条件

原型设计

设置环境变量

跟踪大语言模型调用

跟踪整个管道

从终端检查跟踪记录

测试

收集反馈

记录元数据

生产环境

监控

A/B 测试

深入排查

总结