当前位置：首页 > news >正文

vLLM本地部署 Seed-OSS-36B-Instruct 并通过修改chat_template.jinja模板实现思考与非思考切换版本2

news 2026/7/22 1:23:25

vLLM本地部署 Seed-OSS-36B-Instruct 并通过修改chat_template.jinja模板实现思考与非思考切换

设备：4卡 Tesla T10
模型：cpatonn/Seed-OSS-36B-Instruct-AWQ-8bit
推理框架docker版vLLM：vllm-openai:nightly-8f8fda261a620234fdeea338f44093d5d8072879

实现思考预算设置可以看我上一个博客：https://blog.csdn.net/oTianShiYeDiaoMao123/article/details/155322503

这是第二个版本的jinja模板，修改如下：

去除了jinja模板中的与思考预算相关的部分，个人使用过程中发现思考预算用处不大。
现在只有思考和非思考模式，当在系统提示词中加入“seed_oss_no_reasoning”就会切换为非思考模式。
非思考模式下会自动去除系统提示词中的“seed_oss_no_reasoning”字段，保留系统提示词的其它部分，也就是非思考模式下也能用系统提示词了（第一版jinja模板不行）。

第一次创建容器后，执行以下命令，解决非思考模式下使用非流式输出时的bug（这个bug在第一版jinja文件修改的博客中有提到）

dockerexec-it seedsed-i's/return model_output, None/return None, model_output/'/usr/local/lib/python3.12/dist-packages/vllm/reasoning/basic_parsers.py

创建容器的指令

docker run -d\--gpus all\--memory 28g\--memory-swap 28g\--shm-size 28g\-p8080:8080\-v /home/abc/model:/model\--ipc=host\--name seed\--envVLLM_SLEEP_WHEN_IDLE=1\--envVLLM_USE_FLASHINFER_SAMPLER=1\--envOMP_NUM_THREADS=2\vllm/vllm-openai:nightly-8f8fda261a620234fdeea338f44093d5d8072879\/model/Seed-OSS-36B-Instruct-AWQ-8bit\--served-model-name Seed-OSS-36B-Instruct\--quantization compressed-tensors\--dtype float16\--enable-auto-tool-choice\--tool-call-parser seed_oss\--reasoning-parser seed_oss\--chat-template /model/Seed-OSS-36B-Instruct-AWQ-8bit/chat_template.jinja\--gpu-memory-utilization0.90\--max-model-len61440\--max-num-seqs8\--max-num-batched-tokens2048\--tensor-parallel-size4\--async-scheduling\--enable-prefix-caching\--disable-custom-all-reduce\--attention-config.backend FLASHINFER\--host0.0.0.0\--port8080

完整版jinja文件

{# ----------‑‑‑ special token variables ‑‑‑---------- #} {%- set bos_token = '<seed:bos>' -%} {%- set eos_token = '<seed:eos>' -%} {%- set pad_token = '<seed:pad>' -%} {%- set toolcall_begin_token = '<seed:tool_call>' -%} {%- set toolcall_end_token = '</seed:tool_call>' -%} {%- set think_begin_token = '<seed:think>' -%} {%- set think_end_token = '</seed:think>' -%} {%- set budget_begin_token = '<seed:cot_budget_reflect>'-%} {%- set budget_end_token = '</seed:cot_budget_reflect>'-%} {# Changed By Mokie #} {# ---------- Preprocess the system message ---------- #} {%- if messages[0]["role"] == "system" %} {%- if "seed_oss_no_reasoning" in messages[0]["content"] %} {%- set thinking_budget = 0 -%} {%- set processed_content = messages[0]["content"] | replace("seed_oss_no_reasoning\n", "") | replace("seed_oss_no_reasoning", "") | trim -%} {# 如果系统提示词除去seed_oss_no_reasoning还有其它内容，则将剩余部分赋值给system_message #} {%- if processed_content | length > 0 %} {%- set system_message = processed_content -%} {%- endif %} {%- set loop_messages = messages[1:] -%} {%- else %} {%- set system_message = messages[0]["content"] -%} {%- set loop_messages = messages[1:] -%} {%- endif %} {%- else %} {%- set loop_messages = messages -%} {%- endif %} {# 如果没定义思考预算，则将思考预算设置为无限 #} {%- if not thinking_budget is defined %} {%- set thinking_budget = -1 -%} {%- endif -%} {# ---------- Ensure tools exist ---------- #} {%- if not tools is defined or tools is none %} {%- set tools = [] %} {%- endif %} {# tools2doc.jinja #} {%- macro py_type(t) -%} {%- if t == "string" -%}str {%- elif t in ("number", "integer") -%}int {%- elif t == "boolean" -%}bool {%- elif t == "array" -%}list {%- else -%}Any{%- endif -%} {%- endmacro -%} {# ---------- Output the system block ---------- #} {%- if system_message is defined %} {{ bos_token + "system\n" + system_message }} {%- else %} {%- if tools is iterable and tools | length > 0 %} {{ bos_token + "system\nYou are Doubao, a helpful AI assistant. You may call one or more functions to assist with the user query." }} {%- endif %} {%- endif %} {%- if use_json_tooldef is defined and use_json_tooldef %} {{"Tool List:\nYou are authorized to use the following tools (described in JSON Schema format). Before performing any task, you must decide how to call them based on the descriptions and parameters of these tools."}} {{ tools | tojson(ensure_ascii=False) }} {%- else %} {%- for item in tools if item.type == "function" %} Function: def {{ item.function.name }}( {%- for name, spec in item.function.parameters.properties.items() %} {{- name }}: {{ py_type(spec.type) }}{% if not loop.last %},{% endif %} {%- endfor %}): """ {{ item.function.description | trim }} {# ---------- Args ---------- #} {%- if item.function.parameters.properties %} Args: {%- for name, spec in item.function.parameters.properties.items() %} - {{ name }} ({{ py_type(spec.type) }}) {%- if name in item.function.parameters.required %} [必填]{% else %} [选填]{% endif %}: {{- " " ~ (spec.description or "") }} {%- endfor %} {%- endif %} {# ---------- Returns ---------- #} {%- if item.function.returns is defined and item.function.returns.properties is defined and item.function.returns.properties %} Returns: {%- for name, spec in item.function.returns.properties.items() %} - {{ name }} ({{ py_type(spec.type) }}): {{- " " ~ (spec.description or "") }} {%- endfor %} {%- endif %} """ {%- endfor %} {%- endif %} {%- if tools is iterable and tools | length > 0 %} {{"工具调用请遵循如下格式:\n<seed:tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>value_1</parameter>\n<parameter=example_parameter_2>This is the value for the second parameter\nthat can span\nmultiple lines</parameter>\n</function>\n</seed:tool_call>\n"}} {%- endif %} {# End the system block line #} {%- if system_message is defined or tools is iterable and tools | length > 0 %} {{ eos_token }} {%- endif %} {# ---------- List the historical messages one by one ---------- #} {%- for message in loop_messages %} {%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %} {{ bos_token + message.role }} {%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %} {{ "\n" + think_begin_token + message.reasoning_content | trim + think_end_token }} {%- endif %} {%- if message.content is defined and message.content is string and message.content | trim | length > 0 %} {{ "\n" + message.content | trim + "\n" }} {%- endif %} {%- for tool_call in message.tool_calls %} {%- if tool_call.function is defined %}{% set tool_call = tool_call.function %}{% endif %} {{ "\n" + toolcall_begin_token + "\n<function=" + tool_call.name + ">\n" }} {%- if tool_call.arguments is defined %} {%- for arg_name, arg_value in tool_call.arguments | items %} {{ "<parameter=" + arg_name + ">" }} {%- set arg_value = arg_value if arg_value is string else arg_value | string %} {{ arg_value+"</parameter>\n" }} {%- endfor %} {%- endif %} {{ "</function>\n" + toolcall_end_token }} {%- endfor %} {{ eos_token }} {%- elif message.role in ["user", "system"] %} {{ bos_token + message.role + "\n" + message.content + eos_token }} {%- elif message.role == "assistant" %} {{ bos_token + message.role }} {%- if message.reasoning_content is defined and message.reasoning_content is string and message.reasoning_content | trim | length > 0 %} {{ "\n" + think_begin_token + message.reasoning_content | trim + think_end_token }} {%- endif %} {%- if message.content is defined and message.content is string and message.content | trim | length > 0 %} {{ "\n" + message.content | trim + eos_token }} {%- endif %} {# Include the tool role #} {%- else %} {{ bos_token + message.role + "\n" + message.content + eos_token }} {%- endif %} {%- endfor %} {# ---------- Control the model to start continuation ---------- #} {%- if add_generation_prompt %} {{ bos_token + "assistant\n" }} {%- if thinking_budget == 0 %} {{ think_begin_token + "\n" + budget_begin_token + "The current thinking budget is 0, so I will directly start answering the question." + budget_end_token + "\n" + think_end_token }} {%- endif %} {%- endif %}

查看全文

http://www.cnnetsun.cn/news/110299.html