Tool use enables LLMs to request calls to external functions and APIs through the `/v1/chat/completions` endpoint, via LM Studio's REST API (or via any OpenAI client). This expands their functionality far beyond text output.

## Quick Start ### 1. Start LM Studio as a server To use LM Studio programmatically from your own code, run LM Studio as a local server. You can turn on the server from the "Developer" tab in LM Studio, or via the `lms` CLI: ```bash lms server start ``` ###### Install `lms` by running `npx lmstudio install-cli` This will allow you to interact with LM Studio via the REST API. For an intro to LM Studio's REST API, see [REST API Overview](/docs/developer/rest). ### 2. Load a Model You can load a model from the "Chat" or "Developer" tabs in LM Studio, or via the `lms` CLI: ```bash lms load ``` ### 3. Copy, Paste, and Run an Example! - `Curl` - [Single Turn Tool Call Request](#example-using-curl) - `Python` - [Single Turn Tool Call + Tool Use](#single-turn-example) - [Multi-Turn Example](#multi-turn-example) - [Advanced Agent Example](#advanced-agent-example) ## Tool Use ### What really is "Tool Use"? Tool use describes: - LLMs output text requesting functions to be called (LLMs cannot directly execute code) - Your code executes those functions - Your code feeds the results back to the LLM. ### High-level flow ```xml ┌──────────────────────────┐ │ SETUP: LLM + Tool list │ └──────────┬───────────────┘ ▼ ┌──────────────────────────┐ │ Get user input │◄────┐ └──────────┬───────────────┘ │ ▼ │ ┌──────────────────────────┐ │ │ LLM prompted w/messages │ │ └──────────┬───────────────┘ │ ▼ │ Needs tools? │ │ │ │ Yes No │ │ │ │ ▼ └────────────┐ │ ┌─────────────┐ │ │ │Tool Response│ │ │ └──────┬──────┘ │ │ ▼ │ │ ┌─────────────┐ │ │ │Execute tools│ │ │ └──────┬──────┘ │ │ ▼ ▼ │ ┌─────────────┐ ┌───────────┐ │Add results │ │ Normal │ │to messages │ │ response │ └──────┬──────┘ └─────┬─────┘ │ ▲ └───────────────────────┘ ``` ### In-depth flow LM Studio supports tool use through the `/v1/chat/completions` endpoint when given function definitions in the `tools` parameter of the request body. Tools are specified as an array of function definitions that describe their parameters and usage, like: It follows the same format as OpenAI's [Function Calling](https://platform.openai.com/docs/guides/function-calling) API and is expected to work via the OpenAI client SDKs. We will use [lmstudio-community/Qwen2.5-7B-Instruct-GGUF](https://model.lmstudio.ai/download/lmstudio-community/Qwen2.5-7B-Instruct-GGUF) as the model in this example flow. 1. You provide a list of tools to an LLM. These are the tools that the model can _request_ calls to. For example: ```json // the list of tools is model-agnostic [ { "type": "function", "function": { "name": "get_delivery_date", "description": "Get the delivery date for a customer's order", "parameters": { "type": "object", "properties": { "order_id": { "type": "string" } }, "required": ["order_id"] } } } ] ``` This list will be injected into the `system` prompt of the model depending on the model's chat template. For `Qwen2.5-Instruct`, this looks like: ```json <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {"type": "function", "function": {"name": "get_delivery_date", "description": "Get the delivery date for a customer's order", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } <|im_end|> ``` **Important**: The model can only _request_ calls to these tools because LLMs _cannot_ directly call functions, APIs, or any other tools. They can only output text, which can then be parsed to programmatically call the functions. 2. When prompted, the LLM can then decide to either: - (a) Call one or more tools ```xml User: Get me the delivery date for order 123 Model: {"name": "get_delivery_date", "arguments": {"order_id": "123"}} ``` - (b) Respond normally ```xml User: Hi Model: Hello! How can I assist you today? ``` 3. LM Studio parses the text output from the model into an OpenAI-compliant `chat.completion` response object. - If the model was given access to `tools`, LM Studio will attempt to parse the tool calls into the `response.choices[0].message.tool_calls` field of the `chat.completion` response object. - If LM Studio cannot parse any **correctly formatted** tool calls, it will simply return the response to the standard `response.choices[0].message.content` field. - **Note**: Smaller models and models that were not trained for tool use may output improperly formatted tool calls, resulting in LM Studio being unable to parse them into the `tool_calls` field. This is useful for troubleshooting when you do not receive `tool_calls` as expected. Example of an improperly formatting `Qwen2.5-Instruct` tool call: ```xml ["name": "get_delivery_date", function: "date"] ``` > Note that the brackets are incorrect, and the call does not follow the `name, argument` format. 4. Your code parses the `chat.completion` response to check for tool calls from the model, then calls the appropriate tools with the parameters specified by the model. Your code then adds both: 1. The model's tool call message 2. The result of the tool call To the `messages` array to send back to the model ```python # pseudocode, see examples for copy-paste snippets if response.has_tool_calls: for each tool_call: # Extract function name & args function_to_call = tool_call.name # e.g. "get_delivery_date" args = tool_call.arguments # e.g. {"order_id": "123"} # Execute the function result = execute_function(function_to_call, args) # Add result to conversation add_to_messages([ ASSISTANT_TOOL_CALL_MESSAGE, # The request to use the tool TOOL_RESULT_MESSAGE # The tool's response ]) else: # Normal response without tools add_to_messages(response.content) ``` 5. The LLM is then prompted again with the updated messages array, but without access to tools. This is because: - The LLM already has the tool results in the conversation history - We want the LLM to provide a final response to the user, not call more tools ```python # Example messages messages = [ {"role": "user", "content": "When will order 123 be delivered?"}, {"role": "assistant", "function_call": { "name": "get_delivery_date", "arguments": {"order_id": "123"} }}, {"role": "tool", "content": "2024-03-15"}, ] response = client.chat.completions.create( model="lmstudio-community/qwen2.5-7b-instruct", messages=messages ) ``` The `response.choices[0].message.content` field after this call may be something like: ```xml Your order #123 will be delivered on March 15th, 2024 ``` 6. The loop continues back at step 2 of the flow Note: This is the `pedantic` flow for tool use. However, you can certainly experiment with this flow to best fit your use case. ## Supported Models Through LM Studio, **all** models support at least some degree of tool use. However, there are currently two levels of support that may impact the quality of the experience: Native and Default. Models with Native tool use support will have a hammer badge in the app, and generally perform better in tool use scenarios. ### Native tool use support "Native" tool use support means that both: 1. The model has a chat template that supports tool use (usually means the model has been trained for tool use) - This is what will be used to format the `tools` array into the system prompt and tell them model how to format tool calls - Example: [Qwen2.5-Instruct chat template](https://huggingface.co/mlx-community/Qwen2.5-7B-Instruct-4bit/blob/c26a38f6a37d0a51b4e9a1eb3026530fa35d9fed/tokenizer_config.json#L197) 2. LM Studio supports that model's tool use format - Required for LM Studio to properly input the chat history into the chat template, and parse the tool calls the model outputs into the `chat.completion` object Models that currently have native tool use support in LM Studio (subject to change): - Qwen - `GGUF` [lmstudio-community/Qwen2.5-7B-Instruct-GGUF](https://model.lmstudio.ai/download/lmstudio-community/Qwen2.5-7B-Instruct-GGUF) (4.68 GB) - `MLX` [mlx-community/Qwen2.5-7B-Instruct-4bit](https://model.lmstudio.ai/download/mlx-community/Qwen2.5-7B-Instruct-4bit) (4.30 GB) - Llama-3.1, Llama-3.2 - `GGUF` [lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF](https://model.lmstudio.ai/download/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF) (4.92 GB) - `MLX` [mlx-community/Meta-Llama-3.1-8B-Instruct-8bit](https://model.lmstudio.ai/download/mlx-community/Meta-Llama-3.1-8B-Instruct-8bit) (8.54 GB) - Mistral - `GGUF` [bartowski/Ministral-8B-Instruct-2410-GGUF](https://model.lmstudio.ai/download/bartowski/Ministral-8B-Instruct-2410-GGUF) (4.67 GB) - `MLX` [mlx-community/Ministral-8B-Instruct-2410-4bit](https://model.lmstudio.ai/download/mlx-community/Ministral-8B-Instruct-2410-4bit) (4.67 GB GB) ### Default tool use support "Default" tool use support means that **either**: 1. The model does not have chat template that supports tool use (usually means the model has not been trained for tool use) 2. LM Studio does not currently support that model's tool use format Under the hood, default tool use works by: - Giving models a custom system prompt and a default tool call format to use - Converting `tool` role messages to the `user` role so that chat templates without the `tool` role are compatible - Converting `assistant` role `tool_calls` into the default tool call format Results will vary by model. You can see the default format by running `lms log stream` in your terminal, then sending a chat completion request with `tools` to a model that doesn't have Native tool use support. The default format is subject to change.

Expand to see example of default tool use format

```bash -> % lms log stream Streaming logs from LM Studio timestamp: 11/13/2024, 9:35:15 AM type: llm.prediction.input modelIdentifier: gemma-2-2b-it modelPath: lmstudio-community/gemma-2-2b-it-GGUF/gemma-2-2b-it-Q4_K_M.gguf input: "system You are a tool-calling AI. You can request calls to available tools with this EXACT format: [TOOL_REQUEST]{"name": "tool_name", "arguments": {"param1": "value1"}}[END_TOOL_REQUEST] AVAILABLE TOOLS: { "type": "toolArray", "tools": [ { "type": "function", "function": { "name": "get_delivery_date", "description": "Get the delivery date for a customer's order", "parameters": { "type": "object", "properties": { "order_id": { "type": "string" } }, "required": [ "order_id" ] } } } ] } RULES: - Only use tools from AVAILABLE TOOLS - Include all required arguments - Use one [TOOL_REQUEST] block per tool - Never use [TOOL_RESULT] - If you decide to call one or more tools, there should be no other text in your message Examples: "Check Paris weather" [TOOL_REQUEST]{"name": "get_weather", "arguments": {"location": "Paris"}}[END_TOOL_REQUEST] "Send email to John about meeting and open browser" [TOOL_REQUEST]{"name": "send_email", "arguments": {"to": "John", "subject": "meeting"}}[END_TOOL_REQUEST] [TOOL_REQUEST]{"name": "open_browser", "arguments": {}}[END_TOOL_REQUEST] Respond conversationally if no matching tools exist. user Get me delivery date for order 123 model " ``` If the model follows this format exactly to call tools, i.e: ``` [TOOL_REQUEST]{"name": "get_delivery_date", "arguments": {"order_id": "123"}}[END_TOOL_REQUEST] ``` Then LM Studio will be able to parse those tool calls into the `chat.completions` object, just like for natively supported models.

All models that don't have native tool use support will have default tool use support. ## Example using `curl` This example demonstrates a model requesting a tool call using the `curl` utility. To run this example on Mac or Linux, use any terminal. On Windows, use [Git Bash](https://git-scm.com/download/win). ```bash curl http://localhost:1234/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "lmstudio-community/qwen2.5-7b-instruct", "messages": [{"role": "user", "content": "What dell products do you have under $50 in electronics?"}], "tools": [ { "type": "function", "function": { "name": "search_products", "description": "Search the product catalog by various criteria. Use this whenever a customer asks about product availability, pricing, or specifications.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "Search terms or product name" }, "category": { "type": "string", "description": "Product category to filter by", "enum": ["electronics", "clothing", "home", "outdoor"] }, "max_price": { "type": "number", "description": "Maximum price in dollars" } }, "required": ["query"], "additionalProperties": false } } } ] }' ``` All parameters recognized by `/v1/chat/completions` will be honored, and the array of available tools should be provided in the `tools` field. If the model decides that the user message would be best fulfilled with a tool call, an array of tool call request objects will be provided in the response field, `choices[0].message.tool_calls`. The `finish_reason` field of the top-level response object will also be populated with `"tool_calls"`. An example response to the above `curl` request will look like: ```bash { "id": "chatcmpl-gb1t1uqzefudice8ntxd9i", "object": "chat.completion", "created": 1730913210, "model": "lmstudio-community/qwen2.5-7b-instruct", "choices": [ { "index": 0, "logprobs": null, "finish_reason": "tool_calls", "message": { "role": "assistant", "tool_calls": [ { "id": "365174485", "type": "function", "function": { "name": "search_products", "arguments": "{\"query\":\"dell\",\"category\":\"electronics\",\"max_price\":50}" } } ] } } ], "usage": { "prompt_tokens": 263, "completion_tokens": 34, "total_tokens": 297 }, "system_fingerprint": "lmstudio-community/qwen2.5-7b-instruct" } ``` In plain english, the above response can be thought of as the model saying: > "Please call the `search_products` function, with arguments: > > - 'dell' for the `query` parameter, > - 'electronics' for the `category` parameter > - '50' for the `max_price` parameter > > and give me back the results" The `tool_calls` field will need to be parsed to call actual functions/APIs. The below examples demonstrate how. ## Examples using `python` Tool use shines when paired with program languages like python, where you can implement the functions specified in the `tools` field to programmatically call them when the model requests. ### Single-turn example Below is a simple single-turn (model is only called once) example of enabling a model to call a function called `say_hello` that prints a hello greeting to the console: `single-turn-example.py` ```python from openai import OpenAI # Connect to LM Studio client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio") # Define a simple function def say_hello(name: str) -> str: print(f"Hello, {name}!") # Tell the AI about our function tools = [ { "type": "function", "function": { "name": "say_hello", "description": "Says hello to someone", "parameters": { "type": "object", "properties": { "name": { "type": "string", "description": "The person's name" } }, "required": ["name"] } } } ] # Ask the AI to use our function response = client.chat.completions.create( model="lmstudio-community/qwen2.5-7b-instruct", messages=[{"role": "user", "content": "Can you say hello to Bob the Builder?"}], tools=tools ) # Get the name the AI wants to use a tool to say hello to # (Assumes the AI has requested a tool call and that tool call is say_hello) tool_call = response.choices[0].message.tool_calls[0] name = eval(tool_call.function.arguments)["name"] # Actually call the say_hello function say_hello(name) # Prints: Hello, Bob the Builder! ``` Running this script from the console should yield results like: ```xml -> % python single-turn-example.py Hello, Bob the Builder! ``` Play around with the name in ```python messages=[{"role": "user", "content": "Can you say hello to Bob the Builder?"}] ``` to see the model call the `say_hello` function with different names. ### Multi-turn example Now for a slightly more complex example. In this example, we'll: 1. Enable the model to call a `get_delivery_date` function 2. Hand the result of calling that function back to the model, so that it can fulfill the user's request in plain text

multi-turn-example.py (click to expand)

```python from datetime import datetime, timedelta import json import random from openai import OpenAI # Point to the local server client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio") model = "lmstudio-community/qwen2.5-7b-instruct" def get_delivery_date(order_id: str) -> datetime: # Generate a random delivery date between today and 14 days from now # in a real-world scenario, this function would query a database or API today = datetime.now() random_days = random.randint(1, 14) delivery_date = today + timedelta(days=random_days) print( f"\nget_delivery_date function returns delivery date:\n\n{delivery_date}", flush=True, ) return delivery_date tools = [ { "type": "function", "function": { "name": "get_delivery_date", "description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'", "parameters": { "type": "object", "properties": { "order_id": { "type": "string", "description": "The customer's order ID.", }, }, "required": ["order_id"], "additionalProperties": False, }, }, } ] messages = [ { "role": "system", "content": "You are a helpful customer support assistant. Use the supplied tools to assist the user.", }, { "role": "user", "content": "Give me the delivery date and time for order number 1017", }, ] # LM Studio response = client.chat.completions.create( model=model, messages=messages, tools=tools, ) print("\nModel response requesting tool call:\n", flush=True) print(response, flush=True) # Extract the arguments for get_delivery_date # Note this code assumes we have already determined that the model generated a function call. tool_call = response.choices[0].message.tool_calls[0] arguments = json.loads(tool_call.function.arguments) order_id = arguments.get("order_id") # Call the get_delivery_date function with the extracted order_id delivery_date = get_delivery_date(order_id) assistant_tool_call_request_message = { "role": "assistant", "tool_calls": [ { "id": response.choices[0].message.tool_calls[0].id, "type": response.choices[0].message.tool_calls[0].type, "function": response.choices[0].message.tool_calls[0].function, } ], } # Create a message containing the result of the function call function_call_result_message = { "role": "tool", "content": json.dumps( { "order_id": order_id, "delivery_date": delivery_date.strftime("%Y-%m-%d %H:%M:%S"), } ), "tool_call_id": response.choices[0].message.tool_calls[0].id, } # Prepare the chat completion call payload completion_messages_payload = [ messages[0], messages[1], assistant_tool_call_request_message, function_call_result_message, ] # Call the OpenAI API's chat completions endpoint to send the tool call result back to the model # LM Studio response = client.chat.completions.create( model=model, messages=completion_messages_payload, ) print("\nFinal model response with knowledge of the tool call result:\n", flush=True) print(response.choices[0].message.content, flush=True) ```

Running this script from the console should yield results like: ```xml -> % python multi-turn-example.py Model response requesting tool call: ChatCompletion(id='chatcmpl-wwpstqqu94go4hvclqnpwn', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content=None, refusal=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='377278620', function=Function(arguments='{"order_id":"1017"}', name='get_delivery_date'), type='function')]))], created=1730916196, model='lmstudio-community/qwen2.5-7b-instruct', object='chat.completion', service_tier=None, system_fingerprint='lmstudio-community/qwen2.5-7b-instruct', usage=CompletionUsage(completion_tokens=24, prompt_tokens=223, total_tokens=247, completion_tokens_details=None, prompt_tokens_details=None)) get_delivery_date function returns delivery date: 2024-11-19 13:03:17.773298 Final model response with knowledge of the tool call result: Your order number 1017 is scheduled for delivery on November 19, 2024, at 13:03 PM. ``` ### Advanced agent example Building upon the principles above, we can combine LM Studio models with locally defined functions to create an "agent" - a system that pairs a language model with custom functions to understand requests and perform actions beyond basic text generation. The agent in the below example can: 1. Open safe urls in your default browser 2. Check the current time 3. Analyze directories in your file system

agent-chat-example.py (click to expand)

```python import json from urllib.parse import urlparse import webbrowser from datetime import datetime import os from openai import OpenAI # Point to the local server client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio") model = "lmstudio-community/qwen2.5-7b-instruct" def is_valid_url(url: str) -> bool: try: result = urlparse(url) return bool(result.netloc) # Returns True if there's a valid network location except Exception: return False def open_safe_url(url: str) -> dict: # List of allowed domains (expand as needed) SAFE_DOMAINS = { "lmstudio.ai", "github.com", "google.com", "wikipedia.org", "weather.com", "stackoverflow.com", "python.org", "docs.python.org", } try: # Add http:// if no scheme is present if not url.startswith(('http://', 'https://')): url = 'http://' + url # Validate URL format if not is_valid_url(url): return {"status": "error", "message": f"Invalid URL format: {url}"} # Parse the URL and check domain parsed_url = urlparse(url) domain = parsed_url.netloc.lower() base_domain = ".".join(domain.split(".")[-2:]) if base_domain in SAFE_DOMAINS: webbrowser.open(url) return {"status": "success", "message": f"Opened {url} in browser"} else: return { "status": "error", "message": f"Domain {domain} not in allowed list", } except Exception as e: return {"status": "error", "message": str(e)} def get_current_time() -> dict: """Get the current system time with timezone information""" try: current_time = datetime.now() timezone = datetime.now().astimezone().tzinfo formatted_time = current_time.strftime("%Y-%m-%d %H:%M:%S %Z") return { "status": "success", "time": formatted_time, "timezone": str(timezone), "timestamp": current_time.timestamp(), } except Exception as e: return {"status": "error", "message": str(e)} def analyze_directory(path: str = ".") -> dict: """Count and categorize files in a directory""" try: stats = { "total_files": 0, "total_dirs": 0, "file_types": {}, "total_size_bytes": 0, } for entry in os.scandir(path): if entry.is_file(): stats["total_files"] += 1 ext = os.path.splitext(entry.name)[1].lower() or "no_extension" stats["file_types"][ext] = stats["file_types"].get(ext, 0) + 1 stats["total_size_bytes"] += entry.stat().st_size elif entry.is_dir(): stats["total_dirs"] += 1 # Add size of directory contents for root, _, files in os.walk(entry.path): for file in files: try: stats["total_size_bytes"] += os.path.getsize(os.path.join(root, file)) except (OSError, FileNotFoundError): continue return {"status": "success", "stats": stats, "path": os.path.abspath(path)} except Exception as e: return {"status": "error", "message": str(e)} tools = [ { "type": "function", "function": { "name": "open_safe_url", "description": "Open a URL in the browser if it's deemed safe", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The URL to open", }, }, "required": ["url"], }, }, }, { "type": "function", "function": { "name": "get_current_time", "description": "Get the current system time with timezone information", "parameters": { "type": "object", "properties": {}, "required": [], }, }, }, { "type": "function", "function": { "name": "analyze_directory", "description": "Analyze the contents of a directory, counting files and folders", "parameters": { "type": "object", "properties": { "path": { "type": "string", "description": "The directory path to analyze. Defaults to current directory if not specified.", }, }, "required": [], }, }, }, ] def process_tool_calls(response, messages): """Process multiple tool calls and return the final response and updated messages""" # Get all tool calls from the response tool_calls = response.choices[0].message.tool_calls # Create the assistant message with tool calls assistant_tool_call_message = { "role": "assistant", "tool_calls": [ { "id": tool_call.id, "type": tool_call.type, "function": tool_call.function, } for tool_call in tool_calls ], } # Add the assistant's tool call message to the history messages.append(assistant_tool_call_message) # Process each tool call and collect results tool_results = [] for tool_call in tool_calls: # For functions with no arguments, use empty dict arguments = ( json.loads(tool_call.function.arguments) if tool_call.function.arguments.strip() else {} ) # Determine which function to call based on the tool call name if tool_call.function.name == "open_safe_url": result = open_safe_url(arguments["url"]) elif tool_call.function.name == "get_current_time": result = get_current_time() elif tool_call.function.name == "analyze_directory": path = arguments.get("path", ".") result = analyze_directory(path) else: # llm tried to call a function that doesn't exist, skip continue # Add the result message tool_result_message = { "role": "tool", "content": json.dumps(result), "tool_call_id": tool_call.id, } tool_results.append(tool_result_message) messages.append(tool_result_message) # Get the final response final_response = client.chat.completions.create( model=model, messages=messages, ) return final_response def chat(): messages = [ { "role": "system", "content": "You are a helpful assistant that can open safe web links, tell the current time, and analyze directory contents. Use these capabilities whenever they might be helpful.", } ] print( "Assistant: Hello! I can help you open safe web links, tell you the current time, and analyze directory contents. What would you like me to do?" ) print("(Type 'quit' to exit)") while True: # Get user input user_input = input("\nYou: ").strip() # Check for quit command if user_input.lower() == "quit": print("Assistant: Goodbye!") break # Add user message to conversation messages.append({"role": "user", "content": user_input}) try: # Get initial response response = client.chat.completions.create( model=model, messages=messages, tools=tools, ) # Check if the response includes tool calls if response.choices[0].message.tool_calls: # Process all tool calls and get final response final_response = process_tool_calls(response, messages) print("\nAssistant:", final_response.choices[0].message.content) # Add assistant's final response to messages messages.append( { "role": "assistant", "content": final_response.choices[0].message.content, } ) else: # If no tool call, just print the response print("\nAssistant:", response.choices[0].message.content) # Add assistant's response to messages messages.append( { "role": "assistant", "content": response.choices[0].message.content, } ) except Exception as e: print(f"\nAn error occurred: {str(e)}") exit(1) if __name__ == "__main__": chat() ```

Running this script from the console will allow you to chat with the agent: ```xml -> % python agent-example.py Assistant: Hello! I can help you open safe web links, tell you the current time, and analyze directory contents. What would you like me to do? (Type 'quit' to exit) You: What time is it? Assistant: The current time is 14:11:40 (EST) as of November 6, 2024. You: What time is it now? Assistant: The current time is 14:13:59 (EST) as of November 6, 2024. You: Open lmstudio.ai Assistant: The link to lmstudio.ai has been opened in your default web browser. You: What's in my current directory? Assistant: Your current directory at `/Users/matt/project` contains a total of 14 files and 8 directories. Here's the breakdown: - Files without an extension: 3 - `.mjs` files: 2 - `.ts` (TypeScript) files: 3 - Markdown (`md`) file: 1 - JSON files: 4 - TOML file: 1 The total size of these items is 1,566,990,604 bytes. You: Thank you! Assistant: You're welcome! If you have any other questions or need further assistance, feel free to ask. You: ``` ### Streaming When streaming through `/v1/chat/completions` (`stream=true`), tool calls are sent in chunks. Function names and arguments are sent in pieces via `chunk.choices[0].delta.tool_calls.function.name` and `chunk.choices[0].delta.tool_calls.function.arguments`. For example, to call `get_current_weather(location="San Francisco")`, the streamed `ChoiceDeltaToolCall` in each `chunk.choices[0].delta.tool_calls[0]` object will look like: ```py ChoiceDeltaToolCall(index=0, id='814890118', function=ChoiceDeltaToolCallFunction(arguments='', name='get_current_weather'), type='function') ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='{"', name=None), type=None) ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='location', name=None), type=None) ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":"', name=None), type=None) ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='San Francisco', name=None), type=None) ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"}', name=None), type=None) ``` These chunks must be accumulated throughout the stream to form the complete function signature for execution. The below example shows how to create a simple tool-enhanced chatbot through the `/v1/chat/completions` streaming endpoint (`stream=true`).

tool-streaming-chatbot.py (click to expand)

```python from openai import OpenAI import time client = OpenAI(base_url="http://127.0.0.1:1234/v1", api_key="lm-studio") MODEL = "lmstudio-community/qwen2.5-7b-instruct" TIME_TOOL = { "type": "function", "function": { "name": "get_current_time", "description": "Get the current time, only if asked", "parameters": {"type": "object", "properties": {}}, }, } def get_current_time(): return {"time": time.strftime("%H:%M:%S")} def process_stream(stream, add_assistant_label=True): """Handle streaming responses from the API""" collected_text = "" tool_calls = [] first_chunk = True for chunk in stream: delta = chunk.choices[0].delta # Handle regular text output if delta.content: if first_chunk: print() if add_assistant_label: print("Assistant:", end=" ", flush=True) first_chunk = False print(delta.content, end="", flush=True) collected_text += delta.content # Handle tool calls elif delta.tool_calls: for tc in delta.tool_calls: if len(tool_calls) <= tc.index: tool_calls.append({ "id": "", "type": "function", "function": {"name": "", "arguments": ""} }) tool_calls[tc.index] = { "id": (tool_calls[tc.index]["id"] + (tc.id or "")), "type": "function", "function": { "name": (tool_calls[tc.index]["function"]["name"] + (tc.function.name or "")), "arguments": (tool_calls[tc.index]["function"]["arguments"] + (tc.function.arguments or "")) } } return collected_text, tool_calls def chat_loop(): messages = [] print("Assistant: Hi! I am an AI agent empowered with the ability to tell the current time (Type 'quit' to exit)") while True: user_input = input("\nYou: ").strip() if user_input.lower() == "quit": break messages.append({"role": "user", "content": user_input}) # Get initial response response_text, tool_calls = process_stream( client.chat.completions.create( model=MODEL, messages=messages, tools=[TIME_TOOL], stream=True, temperature=0.2 ) ) if not tool_calls: print() text_in_first_response = len(response_text) > 0 if text_in_first_response: messages.append({"role": "assistant", "content": response_text}) # Handle tool calls if any if tool_calls: tool_name = tool_calls[0]["function"]["name"] print() if not text_in_first_response: print("Assistant:", end=" ", flush=True) print(f"**Calling Tool: {tool_name}**") messages.append({"role": "assistant", "tool_calls": tool_calls}) # Execute tool calls for tool_call in tool_calls: if tool_call["function"]["name"] == "get_current_time": result = get_current_time() messages.append({ "role": "tool", "content": str(result), "tool_call_id": tool_call["id"] }) # Get final response after tool execution final_response, _ = process_stream( client.chat.completions.create( model=MODEL, messages=messages, stream=True ), add_assistant_label=False ) if final_response: print() messages.append({"role": "assistant", "content": final_response}) if __name__ == "__main__": chat_loop() ```

You can chat with the bot by running this script from the console: ```xml -> % python tool-streaming-chatbot.py Assistant: Hi! I am an AI agent empowered with the ability to tell the current time (Type 'quit' to exit) You: Tell me a joke, then tell me the current time Assistant: Sure! Here's a light joke for you: Why don't scientists trust atoms? Because they make up everything. Now, let me get the current time for you. **Calling Tool: get_current_time** The current time is 18:49:31. Enjoy your day! You: ``` ## Community Chat with other LM Studio users, discuss LLMs, hardware, and more on the [LM Studio Discord server](https://discord.gg/aPQfnNkxGC).

Completions (Legacy)