Building a "Jellyfin chatbot": Bridging LLMs and Local Media
In the era of smart homes, the ultimate luxury is a personal assistant that doesn't just tell you the weather, but actually manages your local media library. This Python project demonstrates how to use a local Large Language Model (LLM) like Qwen 2.5 via Llama.cpp to act as an "AI Butler" for a Jellyfin media server. The Concept: Function Calling
The core of this script is a concept called Function Calling (or Tool Use). Instead of the AI just chatting, it is given a set of "tools" (Python functions) that it can choose to execute based on your request. Key Components
Jellyfin API: The bridge that allows the script to search for movies and control playback.
Llama.cpp (Local LLM): The "brain" that interprets user intent. In this case, it’s configured to use the Qwen2.5-7B model.
Function Definitions: Structured JSON schemas that tell the AI exactly what arguments the search_jellyfin and play_media functions require.
Technical Breakdown 1. Media Discovery: search_jellyfin
The search_jellyfin function handles the heavy lifting of querying your server. It supports filtering by genre and sorting by date—allowing the AI to understand phrases like "show me the newest horror movies." Python
params = { "recursive": "true", "limit": 5, "includeItemTypes": media_type, "userId": JELLYFIN_USER_ID, "genres": genre, "sortBy": sort_by, "sortOrder": sort_order }
- Remote Control: play_media
This is where the magic happens. The function first searches for active sessions (like a TV app or a browser tab) that support remote control. If it finds one, it sends a PlayNow command with the specific item_id.
- The Logic Loop: run_query
This function orchestrates the interaction:
The Prompt: You send a request like "Search for comedy series."
The Intent: The LLM analyzes the prompt and returns a tool_calls object instead of a text response.
The Execution: The script sees the tool call, runs the corresponding Python function, and returns the result to you.
Configuration & Setup
To get this running, you need to update the global variables at the top of the script:
Variable Description JELLYFIN_URL The local IP and port of your server (e.g., localhost:8096). JELLYFIN_API_KEY Generated in Jellyfin under Dashboard > API Keys. JELLYFIN_USER_ID Found in the URL when viewing your user profile in the Jellyfin web UI. LLAMACPP The endpoint for your local LLM server.
Here is the code
from dotenv import load_dotenv
import json
import requests
import time
JELLYFIN_URL = os.getenv("JELLYFIN_URL")
JELLYFIN_API_KEY = os.getenv("JELLYFIN_API_KEY")
JELLYFIN_USER_ID = os.getenv("JELLYFIN_USER_ID")
# Qwen2.5-7B-Instruct-Q5_K_M.gguf
LLAMACPP = os.getenv("LLAMACPP_URL")
LLAMACPP = "http://192.168.0.37:5003/v1/chat/completions"
def search_jellyfin(genre=None, media_type="Movie", sort_by=None, sort_order="Descending"):
"""Search for media."""
headers = {
"Authorization": f'MediaBrowser Token="{JELLYFIN_API_KEY}", Client="AI-Agent", Device="Desktop", DeviceId="12345", Version="1.0.0"',
"Accept": "application/json"
}
params = {
"recursive": "true",
"limit": 5,
"includeItemTypes": media_type,
"userId": JELLYFIN_USER_ID,
"genres": genre,
"sortBy": sort_by,
"sortOrder": sort_order
}
params = {k: v for k, v in params.items() if v is not None}
print(f" [Tool Exec] Calling Jellyfin API: Search (Genre: {genre}, Sort: {sort_by})")
try:
response = requests.get(f"{JELLYFIN_URL}/Items", headers=headers, params=params)
return response.json().get("Items", []) if response.status_code == 200 else []
except Exception as e:
return []
def play_media(item_id, item_name):
"""Starts playback of a specific item on the first active session found."""
headers = {"X-MediaBrowser-Token": JELLYFIN_API_KEY}
# 1. Find an active session (e.g., your browser or TV app)
sessions = requests.get(f"{JELLYFIN_URL}/Sessions", headers=headers).json()
active_sessions = [s for s in sessions if s.get("SupportsRemoteControl")]
if not active_sessions:
return f"I found '{item_name}', but no active Jellyfin players are open to play it on."
session_id = active_sessions[0]["Id"]
print(f" [Tool Exec] Sending Play command to session: {active_sessions[0].get('DeviceName')}")
# 2. Send the Play command
play_url = f"{JELLYFIN_URL}/Sessions/{session_id}/Playing"
play_params = {"ItemIds": item_id, "PlayCommand": "PlayNow"}
requests.post(play_url, headers=headers, params=play_params)
return f"Starting playback of '{item_name}' on {active_sessions[0].get('DeviceName')}."
tools = [
{
"type": "function",
"function": {
"name": "search_jellyfin",
"description": "Find movies/series. Set sort_by='DateCreated' for 'new' or 'recent' items.",
"parameters": {
"type": "object",
"properties": {
"genre": {"type": "string", "enum": ["Thriller", "Horror", "Action", "Comedy", "Romance", "War"],"description": "Use 'genre' if the user asks for a specific genre, if not ignore the parameter"},
"media_type": {"type": "string", "enum": ["Movie", "Series"], "default": "Movie"},
"sort_by": {"type": "string", "enum": ["DateCreated", "Name"], "description": "Use 'DateCreated' for new items."},
"sort_order": {"type": "string", "enum": ["Descending", "Ascending"], "default": "Descending"}
}
}
}
},
{
"type": "function",
"function": {
"name": "play_media",
"description": "Play a specific movie by ID.",
"parameters": {
"type": "object",
"properties": {
"item_id": {"type": "string"},
"item_name": {"type": "string"}
},
"required": ["item_id", "item_name"]
}
}
}
]
def run_query(user_prompt):
start_time = time.perf_counter()
payload = {
"model": "qwen-7b",
"messages": [
{"role": "system", "content": "You are a Jellyfin butler. Use tools to find or play media. To find 'new' items, sort by 'DateCreated' Descending."},
{"role": "user", "content": user_prompt}
],
"tools": tools,
"tool_choice": "auto",
"temperature": 0.1
}
print(f"[Agent] Processing request: '{user_prompt}'")
try:
response = requests.post(LLAMACPP, json=payload)
res_json = response.json()
message = res_json["choices"][0]["message"]
if "tool_calls" in message:
tool_call = message["tool_calls"][0]
func_name = tool_call["function"]["name"]
args = json.loads(tool_call["function"]["arguments"])
print(f" [AI Intent] Model decided to call: {func_name}({args})")
if func_name == "search_jellyfin":
results = search_jellyfin(**args)
titles = [f"{item['Name']} (Premiere Date: {item['PremiereDate'][:10]})" for item in results]
output = f"Recently added content: {', '.join(titles)}" if titles else "No recent items found."
elif func_name == "play_media":
output = play_media(**args)
else:
output = message["content"]
end_time = time.perf_counter()
print(f"[Timer] Request completed in {end_time - start_time:.2f} seconds.")
return output
except Exception as e:
return f"Error: {e}"
if __name__ == "__main__":
print(f"Reponse: {run_query('Search for horror movies')}\n")
print(f"AI: {run_query('Play the movie The Mist')}")
Here are some example responses. Note the responses are not generated by the model, only the functions calls to get the results were.
(venv)...> py .\app.py
[Agent] Processing request: 'Search for horror movies'
[AI Intent] Model decided to call: search_jellyfin({'genre': 'Horror', 'media_type': 'Movie', 'sort_by': 'DateCreated', 'sort_order': 'Descending'})
[Tool Exec] Calling Jellyfin API: Search (Genre: Horror, Sort: DateCreated)
[Timer] Request completed in 9.10 seconds.
Reponse: Recently added content: The Mist (Premiere Date: 2007-11-21), Black Phone 2 (Premiere Date: 2025-10-15), The Long Walk (Premiere Date: 2025-09-10), The Conjuring: Last Rites (Premiere Date: 2025-09-03), Devon (Premiere Date: 2024-11-12)
(venv)...> py .\app.py
[Agent] Processing request: 'Search for action series'
[AI Intent] Model decided to call: search_jellyfin({'genre': 'Action', 'media_type': 'Series', 'sort_by': 'DateCreated', 'sort_order': 'Descending'})
[Tool Exec] Calling Jellyfin API: Search (Genre: Action, Sort: DateCreated)
[Timer] Request completed in 9.56 seconds.
Reponse: No recent items found.
(venv)...> py .\app.py
[Agent] Processing request: 'Search for comedy series'
[AI Intent] Model decided to call: search_jellyfin({'genre': 'Comedy', 'media_type': 'Series', 'sort_by': 'DateCreated', 'sort_order': 'Descending'})
[Tool Exec] Calling Jellyfin API: Search (Genre: Comedy, Sort: DateCreated)
[Timer] Request completed in 9.58 seconds.
Reponse: Recently added content: Slow Horses (Premiere Date: 2022-04-01)