chat-agent

The chat-agent module manages chat sessions and LLM interaction. It sits between the FastAPI server and agentlib — the server calls into chat-agent, which calls agentlib for all retrieval and graph operations.

Module layout

chat-agent/
├── agent_handler.py    # Agentic loop: tool-calling LLM, up to 32 iterations
├── lookup_handler.py   # Direct vector search with formatted Markdown output
├── session.py          # Chat history persistence (Valkey)
├── router.py           # Intent classifier: lookup / agent
├── config.py           # LLM provider settings (Pydantic, no env prefix)
├── tools/
│   ├── __init__.py         # TOOLS schemas (8 tools) + TOOL_MAP
│   ├── graph_overview.py   # tool_get_graph_overview()
│   ├── vector_search.py    # tool_vector_search()
│   ├── query_graph.py      # tool_graph_query()
│   └── schema_tools.py     # five schema exploration tools
└── schemas/
    └── chat.py             # RouteType, ChatSession, Message, ChatRequest, ChatResponse, TokenUsage

Routing and dispatch

Every incoming message is classified by router.py before being dispatched to a handler.

POST /chat/message
     │
     ▼
classify_intent()          ← router.py
     │ route + confidence
     ├── lookup  → lookup_handler.run_lookup()
     └── agent   → agent_handler.run_agent()
     │
     ▼
ChatResponse { session_id, answer, route_taken, sources, token_usage }

Route definitions

Route	Intent	Handler
`lookup`	Specific fact, entity, or definition — retrieve and return directly	`lookup_handler`
`agent`	Everything else: explanations, comparisons, multi-step reasoning, graph traversal	`agent_handler`

The router prompt instructs the LLM to use lookup only for narrow data-retrieval requests ("show me the node for BRCA2", "list all pathway nodes"). Anything involving a written answer, explanation, or reasoning defaults to agent.

Confidence escalation

If the classifier picks lookup but its confidence falls below rag_confidence_threshold (default 0.75), the route is automatically escalated to agent:

lookup  (confidence < 0.75) → agent

On parse error (malformed LLM response), the router defaults to agent.

Handlers

`lookup_handler.run_lookup()`

The fastest path. Embeds the question, runs a vector search, and formats the top-k results as Markdown — no LLM call.

message
  → embed (agentlib/rag/embeddings)
  → vector_search (agentlib/rag/index)
  → format as Markdown (### name, node ID, labels, text)
  → return { answer: markdown_string, sources: list[dict] }

`agent_handler.run_agent()`

Agentic loop with full tool access. The LLM decides which tools to call, receives results, and iterates until it produces a final answer or hits the iteration cap (32).

message + skeleton
  → LLM with 8 TOOLS (see tools reference)
  ├── if tool_calls:
  │     execute tools → append results to messages → repeat
  └── if no tool_calls:
        return { answer: string, sources: list[dict], token_usage: TokenUsage }

Key behaviours:

The graph skeleton (generated at startup) is injected into the system prompt when available. It gives the LLM a compact structural overview of the graph before any tool calls.
Schema tools (get_node_labels, get_edge_types, etc.) let the LLM iteratively explore the graph structure before writing a Cypher query.
TuringDB Cypher deviations are listed in the system prompt so the LLM writes valid queries on the first attempt.
Rate-limit retry: if the LLM API returns HTTP 429, the handler retries up to 3 times with a 2-second back-off.
Tool errors are caught per-call and returned as "Tool error: ..." strings so the loop can continue.
Sources accumulate from all vector_search calls during the loop.
Token usage (router + handler) is summed and returned in ChatResponse.token_usage.

LLM configuration

LLM settings live in chat-agent/src/chat_agent/config.py (no TURINGDB_ prefix — these are bare env vars).

Variable	Default	Description
`LLM_PROVIDER`	`ollama`	Agent LLM provider: `ollama` or `mistral`
`ROUTER_PROVIDER`	(unset)	Router LLM provider override; inherits `LLM_PROVIDER` if unset
`OLLAMA_BASE_URL`	`http://localhost:11434/v1`	Ollama API base URL
`OLLAMA_AGENT_MODEL`	`qwen2.5:7b`	Ollama model for the agent
`OLLAMA_ROUTER_MODEL`	`qwen2.5:3b`	Ollama model for the intent router
`MISTRAL_API_KEY`	(empty)	Mistral API key (required if `LLM_PROVIDER=mistral`)
`MISTRAL_AGENT_MODEL`	`mistral-large-latest`	Mistral model for the agent
`MISTRAL_ROUTER_MODEL`	`mistral-small-latest`	Mistral model for the intent router
`RAG_CONFIDENCE_THRESHOLD`	`0.75`	Min confidence for `lookup` route; below this, escalates to `agent`
`VECTOR_SEARCH_TOP_K`	`5`	Default top-k for vector search tool calls
`TURINGDB_URL`	`localhost`	TuringDB host (used by tools)
`TURINGDB_PORT`	`1234`	TuringDB port (used by tools)
`TURINGDB_GRAPH`	`reactome`	Graph name
`EMBEDDING_MODEL`	`minishlab/potion-base-32M`	Model used for embedding queries
`VECTOR_INDEX_NAME`	`reactome_minishlab_potion_base_32M`	Index name in TuringDB

Tip: ROUTER_PROVIDER lets you run a fast local Ollama model for routing while using Mistral for the agent. Set LLM_PROVIDER=mistral ROUTER_PROVIDER=ollama to split the two.

Session management

Chat history is persisted in Valkey under chat:session:{uuid}.

Function	Description
`get_or_create_session(session_id, redis, ttl)`	Loads from Valkey or creates a new session
`save_session(session, redis, ttl)`	Serialises and writes back with TTL

Sessions are keyed by a UUID generated on first creation. The UUID is returned in every ChatResponse and sent back by the client on subsequent requests.

`ChatSession`

class ChatSession(BaseModel):
    session_id: str                   # UUID (auto-generated)
    messages: list[Message]
    pending_job_id: str | None        # job currently processing this session
    pending_events: list[JobEvent]    # events emitted during current job (for SSE)
    name: str | None                  # user-defined display name (set via PATCH /chat/session/{id})
    created_at: float | None          # Unix timestamp of session creation

    def add(role, content, route?, confidence?) -> None
    def add_event(type, data?) -> None      # append event to pending_events
    def history_for_llm(max_turns=10) -> list[dict]

history_for_llm trims to the last max_turns message pairs before passing history to the LLM, keeping prompt size bounded.

`Message`

class Message(BaseModel):
    role: str                  # "user" | "assistant"
    content: str
    route: RouteType | None    # route taken for this assistant message
    confidence: float | None   # router confidence score
    events: list[JobEvent]     # agent step events captured during this message

Data flow

POST /chat/message  {message, session_id?}
        │
        ▼
 rate-limit check (X-Anon-Id header → user_id)
        │
        ▼
 load or create ChatSession (Valkey)
 enqueue JobRecord → chat:job_queue (Valkey)
        │
        ▼
 ← returns 202 JobSubmission {job_id, session_id, status: "pending"}

─── background worker (BLPOP chat:job_queue) ────────────────────────

        ▼
  classify_intent()  ──►  router LLM   [emits routing_started / routing_done events]
        │ route + confidence
        │
        ├─ lookup ──► embed question
        │               │
        │               ▼
        │             vector_search (agentlib/rag/index.py)
        │               │
        │               ▼
        │             format as Markdown
        │
        └─ agent  ──► agentic loop (max 32 iterations)        [emits agent_step / tool_call / tool_result events]
                          │
                          ├─ vector_search (tool)  ──► agentlib/rag
                          ├─ graph_query   (tool)  ──► agentlib/graph/query.py
                          └─ schema tools  (tools) ──► agentlib/graph/schema.py
        │
        ▼
 session.add(assistant message + route + confidence + events)
 save_session (Valkey, TTL 7 days)
        │
        ▼
 update JobRecord → status: "done", result: ChatResponse

Schemas

`ChatRequest`

class ChatRequest(BaseModel):
    session_id: str | None   # omit to start a new session
    message: str

`ChatResponse`

class ChatResponse(BaseModel):
    session_id: str
    answer: str
    route_taken: RouteType          # "lookup" | "agent"
    sources: list[Any]              # node dicts from vector_search calls
    token_usage: TokenUsage | None  # combined router + handler token counts

`TokenUsage`

class TokenUsage(BaseModel):
    input_tokens: int
    output_tokens: int

Supports + operator — router and handler usages are summed.

`RouterDecision`

class RouterDecision(BaseModel):
    route: RouteType
    confidence: float
    reasoning: str
    token_usage: TokenUsage | None

`JobEvent`

class JobEvent(BaseModel):
    type: str          # e.g. "routing_done", "agent_step", "tool_call"
    data: dict[str, Any]

`JobStatus`

class JobStatus(str, Enum):
    PENDING = "pending"
    RUNNING = "running"
    DONE    = "done"
    ERROR   = "error"

`JobRecord`

class JobRecord(BaseModel):
    job_id: str
    session_id: str
    user_id: str | None
    is_new_session: bool
    message: str
    status: JobStatus
    created_at: float
    result: ChatResponse | None
    error: str | None

`JobSubmission`

class JobSubmission(BaseModel):
    job_id: str
    session_id: str
    status: JobStatus   # always "pending" at submission time