Valkey rate limiting

Two independent sliding-window counters are enforced on every POST /chat/message request, both keyed on user_id. If either limit is exceeded the server returns HTTP 429.

Relevant file: server/rate_limit.py, server/routes/chat.py

Configuration

Relevant file: server/config.py

Key schema

TTL is refreshed on every write so the key expires automatically after a full window of inactivity.

Request-count limiter — rate_limit:{user_id}

Algorithm: sliding window using a Valkey list of Unix timestamps.

check_rate_limit(user_id):
  entries  = LRANGE rate_limit:{user_id} 0 -1
  recent   = count entries where float(e) > (now - window_seconds)
  if recent >= max_requests → return False  (reject)

  LPUSH  rate_limit:{user_id}  now
  EXPIRE rate_limit:{user_id}  window_seconds
  LTRIM  rate_limit:{user_id}  0  (max_requests - 1)   # cap list size
  return True  (allow)

The list is trimmed to max_requests entries so it never grows unbounded.

Token-budget limiter — token_limit:{user_id}

Algorithm: sliding window using a Valkey sorted set. Each member encodes {timestamp}:{token_count}:{uuid} with the timestamp as its score, enabling efficient range-based removal of expired entries.

Checking the budget (before the request)

check_token_budget(user_id):
  ZREMRANGEBYSCORE token_limit:{user_id}  -inf  (now - window_seconds)
  entries = ZRANGE token_limit:{user_id} 0 -1
  used    = sum(int(e.split(":")[1]) for e in entries)
  return used < max_tokens

Recording usage (after the request)

Token counts for the intent router and the selected handler are summed and recorded as a single entry:

record_token_usage(user_id, tokens):
  member = f"{now}:{tokens}:{uuid4().hex}"
  ZADD   token_limit:{user_id}  {member: now}
  EXPIRE token_limit:{user_id}  window_seconds

The uuid4().hex suffix guarantees uniqueness when two requests finish in the same millisecond (sorted-set members must be unique).

Error responses

Known limitations and planned work