Monitoring

Logs

The server writes to standard error, which the systemd journal collects with the time:

journalctl -u ishtaria-server -f

By default (access_log = "errors") a line is written for every answer with status 400 or more and for every request slower than 500 ms; "all" writes every request and "off" none. A line looks like this and never contains a query string, a token or a password:

access ip=203.0.113.5 method=POST path=/players/me/cast status=429 ms=1
access ip=203.0.113.5 method=POST path=/players status=429 limited=true

Metrics

GET /metrics gives counters and gauges in the Prometheus text format. From the server machine:

curl -s http://127.0.0.1:7400/metrics

With a metrics_token a remote scraper may read it (Authorization: Bearer ...). A Prometheus job:

scrape_configs:
  - job_name: ishtaria
    metrics_path: /metrics
    authorization: { type: Bearer, credentials: long-random }
    static_configs: [{ targets: ["world.example.org:443"] }]
    scheme: https

Metric	Meaning
`ishtaria_http_requests_total{class}`	answers by status class (2xx, 3xx, 4xx, 5xx)
`ishtaria_http_requests_by_group_total{group}`	requests by area: players, world, chat, shops, trades, placed, story, …
`ishtaria_http_rate_limited_total`	requests refused by a limit
`ishtaria_http_timeouts_total`	requests that ran into the 30 s limit
`ishtaria_http_request_duration_ms_sum` / `_count`	total time and count (their ratio is the mean)
`ishtaria_players_online` / `ishtaria_players_alive`	living characters active in the last 30 s / all living characters
`ishtaria_db_pool_connections{state}`	open and idle connections of the database pool (at most 10)
`ishtaria_uptime_seconds`, `ishtaria_build_info{version}`	process age and version

Things worth an alert: 5xx growing, a pool that stays at 10 open and 0 idle, rate_limited jumping (an attack, or a limit that is too small for a school), timeouts above zero, and no data at all. GET /health is a cheap liveness check (it asks the database).

Capacity: what one machine did

tools/loadtest.py of ishtaria-server simulates players who register and then walk, look around, poll events, chat and ask for their profile. A measurement of 2026-10-07 with a release build of version 0.2.1, PostgreSQL on the same machine and the load generator on the same machine too (a desktop with a consumer GPU; so the real server would have the CPU for itself):

100 simulated players, 30 s, one request every 100 ms each	result
throughput	about 940 requests per second, no `5xx`, no timeouts
walking (`move`), 15 700 requests	mean 4.5 ms, p95 10 ms, p99 17 ms
looking around (`nearby`), events, profile	p95 between 2 and 9 ms
registration of 100 players at once	mean 0.7 s, p95 1.3 s (passwords are hashed two at a time, on purpose)
process	about 200 MB of memory

These numbers say that a small world has plenty of headroom; they do not say how a world with thousands of players behaves, how the network of real players looks, or how the server performs with a slow database. A real client sends about ten requests per second while walking and fewer when standing. Measure your own setup before an event: raise the limits for the test, run the tool from another machine, and watch /metrics while it runs. Debug builds are several times slower, notably in password hashing.