One server, two cache hit rates
Left, the cache_hit_rate gauge, overwritten on each log interval, reads 0.942 for the last prefill interval. Right, the device_hit counter at 182,079 tokens and the input counter at 104,057 tokens give 182,079 divided by 286,136, which is 0.636 over the whole run.
A · THE GAUGE
B · THE COUNTERS
sglang:cache_hit_rate
gauge, overwritten each log interval
0.942
the last prefill interval only
device_hit
182,079 tokens
input
104,057 tokens
182,079 / 286,136 = 0.636
all prefill since the server started