Latency
Latency is the delay between a request and the first useful response.
Category
Serving Efficiency
These terms cover delay, work rate, request grouping, and reused results.
How to recognize this theme
Terms that describe how model systems stay fast under load.
In a daily board, this category groups terms by their shared role. Look for four cards that describe the same mechanism, risk area, or workflow rather than four words that merely sound similar.
Educational context
These entries are vocabulary notes for learning. They are not project endorsements, token recommendations, exchange rankings, or trading signals.
Throughput
Throughput is the amount of work a system can process per unit time.
Batching
Batching groups multiple requests together to improve efficiency.
Caching
Caching stores reusable results so the system does not recompute them every time.
Batching
Batching combines multiple requests so a model can process them more efficiently.
Prefill
Prefill is the inference stage that processes the input prompt before generation starts.
KV Cache
A KV cache stores past attention keys and values so generation can continue faster.
Quantization
Quantization reduces numerical precision so models use less memory and often run faster.