Latency
Latency is the delay between a request and the start or completion of a response.
Category
Model Serving Economics
These terms cover delay, work rate, grouped requests, and reduced precision.
How to recognize this theme
Terms that show up when teams optimize speed, cost, and throughput.
In a daily board, this category groups terms by their shared role. Look for four cards that describe the same mechanism, risk area, or workflow rather than four words that merely sound similar.
Educational context
These entries are vocabulary notes for learning. They are not project endorsements, token recommendations, exchange rankings, or trading signals.
Throughput
Throughput is the amount of work a system can process in a given time period.
Batching
Batching groups multiple inputs or requests together to improve efficiency.
Quantization
Quantization reduces numerical precision to lower memory use and speed up inference.
KV Cache
A KV cache is saved key-value attention state reused to speed up token generation.
Batch Scheduling
Batch scheduling groups requests so a server can process them efficiently.
Quantization
Quantization reduces numeric precision to make a model smaller and faster.
Inference Latency
Inference latency is the time it takes a model to produce an answer.