Batch Size
Batch size is the number of requests or tokens processed together in one serving step.
Category
Compute Serving Metrics
These terms describe how systems batch requests, measure speed, and stay within capacity.
How to recognize this theme
Words for performance and capacity in model serving.
In a daily board, this category groups terms by their shared role. Look for four cards that describe the same mechanism, risk area, or workflow rather than four words that merely sound similar.
Educational context
These entries are vocabulary notes for learning. They are not project endorsements, token recommendations, exchange rankings, or trading signals.
Inference Latency
Inference latency is the time between sending a prompt and receiving a model's output.
Throughput
Throughput is the amount of model work a serving system can complete over a given period.
Utilization Rate
Utilization rate is the portion of available compute capacity that is actively being used.