Batching
Batching combines multiple inference requests so they can run more efficiently together.
Category
Inference Efficiency
These words describe the techniques that reduce latency, increase throughput, and save compute during serving.
How to recognize this theme
Terms you see when a model serves responses faster.
In a daily board, this category groups terms by their shared role. Look for four cards that describe the same mechanism, risk area, or workflow rather than four words that merely sound similar.
Educational context
These entries are vocabulary notes for learning. They are not project endorsements, token recommendations, exchange rankings, or trading signals.
KV Cache
A KV cache stores key and value tensors so a model can reuse attention history.
Quantization
Quantization reduces numerical precision to make model serving faster and lighter.
Speculative Decoding
Speculative decoding speeds generation by drafting tokens with a smaller model and verifying them with a larger one.