Batching
Batching combines multiple requests so a model can process them more efficiently.
Category
Inference Ops
These terms describe the main levers for lowering latency and using GPU memory more efficiently.
How to recognize this theme
Terms that improve throughput when serving AI models.
In a daily board, this category groups terms by their shared role. Look for four cards that describe the same mechanism, risk area, or workflow rather than four words that merely sound similar.
Educational context
These entries are vocabulary notes for learning. They are not project endorsements, token recommendations, exchange rankings, or trading signals.
KV Cache
A KV cache stores past attention keys and values so generation can continue faster.
Quantization
Quantization reduces numerical precision so models use less memory and often run faster.
Speculative Decoding
Speculative decoding uses a smaller draft model to propose tokens before verification.
Batching
Batching combines multiple inference requests so they can run more efficiently together.
KV Cache
A KV cache stores key and value tensors so a model can reuse attention history.
Quantization
Quantization reduces numerical precision to make model serving faster and lighter.
Throughput
Throughput measures how many requests or tokens a serving system can process over time.