Calculations run on this device. Scenario values are not sent anywhere — the engine is fully local.
inference calculator · free
Quantization Microscope
Slide the bit-width and watch weights shrink, decode speed rise, and quality risk creep in.
What does dropping from 16 to 4 bits actually buy?
Engine 1.0.0 · calculation
4 bit
Decode is memory-bandwidth bound, so bytes per weight ≈ speed.
4.0×decode speed headroom vs FP16
- Weights at this width
- 35.0 GB
- Weights at FP16
- 140.0 GB
- Saved
- 75% smaller
- Quality risk
- Minor — standard serving trade
Below 4 bits, perplexity climbs fast for most checkpoints; the safe default for serving is 4–5 bit weight quantization with activations kept higher.
Method
- Weight size scales linearly with bits; decode speed headroom approximates 16/bits under bandwidth-bound decoding.
- Quality-risk bands are heuristic (≥6 negligible, 4–5 standard, 3 evaluate, 2 severe) and point to real evals.