Skip to main content
fanout

Calculations run on this device. Scenario values are not sent anywhere — the engine is fully local.

inference calculator · free

GPU Model Fit

Multiply parameters by precision bytes, add runtime overhead, and see which cards say yes.

Does this checkpoint actually fit on that GPU?

Engine 1.0.0 · calculation
7 B
15 %

Weights fragmentation, activations, CUDA context, runtime.

Weights
14.0 GB
Needed with overhead
16.1 GB
Card memory
24 GB
Verdict
Fits comfortably

Leaves 7.9 GB for the KV cache and activations during serving.

Method

  • Weights = parameters × bytes per element (FP32 4B, FP16 2B, INT8 1B, INT4 0.5B).
  • Adds a configurable runtime overhead for activations, fragmentation, and the CUDA context.
  • Fit verdicts: comfortable under 85% of card memory, tight 85–100%, no fit above.

Sources

Keep going in inference

inference calculatorKV Cache CalculatorWhy do long context and batch size compete for the same GPU?inference calculatorLatency Numbers Every Engineer Should KnowWhich operation dominates this request's budget?inference calculatorQuantization MicroscopeWhat does dropping from 16 to 4 bits actually buy?