A local model needs memory for its weights, for a KV cache that grows with context and parallel slots, and for runtime overhead. Here are the formulas, three worked...
Categories
