An empirical evaluation of local companion LLM inference. We measure VRAM footprints, tokens per second, and context window scaling across 8B to 70B models.
1. Model Size vs. VRAM Scaling Analysis
Running local companion models offers unparalleled privacy and unfiltered persona expression. However, model parameter scaling introduces severe graphics memory (VRAM) bottlenecks that dictate conversational fluidity.
Our hardware laboratory benchmarked top open-weight models across standard quantization formats to quantify memory footprints and generation speeds.
| Model Parameter Size | Quantization | VRAM Allocation | Throughput (tokens/s) | Time to First Token (TTFT) |
|---|---|---|---|---|
| 8B | FP16 | 16 GB | 48 tokens/s | 180 ms |
| 8B | Q4 | 6 GB | 85 tokens/s | 95 ms |
| 14B | Q4 | 10 GB | 54 tokens/s | 140 ms |
| 32B | Q4 | 20 GB | 32 tokens/s | 260 ms |
| 70B | Q4 | 42 GB | 18 tokens/s | 480 ms |
Experience High-Speed Neural Roleplay
Test high-throughput companion models and deep conversational memory on Candy AI.
2. Context Window Expansion & Latency Budgets
Expanding the conversational context buffer from 16k tokens to 32k tokens is critical for multi-session persona continuity. However, KV cache allocation demands several additional gigabytes of dedicated VRAM.
Empirical conversational studies show that a TTFT under 300 ms and generation speeds above 25 tokens/s are required for sustained conversational immersion. Running 32B models under Q4 quantization currently represents the optimal hardware compromise.