Running LLMs on Linux Virtual Machines Without a GPU: A Sizing Guide

For single-stream CPU inference, token generation is usually dominated by memory bandwidth rather than raw compute. Once you internalise that, sizing cloud linux virtual machines for LLM inference becomes arithmetic you can do before provisioning. Prompt processing and concurrent serving shift the bottleneck back toward compute and parallelism, so treat those as separate problems.

This guide covers the sizing estimate, the RAM maths, the quantisation trade-off, the CPU characteristics that matter, and an honest list of workloads that do not belong on a CPU. It applies whether you provision a single virtual private server or a fleet. Figures reflect documentation and community benchmarks current as of August 2026.

Why CPU Inference Became Viable

Two developments changed the picture. The llama.cpp project produced a C and C++ inference engine tuned for CPU SIMD instructions, and the GGUF format packages quantised weights so they can be memory-mapped and loaded almost instantly rather than deserialised through Python.

Quantisation is the other half. Reducing weight precision from 16-bit floating point to four or five bits cuts memory footprint substantially with modest quality loss. K-quant formats such as Q4_K_M, Q5_K_M, Q6_K and Q3_K_M offer different quality, memory and throughput trade-offs compared with the older Q4_0 and Q4_1 formats.

The First-Order Sizing Estimate

For bandwidth-bound single-stream decode, each token requires reading the active model weights from memory, which gives a useful upper bound.

tokens per second ≈ effective memory bandwidth (GB/s) ÷ model weight size (GB)

Two cautions. Use effective bandwidth rather than the platform’s theoretical figure, because usable bandwidth during inference is lower. And treat the result as a ceiling to validate against a real benchmark, not a prediction. Community-reported observations give a sense of the gap.

A 7B model at Q4_K_M occupies roughly 4.5 GB. A dual-channel DDR4 system in the region of 50 GB/s puts the ceiling near 11 tokens per second. In a llama.cpp GitHub discussion, a contributor reported 11 tokens per second for a 7B Q4_K_M model on a Ryzen 5700G with DDR4.
A 70B model at Q4_K_M occupies roughly 38 GB. The same class of system puts the ceiling near 1.3 tokens per second, and the same contributor reported about 0.8 tokens per second on a 64 GB DDR4 machine.

These are individual community reports rather than controlled benchmarks, so treat them as directional. What they illustrate is the shape of the relationship. The same contributor measured 0.75 tokens per second for a 7B model on a Raspberry Pi 4 with 8 GB, which has four cores but very limited memory bandwidth. Adding vCPUs to a bandwidth-starved instance changes little.

One important exception. Prompt processing, meaning the initial read of your input, is compute-bound rather than bandwidth-bound and does benefit from more cores. Generation is where bandwidth rules. When you benchmark a linux virtual machine for this workload, measure both separately.

Sizing RAM Properly

Three components consume memory, and most sizing exercises count only the first.

Model weights. As a rough planning heuristic, Q4-class GGUF weights run around 0.5 to 0.6 GB per billion parameters, but always size from the actual model file rather than parameter count. Real file size depends on architecture, vocabulary, embedding and output layer treatment, and the exact quantisation variant.
KV cache. Every generated token stores a key and value vector. Size depends on layer count, KV head configuration, head dimension, context length, cache precision and whether the runtime quantises the cache, so two models with the same parameter count can differ substantially. Use the runtime’s reported cache usage rather than a fixed per-token figure. As an illustration of scale, published examples put a 7B-class model in the region of 80 to 120 MB per 1,000 tokens, with larger models several times higher.
Runtime overhead. Allocator behaviour, the operating system and your application need headroom. One published benchmark found a 27B model shipping as a 15.6 GB file requiring over 20 GB at 64K context.

A practical rule: take the actual model file size, add your measured KV cache for the maximum context you will serve, then add 20 to 25 percent. If the working set exceeds available RAM and the system begins paging or swapping, performance can collapse rather than degrade gracefully. This is the most common sizing failure on cloud linux virtual machines, and it is avoidable with five minutes of arithmetic.

Choosing a Quantisation Level

Q4_K_M is the common default, balancing memory savings against quality loss.
Q5_K_M and Q6_K improve output quality at higher memory cost and lower throughput on the same bandwidth.
Q8_0 approaches full precision and is rarely worth the bandwidth penalty on CPU.
Q3_K_M is the aggressive option where memory is tight.

One caveat worth knowing, because it is counterintuitive. Smaller is not automatically faster. A smaller file does reduce bytes read per token, but decode complexity matters too, and llama.cpp contributors have noted that Q3_K can run slower than Q4_K despite being smaller, because its block structure does not align as neatly with the data sizes processors handle efficiently. Benchmark the specific quantisation rather than assuming the ranking.

Which CPU Characteristics Actually Matter

Memory bandwidth first. DDR5 generally outperforms DDR4, and channel count matters as much as clock speed. This is the most predictive specification for generation throughput.
SIMD instruction support. AVX2 is widely supported and useful. AVX-512 can deliver substantial gains on supported processors, and Intel AMX can accelerate relevant matrix operations, but only where the inference runtime and its kernels actually target those instructions. Check what your build supports.
Cores, with diminishing returns. Useful for prompt processing and concurrent requests. Beyond the point where memory saturates, additional threads add contention rather than throughput.
NUMA topology. On multi-socket instances, a model spanning sockets pays a cross-node memory penalty. Pin the process where you can.

When comparing linux virtual machine hosting providers for this workload, ask about the underlying memory generation and channel configuration, not just vCPU count and RAM total. Most rate cards do not publish it, which is precisely why a short benchmark on hourly billing is worth more than any specification sheet.

Workloads That Fit, and Workloads That Do Not

Well suited to CPU inference

Asynchronous and batch jobs: document summarisation, classification, tagging, extraction and enrichment pipelines where latency is measured in seconds.
Retrieval augmented generation where embedding and retrieval do most of the work and generation is short.
Small models in the 3B to 8B range serving internal tools, which are generally the easiest CPU targets.
Privacy-sensitive inference on data that should not leave your infrastructure.
Development, prototyping and evaluation before committing to accelerated hardware, which is where cloud VPS hosting in India earns its place on cost alone.
Low and predictable request volumes, where an idle GPU is pure waste.

Not suited to CPU inference

Interactive chat at conversational speed, which becomes progressively harder as model size and concurrency rise, particularly beyond the low teens of billions of parameters.
High concurrency serving, where GPU batching advantages are decisive.
Any form of training or fine-tuning.
Very long context workloads, where KV cache growth overwhelms both memory and bandwidth.
Latency-critical user-facing generation with strict response time targets.

One further honest point. Community discussion on the llama.cpp repository notes that CPU inference is considerably less power-efficient per token than GPU inference, so at sustained high utilisation a GPU can win on running cost even where a CPU instance is cheaper per hour. Utilisation decides this, not the hourly rate.

Sizing Heuristics by Model Class

Approximate starting points only. Context length, KV cache, concurrency and runtime overhead can raise these materially, so verify against the actual model file and a real benchmark.

3B class, around 2 GB of Q4 weights. 8 GB instance. Classification, extraction, tagging, routing.
7B class, around 4 to 5 GB. 8 to 16 GB instance. RAG, internal assistants, summarisation.
13B class, around 8 GB. 16 to 32 GB instance. Higher-quality generation where latency is flexible.
30B class, around 18 to 20 GB. 32 GB or more. Selective batch workloads only.
70B class, around 38 to 40 GB. 64 GB or more. Batch and experimental use, given low token rates.

Provisioning the Instance

Start with a memory-optimised instance carrying enough RAM for weights plus context plus headroom, running a standard Linux distribution with llama.cpp or Ollama. Providers market these as virtual machines or as a virtual private server, and while the terms overlap the underlying virtualisation model can differ, so confirm you are getting dedicated memory rather than an oversubscribed slice.

Neon Cloud fits this pattern well. It offers India-hosted memory-optimised virtual machines with full root access, hourly as well as monthly billing, a published 99.95 percent uptime SLA, and data centres in Delhi NCR and Mumbai. Cloud firewalls and virtual private cloud are included rather than metered, and management is available as an opt-in layer per virtual machine that retains root access.

Two of those details matter specifically here. Hourly billing means you can provision, benchmark a real model against your real prompts, and release the instance the same afternoon, which is the only reliable way to size this workload. And domestic hosting can help keep primary inference workloads within India, subject to your application architecture, backups, telemetry and third-party dependencies. For that combination, cloud VPS hosting in India is frequently a better match than a distant GPU instance billed in dollars.

Benchmark Before You Commit

Provision the candidate instance on hourly billing.
Install llama.cpp or Ollama and pull the exact model and quantisation you intend to run.
Measure prompt processing and token generation separately, since they scale on different resources.
Test at the context length you will genuinely use, not the default.
Run concurrent requests if that reflects your production pattern.
Confirm total memory use under load, including KV cache at maximum context. Repeat across candidate linux virtual machine hosting providers if you are comparing more than one.
Compare against your latency requirement, then resize or release.

Conclusion

Sizing CPU inference is straightforward once you stop thinking in vCPUs. Divide effective memory bandwidth by model weight size for a ceiling, add weights plus measured KV cache plus a quarter for RAM, pick a quantisation and benchmark it rather than assuming, then verify on the actual instance. Well-specified cloud linux virtual machines handle asynchronous, batch, retrieval and small-model workloads at a fraction of GPU cost, and handle interactive large-model chat poorly. Knowing which side your workload falls on is the whole exercise.Frequently Asked Questions

Can you run an LLM without a GPU?

Yes, using llama.cpp or Ollama with quantised GGUF models. Small models in the 3B to 8B range run usably on modern server CPUs, with reported throughput varying widely by processor, memory configuration and quantisation. Generation speed is dominated by memory bandwidth rather than processor clock speed.

How much RAM do I need to run a 7B model?

Roughly 4.5 GB for Q4-class weights, plus KV cache for your context length, plus 20 to 25 percent overhead. An 8 GB instance is workable for short contexts and 16 GB is more comfortable for production. Size from the actual model file, not the parameter count.

How many vCPUs do I need for LLM inference?

Fewer than most people expect. Token generation is bandwidth-bound, so extra cores add little once memory saturates. Cores help with prompt processing and concurrent requests. Prioritise memory bandwidth and channel configuration, then SIMD support your runtime actually uses.

Which quantisation should I use for CPU inference?

Q4_K_M is a reasonable default. Do not assume smaller is faster, since decode complexity varies by format and contributors have observed Q3_K running slower than Q4_K despite the smaller file. Benchmark your candidate formats on the target instance.

Is CPU inference cheaper than renting a GPU?

Often for low, predictable or bursty volumes, where an idle GPU is wasted spend. GPUs win on high concurrency, large models, interactive latency and power efficiency per token. Model your actual request pattern rather than comparing hourly rates, because utilisation drives the answer.