Sizing VRAM to Generative AI & LLM Workloads

Introduction

Generative AI has come a long way in the last few years – and even that is probably a vast understatement. Many people now benefit from GenAI on a daily basis, even if they don’t realize it. A few examples of this include Google’s first answer to most search queries being AI-based, Amazon using AI to summarize customer reviews, and AI-based image and video editing tools becoming more widespread – though none of these are without controversy.

The largest and most powerful generative models are cloud-based, trained and running across massive server clusters with thousands of graphics processing units (GPUs – colloquially called video cards, but now capable of so much more than just generating computer graphics and video output). That level of hardware allows quick responses to the huge number of queries coming in at the same time, but has astronomical purchasing and energy costs.

However, it is entirely possible to run many forms of generative AI, especially large language models (LLMs), with more modest hardware. There are three phases to using genAI:

1. Training a model is the most processing-intensive and thus needs extremely powerful hardware. This is almost exclusively done with powerful server clusters.
2. Fine-tuning an existing model is still demanding but can be done without needing a full-blown supercomputer.
3. Inference, the term for normal use of a model to generate content, is the least taxing on hardware and the step that most users are looking to perform.

Inference with some small variants of LLMs – yes, I know that sounds contradictory – can be run on a PC with a single GPU. Most reasonably strong models will need a bit more processing power, though, and a number of guides and online tools have been created to help users understand how much hardware they will need for either inference or fine-tuning of various LLMs. This article will not attempt to reinvent that wheel, but rather provide links to some of those resources and then guidance on how to take that information and use it to purchase or build the right type of computer.

LLM Resources

VRAM Guide

First up, here is a guide from Wei-Ming Thor explaining the math behind calculating the amount of VRAM (the memory on GPUs) that LLMs need. If you want to understand what factors go into determining VRAM requirements, this is a great place to start!

VRAM Calculator

Next, we have a web-based LLM calculator from ApX. It will help make sure you get enough VRAM, using the methods described above, and can also estimate the number of tokens per second that a particular set of GPUs can provide with a given LLM configuration. The default setting is for inference, but there is a tab where you can switch to fine-tuning if you want – which significantly increases the hardware required!

I find that using a calculator like this is most intuitive in one of two ways:

• If you know exactly what sort of model you want to run, plug in all of those parameters and see how much VRAM it will need. Then, adjust the GPU selection and quantity until you reach that total – with some headroom, to be safe.
• If you are unsure of the model, but know what GPUs you have available (or have the budget for) then start by selecting the version and quantity of cards and then play around with the various model options until you find a configuration that will work – again, with some headroom.

One small aside about this tool: it doesn’t always have the latest GPUs listed, but there is a “Custom” option which lets you manually plug in the VRAM per card.

Hosting Software

Experienced users may want to manually configure their LLM software setup, but if you are just getting started or want a simple interface to use then check out LM Studio. It provides a graphical user interface (GUI) for running a range of supported models inside Windows, MacOS, and Linux… and it’s free!

While not nearly as polished as LM Studio, Stability Matrix provides a GUI for installing and running image generation models. It also supports all three major OSes, and is free to use, but it is worth noting that there is a lot of controversy over how image-generating models are trained.

There are many other great tools and articles out there – these are just a sample! If there is a resource you’ve found particularly helpful, please feel free to share a link in the comments.

Hosting your own LLM

Once you’ve used a calculator, like the one linked above, to determine how many of what model GPUs you need, it is time to build your system. Here is a general breakdown of how many GPUs you can fit in different classes of systems:

For example, let’s say a business wanted to stand up a DeepSeek-R1 70B model for an internal team which needs to be able to ask questions that won’t end up being recorded on a cloud service, so they need an on-prem system. They’ve got a server closet with some space available, and would like the following parameters: Q8 inference precision, FP16 KV cache precision, 8K max sequence length, and up to 8 concurrent users. Plugging those values into the calculator, we get a result of about 125GB of VRAM being needed.

A popular choice for running LLMs in a rackmount system is NVIDIA’s L40S, which has 48GB of VRAM per card. Selecting that in the calculator shows that a minimum of three (3) cards should be able to run this model, but it would be cutting things pretty close. To be on the safe side, going for four of them will ensure plenty of extra overhead in case there end up being more users or a higher number of tokens needed per input than initially expected. Four L40S GPUs can fit perfectly into a 2U like the Puget Systems Quad GPU LLM Server.

If this fictional company didn’t have a dedicated space for rackmount servers, a desktop tower could be a viable alternative. NVIDIA has recently launched a new top-end professional GPU, the RTX PRO™ 6000 Blackwell, which has 96GB of VRAM and comes in a few variants – including the Max-Q Workstation Edition, which is designed for multi-GPU systems. The calculator shows that two of those (using the Custom option in the GPU drop-down) would work nicely, and those could fit in a full tower like the Puget Systems Multi GPU AI workstation.

Of course, many models can be run with even more modest hardware; this is just an example to help you understand the way to approach configuring a system to run your own LLM. Here at Puget Systems, we have options from compact workstations with a single GPU all the way up to servers with eight NVIDIA H200s – and you can even combine those into clusters for even more total VRAM! If you’d like personal guidance on selecting the right system(s) for your needs, our expert consultants are available to help with no cost or obligation. 

Overflowing VRAM

With some frameworks, there are options to utilize system memory (RAM) if a model needs more memory space than the available VRAM. LM Studio, for example, can configure this for you with a simple toggle option, making it very easy for even novice users. However, doing this comes with a very steep performance penalty!

I did some testing on my home PC, sporting an NVIDIA GeForce RTX™ 4070 SUPER 12GB, with three sizes of the Gemma 3 LLM. LM Studio configured quantization defaults for each one, which were pretty steep in order to let these run on my relatively modest GPU. I measured how much memory each version consumed via Task Manager, and LM Studio reported how many tokens per second they ran at when I asked them each the same query.

Even with very aggressive quantization settings, the 27 billion parameter version of Gemma 3 was too large for my GPU’s VRAM. LM Studio allowed it to overflow into system memory, so it still ran – but it took almost 30 times longer to answer the question than the 12 billion parameter variant. Was the bigger model able to answer questions more accurately, though? Let’s dig into that in the next section!

Accuracy and Hallucination

It is important to remember when working with LLMs that they are not actually conscious or intelligent, at least not in the way we think of those concepts for humans. Instead, they are extremely advanced prediction engines. However, similar to a human, they can’t “know” (reply with) information they were not trained on. Unlike most humans, though, LLMs are frequently observed making stuff up without giving any indication that they are ‘guessing’! This is commonly referred to as hallucination, and since LLMs generally don’t warn users that they are uncertain about a response, it makes taking their replies at face value a risk. Generally, models with a lower number of parameters are more likely to behave this way and provide incorrect responses.

I wanted to demonstrate this phenomenon, so during the performance testing I described above, I asked each of the models “What is the airspeed velocity of an unladen swallow?” I was expecting both a real-life answer as well discussion of this question’s famous inclusion in Monty Python and the Holy Grail. Here is my subjective analysis of what each model gave me:

Granted, this experiment was extremely subjective – but based on my experience with these three models I would say that the smallest one was very unreliable with its responses. It identified the correct movie that this quote was from, but then made up dialogue about it that never occurs in the film. The middle model, which was about half as fast, did a much better job and was more accurate with its description of how the quote was used in the movie. The largest model, which was roughly 30 times slower on my system, did not provide a substantial improvement – certainly nothing worth waiting so much longer to get a reply!

Conclusion

Generative AI is still a fast-growing field, and there are numerous advantages to hosting your own models including privacy, flexibility, and cost-savings over the long-term compared to paying for cloud services. I didn’t even touch on one of the most significant: retrieval-augmented generation (RAG), which lets you connect a LLM to your own data sources so that it can reference that info when responding to queries. Hopefully the tools and information presented here are helpful! I would love to hear about your experiences running local LLMs in the comments below.

Looking for an AI workstation or server?

We build computers tailor-made for your workflow. 

Configure a System

Don’t know where to start?
We can help!

Get in touch with one of our technical consultants today.

Talk to an Expert