Glossary
The words this site uses, explained without jargon. Every definition is written by us. The terms link here from the toolkit and the deals page; hover or focus a dotted term to see the short version.
Jump to: Computed bandwidth ceiling · Context window · Fully loaded hourly cost · Hybrid · Inference · Local AI · Memory bandwidth · Open-weight model · Parameters · Place it signal scores · Quantization · Readiness score · Rent vs own · Tokens per second · Unified memory · VRAM
Local AI in five minutes
An AI chat assistant is a program (a model) answering your questions. Most people use one that runs in someone else's data center: what you type travels over the internet, is processed there, and the answer comes back. That is cloud AI.
Local AI means the model runs on a computer you control: a laptop, a desktop, a small box in a closet, or a server in your building. Your question and the documents behind it never have to leave that machine, and it keeps working when the internet is down. The airplane-mode experiment on the home page shows the difference in one minute.
You can only run a model locally if its maker lets you download it. Those are called open-weight models. They come in sizes, counted in billions of parameters (a "7B" or "70B" model). Bigger models are usually more capable and need more memory.
Memory is the main hardware question. The whole model has to fit in fast memory (a graphics card's VRAM, or a computer's unified memory) to run well. Quantization shrinks a model so it fits in less memory. How quickly the computer can read that memory (memory bandwidth) largely sets how fast answers appear, measured in tokens per second.
For a team, the question is rarely "cloud or local" for everything. It is which jobs should stay in-house, which can go to the cloud, and which can be split (hybrid). The toolkit walks through that one job at a time: place it, check the knowledge behind it, treat the gaps, re-check, then model the cost.
Next: try the airplane-mode experiment · place a job · see what hardware fits which model size.
Terms
- Computed bandwidth ceiling
Our calculation of the most bandwidth a chip's memory spec allows: speed (MT/s) x 128 bits / 8. Not measured.
For processors whose memory is chosen by the box maker (for example Ryzen AI 9 HX 370 or Core Ultra mini PCs), the chip maker publishes the fastest memory it supports, not a bandwidth figure. We compute the theoretical ceiling as memory speed in MT/s x 128-bit bus / 8 / 1000 = GB/s (two 64-bit channels). We use the speed stated in the listing when it is lower than the chip's maximum, and leave it blank when the listing does not say which memory type is installed. Real-world speed is lower than the ceiling. See also: Memory bandwidth.
Also called: theoretical bandwidth.
- Context window
How much text a model can consider at once: the conversation plus any documents you give it.
The amount of text, in tokens, a model can take into account at once: your question, the conversation so far, and any documents you paste in. Longer context uses more memory on top of the model itself, so a machine that just fits a model may not fit it with a very long document. See also: Tokens per second, VRAM.
Also called: context length.
- Fully loaded hourly cost
What an hour of someone's time costs the organization: pay plus benefits, payroll taxes and overhead.
What one hour of a person's time actually costs the organization, not just their hourly pay. It adds benefits, employer payroll taxes, and a share of overhead such as equipment, software and space. Your finance team usually has a standard figure. The capacity scenario uses it to put a value on time saved; use your own number, and treat the result as capacity freed up, not cash saved. See also: Rent vs own.
Also called: loaded labor rate.
- Hybrid
Splitting a job: sensitive or routine parts run locally, and the cloud is used only where the task needs it.
Running one job partly locally and partly in the cloud. A common split is to keep the sensitive documents and routine steps on your own hardware and send only non-sensitive, harder steps to a cloud model, removing or masking private details first. In Place it, Hybrid is one of the three possible starting recommendations, alongside Local and Cloud. See also: Place it signal scores, Rent vs own, Local AI.
Also called: split placement.
- Inference
Using an already-trained model to produce answers. This is what local AI hardware does.
Using a model that has already been trained to produce an output: answering a question, summarizing a document, sorting a ticket. Training builds the model and takes enormous computing power; inference is the everyday use and needs far less. When this site talks about running AI locally, it means inference, not training. See also: Tokens per second, VRAM.
Also called: running a model.
- Local AI
Running an AI model on hardware you control instead of a provider's servers.
Running an AI model on a computer you own or control: a laptop, a desktop, a mini PC, or a server on your premises. Prompts and documents stay on that machine unless you choose to send them elsewhere, and the model keeps working without an internet connection once it is installed. The tradeoff is that you supply and maintain the hardware, and the largest frontier models are not available to run this way. See also: Open-weight model, Hybrid, Rent vs own.
Also called: on-prem AI, in-house AI.
- Memory bandwidth
How fast the processor can read memory, in GB per second. It largely sets how fast a local model writes answers.
How much data the processor can read from memory each second, in gigabytes per second (GB/s). To write each new token, a model has to read through its weights (for a mixture-of-experts model, the part it uses for that token), so bandwidth puts a ceiling on tokens per second. Twice the bandwidth roughly means up to twice the speed on the same model. On this site, figures for GPUs and unified-memory chips are the maker's published spec; for mini PCs whose memory the box maker picks, we show a computed theoretical ceiling with the formula. None of these are benchmarks we ran. See also: Computed bandwidth ceiling, Tokens per second, VRAM, Unified memory.
Also called: GB/s.
- Open-weight model
A model whose trained weights the maker publishes so anyone can download and run it.
An AI model whose trained weights (the numbers that make up the model) the maker has published for download, so you can run it on your own hardware. Open-weight is not the same as open source: the training data and code may stay private, and the license can still restrict commercial use or require attribution. Read the license on the model's page before using one at work. See also: Parameters, Inference, Quantization.
Also called: open model, downloadable model.
- Parameters (model size, e.g. 7B, 70B)
The count of learned numbers in a model. "7B" means about 7 billion; more parameters need more memory.
The learned numbers inside a model, counted in billions: a "7B" model has about 7 billion, a "70B" model about 70 billion. More parameters usually means more capability and always means more memory to hold the model. As a rough guide, the weights take about 2 bytes per parameter at 16-bit precision, 1 byte at 8-bit and about half a byte at 4-bit, plus extra room for the conversation and the operating system. Some models (mixture-of-experts) only use part of their parameters for each token, so they run faster than their size suggests, but all of the parameters still have to fit in memory. See also: Quantization, VRAM, Unified memory.
Also called: model size, B.
- Place it signal scores (e.g. "Local 12")
Points each answer adds to Local, Hybrid or Cloud in Place it. The highest total is the starting recommendation; Local maxes out at 12.
In Place it, each of your five answers adds points to Local, Hybrid or Cloud, and the highest total becomes the starting recommendation (ties go to Local, then Hybrid). High data sensitivity adds 3 to Local; routine tasks add 2 to Local, while tasks that likely need frontier capability add 3 to Cloud; high-volume use adds 2 to Local; work that must continue during an internet outage adds 3 to Local; and "core infrastructure" adds 2 to Local, while "renting is clearly sufficient" adds 2 to Cloud. Middle answers mostly add 1 or 2 to Hybrid. So "Local 12" is the maximum: every factor pointed local. Hybrid and Cloud can each reach 7. The scores show how strongly your answers lean, not a measure of quality or risk. See also: Hybrid, Rent vs own.
Also called: signals.
- Quantization
Storing a model's numbers with fewer bits so it needs less memory, usually with a small loss of quality.
Storing a model's weights with fewer bits each, for example 4 bits instead of 16, so the model takes much less memory and usually runs faster. A 4-bit version needs roughly a quarter of the memory of the 16-bit original. Quality usually drops a little, and more at very low bit counts, so test the quantized model on your actual tasks. Most models people run locally are quantized. See also: Parameters, VRAM.
Also called: 4-bit, 8-bit, Q4.
Source: Hugging Face Transformers docs: Optimizing LLMs for speed and memory (8-bit and 4-bit quantization).
- Readiness score
A 0 to 100 score from the 21-statement check of whether a team's knowledge is ready for an in-house AI.
The result of the Knowledge Readiness Check: 21 statements answered from No to Yes, turned into a 0 to 100 score overall and per area (such as sources, ownership and access). It points to the protocols that fix the weakest areas first. Re-check means taking it again later on the same device to see what moved. It describes how organized your knowledge is, not how good your team is. See also: Open-weight model.
Also called: Knowledge Readiness Check score.
- Rent vs own
Paying a provider per seat or per use (rent) versus buying hardware and running the model yourself (own).
The choice between paying a cloud provider per seat or per use (renting) and buying hardware to run models yourself (owning). Renting has little upfront cost, gets you the newest frontier models and scales up or down quickly. Owning costs more upfront and needs someone to look after it, but the ongoing cost is mostly fixed and the data stays with you. It is one of the five factors in Place it: "becoming core infrastructure" leans toward owning. See also: Hybrid, Place it signal scores, Local AI.
Also called: subscription vs hardware.
- Tokens per second
How fast a model writes. A token is a word or part of a word; in English, a token is about three-quarters of a word on average.
How fast a model produces text. Models read and write in tokens: whole words, pieces of words, or punctuation. In English, a token averages roughly three-quarters of a word, so 100 tokens is about 75 words. A model that writes faster than you read feels responsive for chat. Speed depends mostly on memory bandwidth and model size; the time to read a long document before answering depends more on raw compute. See also: Memory bandwidth, Inference.
Also called: tok/s, token.
Source: OpenAI Help Center: What are tokens and how to count them.
- Unified memory
One pool of memory shared by the CPU and the built-in GPU, so a large share can hold a model.
A design where the processor and the built-in graphics share one pool of memory instead of the GPU having its own VRAM. Apple silicon Macs, AMD Ryzen AI Max+ systems and NVIDIA GB10 systems work this way. Because much of the pool can be used for the model, a machine with 64 GB or 128 GB of unified memory can hold models far larger than most single graphics cards. The memory is usually soldered, so buy the amount you need up front. See also: VRAM, Memory bandwidth.
Also called: shared memory.
- VRAM
Memory built into a graphics card. A model must fit in it for the card to run the model at full speed.
The memory built into a graphics card (GPU), separate from the computer's main RAM. For local AI it is usually the deciding number on a GPU: the model, plus room for the conversation, has to fit in VRAM for the card to run it at full speed. If it does not fit, software can split the model between the GPU and the main RAM, but it runs much slower. A 24 GB card holds a lot more model than a 12 GB card, whatever its other specs. See also: Unified memory, Memory bandwidth, Quantization.
Also called: video memory, GPU memory.
These definitions are general explanations, not advice for a specific purchase or deployment. Spotted something wrong or unclear? feedback@folktechai.com. The same glossary is available as JSON and through the MCP tool get_glossary.