Running Mistral models on a NAS

Which Mistral model should you run on a NAS?

Mistral 7B is compact and quick, about 4.4GB at 4-bit. Mistral NeMo 12B adds a long context window and better multilingual answers for around 7.5GB. Mistral Small 24B is a strong mid-size model at about 14GB that suits a 32GB NAS. Mistral Large 123B, at around 73GB at 4-bit, needs a 96GB-class system or more (Estimated).

Why choose Mistral on a NAS?

Mistral models are known for efficient answers and good European language support, which suits UK and European businesses handling documents in several languages. NeMo's long context helps when asking questions of long documents.

Can a NAS run Mistral Large?

Only systems with 96GB or more, and slowly without a GPU. Its 123B parameters are all active for every word, so a processor has a great deal of work to do for each part of an answer. If you want Mistral Large quality day to day, plan for a NAS with a GPU path and a card with plenty of memory, or accept that answers will take a while. For most NAS owners, Mistral Small 24B gives the best balance of quality and speed.

Common questions

Which quantizations are published?

Ollama publishes Q2 to Q8 for Mistral 7B, NeMo and Large, but only Q4, Q8 and full precision for the current Mistral Small.

Can Mistral run on a 16GB NAS?

Mistral 7B and NeMo 12B fit comfortably.