World 1-5: the castle

Décio Dalke Jr. — Arquiteto de negócios. Sócio-gerente no ecossistema MitUP (mitup.pt).

World 1 of AI series — level 5 of 5: World 1-1 · World 1-2 · World 1-3 · World 1-4 · World 1-5

In classic Super Mario, every world ends in a castle. It’s the darkest level, the hardest one, and the one with the boss at the end. It’s also the one everybody knows, even people who never played.

AI works the same way. The public conversation revolves around the model: which one is smartest, which one came out this week, which one won the benchmark. I left the model for last on purpose. It’s the most famous piece and, for most companies, the one that decides the least.

An AI model is the engine that generates the answers: a program trained on an enormous amount of text to predict the next chunk of text (World 1-2 explains the rest). Choosing a model means deciding four things: its size, how open it is, whether it will be fine-tuned, and where it will run. Each of them has an equivalent that any manager has already decided at some point, and that’s how I’ll explain them.

LLM or SLM: does size matter?

First, a word that turns up in every AI news story: parameter. Picture a mixing desk with billions of knobs. Training a model is the process of turning each of those knobs, millions of times, until the music comes out in tune. Each knob is a parameter. The more knobs, the more nuance the desk can reproduce, and the more expensive it is to build and to run.

An LLM, a large language model, has hundreds of billions of parameters or more. An SLM, the small one, has anywhere from a few million to a few billion [1].

The difference is a bit like a hospital. The LLM is the experienced GP: sees any case, copes well with the unexpected, but the appointment is expensive and slow. The SLM is the lab technician who runs one test very well, quickly and cheaply, and who gets lost if you ask for a complex diagnosis [1]. You don’t have the GP doing blood tests all day.

That’s precisely the argument of a 2025 NVIDIA study: inside agent systems, much of the work is repetitive, narrow tasks, and for those, small models are sufficient, better suited and more economical [2].

So the manager’s question isn’t “which is the best model”. It’s “what’s the smallest model that does this task well”. Classifying an invoice doesn’t need the same professional who reviews a contract.

How much does it cost to train a model from scratch?

A model’s initial training, what the technical people call pre-training, is the full education of a professional: school, university, residency. That’s where it learns the language, the logic and general knowledge of the world.

And it costs accordingly, multiplied by millions. According to Stanford’s AI Index, using estimates from Epoch AI, training GPT-4 in 2023 cost about $79 million, and Gemini 1.0 Ultra about $192 million [3]. In compute alone, without counting the teams.

For almost no company does this make sense. It’s the difference between founding a medical school to have a doctor on staff and hiring a doctor who is already qualified. The second option exists, it’s cheaper, and the qualified doctor knows more than anyone you could train on your own.

Open or closed model: what’s the difference?

A closed model is like hiring a top-tier consultancy: you get the work, you pay for usage, but you don’t see the methods, you can’t take the consultant home, and you depend on the contract. You reach that model through an API, which in practice is a power socket: your system plugs into the supplier’s model over the internet and pays for every question.

An open model is one you can download and run wherever you like, on your own machines.

There’s a naming mix-up here worth understanding. Almost everything called an “open model” is actually open-weights (the weights are the settings of those mixing-desk knobs, already adjusted). Think of a cake. Open weights means receiving the finished cake: you can slice it, decorate it, serve it to whoever you like. Genuine open source means receiving the recipe and the list of ingredients, so you can bake an identical one yourself. In 2024 the Open Source Initiative published a definition of open source AI that requires, besides the model, the complete training code and detailed information about the data used [4]. By that yardstick, most “open” models don’t make the cut: they hand over the cake, not the recipe. OpenAI itself, when it released its gpt-oss models in 2025, described them as open-weights [5].

For a company, open gives you control: the data stays in-house, and the cost doesn’t grow with every question. Closed gives you the most capable model of the moment without having to look after infrastructure. There’s no right answer outside the concrete case.

What does fine-tuning change, and what doesn’t it?

Fine-tuning means taking an already trained model and continuing its training with your own examples.

The best comparison is onboarding a new employee. You hire someone already qualified and, in the first few weeks, teach them how things are done here: the report format, the tone with clients, how to answer a request for a proposal. That’s what fine-tuning does well. It changes behaviour: format, style, the way it answers a certain kind of request.

It got much cheaper with techniques like LoRA [6]. Instead of rebuilding the whole building, it refurnishes a few rooms: it adjusts only a small part of the model and, in the original test, cut the number of parameters that need training by up to 10,000 times.

What fine-tuning doesn’t do well is put new knowledge into the model’s head. A 2023 study compared the two ways of teaching a model new facts and concluded that retrieving the facts from documents (the RAG from World 1-4) consistently beat fine-tuning [7]. Another, from 2024, went further: the more new facts fine-tuning forces the model to learn, the more it starts to hallucinate [8].

Go back to the new employee. Onboarding teaches how the company works. Nobody expects them to memorise the entire company archive in the first week; for facts, they look them up in the archive. Force someone to memorise, under pressure, thousands of data points they don’t yet understand, and you get an employee who answers everything with confidence, and gets things wrong with the same confidence.

That’s the reason for the table in World 1-4. Fine-tuning teaches the model to work your way, not to know what only your company knows. Whoever fine-tunes a model so it will “learn the company” usually ends up with an assistant that is more confident, not more correct.

What is distillation?

It’s the relationship between master and apprentice. The apprentice doesn’t read the whole library the master read; they watch the master at work, thousands of times, and learn to answer the way the master does. In distillation, a large model (the master) generates answers, and a small model (the apprentice) is trained to reproduce them.

The concept was formalised in 2015 [9] and became famous in January 2025, when DeepSeek distilled the reasoning of its R1 model into six smaller models. One of them, with 32 billion parameters, beat OpenAI’s o1-mini on several reasoning benchmarks [10]. These benchmarks are standardised tests, a kind of entrance exam for models, with questions on maths, science and programming.

For a manager, distillation is why small models got so good so fast. Today’s lab technician learned from yesterday’s GP.

Is it worth running a model on your own computer?

Today you can run models on your computer or on your own server, with free tools like Ollama, LM Studio or llama.cpp. The smallest gpt-oss model, for example, runs with about 16 GB of memory [5], which already fits on a higher-end work machine.

The comparison I use is taxi versus your own car. The cloud model is the taxi: better car, driver included, you pay per ride and don’t worry about maintenance, but the taxi company knows where you went. The local model is your own car: smaller and more modest, more predictable cost, nobody knows your route, and the maintenance is yours.

There are three reasons to own the car: privacy (the data stays in-house), predictable cost, and working without internet. The price is just as clear: the model that fits on your machine is smaller and less capable [1], and someone has to look after the machine. It makes sense for sensitive data and narrow tasks. For general work, the cloud model is still ahead.

So which model should you use?

Start with what already exists: the socket. That’s what most companies do: in a 2024 a16z survey, 72% of companies accessed models through an API [11]. Build the pieces from World 1-4 around it. Measure. And only then ask whether a smaller, open or fine-tuned model solves a specific problem better and more cheaply.

The reverse order, picking the model and then figuring out what it’s for, is the most expensive way to learn what World 1-1 already said: AI is only as good as whoever runs it.

In the first world’s castle, Mario beats the boss and gets the news: “Thank you Mario! But our princess is in another castle!” The first world wasn’t the game. It was where you learn the rules.

AI is the same. Whoever beats the model castle finds out that the real game is still out there, in the company’s processes, people and decisions.

Quick questions

What’s the difference between an LLM and an SLM? Size. LLMs have hundreds of billions of parameters or more and handle complex tasks better; SLMs have from millions to a few billion, are cheaper and faster, and are enough for many narrow tasks.

Is an open model the same as open source? Almost never. Most “open” models are open-weights: you get the finished cake but not the recipe, meaning neither the training code nor detailed information about the data.

Is fine-tuning the way for AI to learn my company’s data? It isn’t the best tool for that. Fine-tuning changes behaviour, like onboarding a new employee; for business facts, retrieving from documents (RAG) works better and with less risk of hallucination.

Is it worth training your own model? From scratch, almost never: training frontier models costs tens to hundreds of millions of dollars. Fine-tuning or using an off-the-shelf model covers the vast majority of cases.

References

1. IBM — Small language models (SLMs) (IBM Think), 31 October 2024, updated 2 April 2026. https://www.ibm.com/think/topics/small-language-models — accessed 2026-09-20.
2. NVIDIA Research — Small Language Models are the Future of Agentic AI, 2025. https://arxiv.org/abs/2506.02153 — accessed 2026-09-20.
3. Stanford HAI — Artificial Intelligence Index Report 2025, chapter 1. https://hai.stanford.edu/assets/files/hai_ai-index-report-2025_chapter1_final.pdf — accessed 2026-09-20.
4. Open Source Initiative — The Open Source AI Definition 1.0, 28 October 2024. https://opensource.org/ai/open-source-ai-definition — accessed 2026-09-20.
5. OpenAI — Introducing gpt-oss, 5 August 2025. https://openai.com/index/introducing-gpt-oss/ — accessed 2026-09-20.
6. Hu, E. et al. — LoRA: Low-Rank Adaptation of Large Language Models, 2021. https://arxiv.org/abs/2106.09685 — accessed 2026-09-20.
7. Ovadia, O. et al. — Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, 2023/2024. https://arxiv.org/abs/2312.05934 — accessed 2026-09-20.
8. Gekhman, Z. et al. — Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? EMNLP 2024. https://arxiv.org/abs/2405.05904 — accessed 2026-09-20.
9. Hinton, G.; Vinyals, O.; Dean, J. — Distilling the Knowledge in a Neural Network, 2015. https://arxiv.org/abs/1503.02531 — accessed 2026-09-20.
10. DeepSeek — DeepSeek-R1, January 2025. https://arxiv.org/abs/2501.12948 — accessed 2026-09-20.
11. a16z — 16 Changes to the Way Enterprises Are Building and Buying Generative AI, 21 March 2024. https://a16z.com/generative-ai-enterprise-2024/ — accessed 2026-09-20.