Your GPU Is Right There, Stop Renting an H200 to Write Haikus

Spinning up an AI model used to be a personality test. You'd budget an afternoon, and three days later you'd be nine tabs deep in an NVIDIA forum thread from 2019 where the accepted answer was "reinstall the driver." Then a segfault. Then the discovery that your batch size was one larger than your VRAM could tolerate, which CUDA communicates by simply dying without comment.

Now it's one command and you're serving tokens.

But the Hugging Face container ecosystem has moved a lot since I wrote about dockerizing vLLM — including one plot twist I genuinely did not see coming, which we'll get to in about ninety seconds. Docker Model Runner is now a first-class citizen on the Hub. The old "serverless API" quietly got renamed and repurposed. And the pricing has enough layers that I built a spreadsheet, felt bad about myself, and deleted it.

So here's the thesis, up front, so you can stop reading early if you want:

The GPU in your machine is right there. It has been playing video games and rendering Slack. It can serve an LLM. Start there, and only escalate when something physically stops you.

First, a Funeral

Every "Hugging Face containers" guide on the internet — including, embarrassingly, the draft of this one — opens by telling you that Text Generation Inference is the production-grade heavy lifter you should reach for.

TGI has been in maintenance mode since December 11, 2025, and the GitHub repository was archived on March 21, 2026. It's read-only. It's done. The Hugging Face docs for Inference Endpoints now open with a big yellow CAUTION box that politely suggests you go use something else, followed by a migration guide.

Which is a genuinely strange feeling! TGI was good. Rust core, continuous batching, tensor parallelism, speculative decoding, an OpenAI-compatible Messages API — it did everything right. It just got out-competed by vLLM and SGLang, which iterated faster and now own the space. Hugging Face, to their credit, didn't pretend otherwise. They put it down and pointed at the alternatives.

You can still pull it. The image is still sitting there at ghcr.io/huggingface/text-generation-inference:3.3.5, permanently frozen at its final version, like a photograph of someone at their best.

# This still works. It will always work. It will never improve.
docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data \
    ghcr.io/huggingface/text-generation-inference:3.3.5 \
    --model-id meta-llama/Llama-3.3-70B-Instruct

What to use instead: vLLM if you want the biggest ecosystem and the most model support. SGLang if your workload has heavy prefix reuse — long system prompts, agent loops, RAG contexts that repeat. llama.cpp if you're on modest hardware or CPU-only. All three are selectable engines on Inference Endpoints, and all three run fine in a container on your own box.

While we're holding a funeral: huggingface/transformers-inference on Docker Hub, which a lot of guides still list as an "official container type," was last updated over three years ago. Its newest tag is 4.24.0 — that's transformers 4.24, from late 2022. If you find yourself pulling it in 2026, something has gone wrong in your life. The modern equivalent is the Hugging Face Inference Toolkit, which is what actually runs under the hood on Endpoints.

Rung 1: The Computer You Already Own

In July 2025, Docker and Hugging Face did something quietly excellent: they made Docker Model Runner a first-class Local Apps provider on the Hub. It's now the default one.

What that means in practice: you go to any GGUF model page on the Hub, pick Docker Model Runner from the dropdown, and it hands you a command. That's the entire onboarding experience.

# Pull a model straight from the Hub
docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

# Or just run it — pulls if needed, drops you into a chat
docker model run hf.co/unsloth/gpt-oss-20b-GGUF:Q2_K_L

No CUDA toolkit. No PyTorch install. No pip resolving dependencies for eleven minutes before informing you that two of your packages have opinions about each other.

A few things worth knowing that the marketing pages skip:

It's not actually a container. Despite the name and the CLI, Docker Model Runner executes models as a native host process on top of llama.cpp. That's deliberate — it means direct GPU access with no passthrough shenanigans. You get Docker's distribution model (models are OCI artifacts, pulled and cached like images) without the runtime penalty.

It speaks OpenAI. The API lands on http://localhost:12434/engines/v1 with the usual /chat/completions, /completions, /embeddings, and /models routes. From inside another container, it's http://model-runner.docker.internal/engines/v1. You may need to switch on TCP host access first:

docker desktop enable model-runner --tcp 12434

Linux users need one extra step. If you're on Docker Engine without Docker Desktop — which describes most of my homelab — install the plugin:

sudo apt-get update && sudo apt-get install docker-model-plugin
docker model version

Finding models: the Hub has a filter for exactly this — huggingface.co/models?apps=docker-model-runner — which lists every repo containing GGUF weights. That's the constraint, by the way: GGUF only. Which brings us to the part where your hardware and your ambition negotiate.

"But My GPU Is Small"

Mine too. This is what quantization is for, and I wrote a whole post on what Q2 through Q6 actually mean so I don't have to re-explain it here. The short version: a Q4 quant of a 20B model will fit in far less VRAM than you think and will be far less stupid than you fear.

And if you're running into the wall anyway, I have been there so recently and so loudly that there's a post about it. The wall is real. But it's usually further out than people assume, and "I don't have enough VRAM" is a claim most people make without ever having tried.

Already running Ollama? Then you've already got Rung 1 covered and you can skip ahead. Docker Model Runner and Ollama are solving the same problem from different directions — DMR wins if your whole workflow is already Docker Compose, Ollama wins on ecosystem maturity. Pick one, don't run both, don't overthink it.

The One Container You Should Still Run Yourself

TGI is gone, but Text Embeddings Inference is alive and actively developed, and it remains the correct answer for embeddings. It's the same Rust-and-Candle engineering philosophy, aimed at a workload where it still has no real competition.

model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data

docker run --gpus all -p 8080:80 -v $volume:/data --pull always \
    ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
    --model-id $model

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

What makes TEI worth the container:

Token-based dynamic batching. It batches at the token level rather than the request level, so one giant document doesn't hold seventeen one-sentence queries hostage behind it.

It boots absurdly fast. Small images, minimal warm-up. This is the thing that makes it viable for scale-to-zero deployments where a thirty-second cold start would ruin everything.

It does more than embeddings. This is the part everyone misses. TEI also serves re-rankers (/rerank) and sequence classification models (/predict). If you're building RAG, that means one container gives you both halves of a retrieve-then-rerank pipeline instead of two separate services.

curl 127.0.0.1:8080/rerank \
    -X POST \
    -d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."], "raw_scores": false}' \
    -H 'Content-Type: application/json'

And there's an OpenAI-compatible /v1/embeddings route, so anything already pointed at OpenAI embeddings can be redirected at your own box with a base URL change.

Rung 2: Somebody Else's Free GPU

No GPU? Laptop that thermally throttles if you open a second Chrome tab? Fine. There are two genuinely useful free options, and they are not equally good.

Groq — The One I'd Actually Recommend

I'll just say it: if you have no local GPU and you want to build something today, use Groq. The free tier is the most generous one in the business and it requires no credit card.

• 30 requests per minute
• 6,000 tokens per minute
• 14,400 requests per day
• Llama, Qwen 3, Kimi K2, GPT-OSS 120B, and friends

Fourteen thousand requests a day. For free. On LPU hardware that returns first tokens so fast it feels like the response was already sitting there waiting for you. For prototyping, for a side project, for the demo you're building this weekend — this is not a trial, it's a real allowance. Adding a card (with zero minimum spend) unlocks roughly 10× those limits and a 25% discount, which tells you they're not exactly desperate for your money either.

ZeroGPU Spaces — Great, But Read the Meter

Hugging Face Spaces can run on shared, dynamically allocated GPUs. The hardware is NVIDIA RTX Pro 6000 Blackwell, and you don't reserve one — you borrow one for the duration of a decorated function call.

Note the unit. That's minutes of GPU time per day, not requests. Five minutes goes quickly when you're iterating on an image model. The quota resets 24 hours after your first use, and PRO/Team/Enterprise can spill over into pre-paid credits at $1 per 10 minutes.

Two more details that matter: the default large size gives you half a Blackwell (48 GB), while xlarge gives you the full 96 GB but burns quota at 2×. And ZeroGPU Spaces are Gradio-only — if you were planning a FastAPI Space, that's a different hardware tier.

import spaces
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(...)
pipe.to('cuda')          # load at module level, not inside the function

@spaces.GPU(duration=120)
def generate(prompt):
    return pipe(prompt).images

Free CPU Spaces — Underrated

2 vCPU, 16 GB RAM, 50 GB disk, no hourly cost. That is a genuinely capable little server, and it will happily run an embedding model or a classification endpoint indefinitely. Two caveats worth knowing before you plan around it: free-hardware Spaces get suspended after about 48 hours of inactivity (any visitor wakes them back up), and creating a new compute Space now requires a paid plan — static Spaces remain free for everyone.

Rung 3: Inference Providers, and the Credits Nobody Explains Correctly

This is the piece of the ecosystem I think is most underrated. Inference Providers is one OpenAI-compatible endpoint — router.huggingface.co/v1 — that routes to sixteen different backends: Groq, Cerebras, Together, Fireworks, Nebius, Novita, SambaNova, Hyperbolic, Featherless, nscale, fal, Cohere, Replicate, Scaleway, Public AI, and Hugging Face's own hf-inference. Over 200 models.

Switching providers is a string change. No new account, no new API key, no new billing relationship, no new SDK.

from openai import OpenAI

client = OpenAI(
    base_url="https://router.huggingface.co/v1",
    api_key=os.environ["HF_TOKEN"],
)

completion = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3-0324",
    messages=[{"role": "user", "content": "How many 'G's in 'huggingface'?"}],
)

And Hugging Face takes no cut. You pay the provider's published rate, full stop. No markup, no convenience fee. OpenRouter — the obvious comparison — adds a small markup at most tiers in exchange for a much bigger catalog including proprietary frontier models. For open-weight models specifically, HF is cheaper. For GPT-5 and Claude, OpenRouter is the one with the catalog.

Now, the credits, because I have seen this number mangled everywhere including in my own notes:

Two dollars. Not two million tokens, not two million credits — two American dollars of inference, and ten cents if you're on the free plan. If you've read otherwise, so had I, and we were both wrong.

This is not a reason to skip PRO, incidentally. At $9/month you're really buying the 40 minutes of daily Blackwell time, the ability to host 10 ZeroGPU Spaces instead of 2, and 1 TB of private storage. The inference credits are a rounding error attached to a good deal.

A Note on the Artist Formerly Known as the Serverless API

Half the tutorials online still tell you to use the "Serverless Inference API" for free Llama access. That product was folded into Inference Providers as the hf-inference provider, it bills against your credits by compute-time, and since July 2025 it focuses mostly on CPU inference — embeddings, text ranking, classification, and historically significant smaller models like BERT and GPT-2.

It is not a free Llama-70B tap. It never really was, but now it's officially not. If a tutorial promises you one, that tutorial is from a previous geological era.

Rung 4: Renting the Whole Machine

When you need dedicated hardware, Inference Endpoints is the move. You pick a model, pick an engine (vLLM, SGLang, llama.cpp, TEI, TGI if you're feeling nostalgic, or your own container), pick hardware, and Hugging Face handles provisioning, autoscaling, and the API in front of it. Billing is by the minute, even though everything is quoted hourly.

CPU — Cheaper Than You'd Guess

Twenty-four dollars a month for a dedicated, always-on inference endpoint. That's less than most people's streaming bundle. Azure and GCP are also available at somewhat higher rates.

GPU — The Part Where You Should Sit Down

A few corrections to things you'll read elsewhere, including in the draft I started from: there is no B200 on the price list, and there is no H100 on AWS — H100 is GCP-only and starts at $10.00/hr for a single card, which is double the AWS H200 that beats it on memory. There are also AWS Inferentia2 and Google TPU v5e tiers if your workload suits them.

Eight H200s is $40/hr. That's $960 a day. That is a car payment before lunch.

Scale-to-Zero, Honestly

Scale-to-zero is the feature that makes dedicated endpoints viable for bursty traffic: no requests, no replicas, no charge. When a request arrives, a replica spins up and serves it.

Two things the marketing pages soft-pedal, and I'd rather you hear them from me:

The cold start is real. That's the entire trade. You are exchanging latency on the first request for not paying for idle GPU time. This is a fine trade for a Slack bot and a terrible one for a checkout flow. TEI's fast boot makes this much less painful than a 70B model waking up.

Scaled-to-zero still counts against your endpoint quota. Only paused endpoints free up the allocation. If you're hoarding a dozen scaled-to-zero endpoints and wondering why you can't create another, that's why.

Realistically: a bursty workload at 100–1,000 requests/day on a T4 with scale-to-zero lands somewhere in the $35–90/month range. A steady 100K+ requests/day on an always-on A100 is ~$1,825/month, at which point renting raw GPUs from RunPod or Lambda usually wins and you've reached Rung 5.

The Part Everybody Misses: Hugging Face Is a Deployment Target

Here's the thing I want people to internalize, because it took me embarrassingly long. Most developers treat the Hub like a download site — a museum where you go look at other people's models and take one home.

It's a deployment target. You train a model, you push it, and the same infrastructure that serves Llama serves yours. Same API, same hardware tiers, same scale-to-zero. Fine-tune a small classifier on your own data, push it, deploy it on a $0.033/hr CPU endpoint, and you have a private production API for the price of a sandwich.

And when the managed engines don't fit — custom preprocessing, weird dependencies, an architecture vLLM has never heard of — you bring your own container.

There is exactly one rule you must not get wrong:

Load your model from /repository, never from the Hub.

Hugging Face mounts the model you selected at /repository inside your container using a very fast internal download path. If your code calls out to the Hub at startup instead, you're re-downloading weights over the public internet on every cold start, fighting the platform that is actively trying to help you.

MODEL_ID = "/repository"      # not "meta-llama/Llama-3.3-70B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=DTYPE).to(DEVICE).eval()

The rest of the contract is refreshingly boring. Expose a port and declare it in the UI. Serve a /health route that returns non-200 until your model is actually loaded — the readiness probe hits it every second, and getting this wrong means traffic arriving before you're ready. Don't bake weights into the image; it only needs your code and dependencies. And if you build on a Mac, --platform linux/amd64, because arm64 images will be rejected.

@app.get("/health")
def health():
    try:
        model_manager.get()
    except ModelNotLoadedError as exc:
        raise HTTPException(status_code=503, detail=str(exc)) from exc
    return {"message": "API is running."}

Push to Docker Hub, ECR, ACR, or GCR, point the endpoint at the image, and you're serving your own model on somebody else's H200 without ever having thought about a driver.

While you're at it: whatever you deploy, put a bouncer in front of it. Ask me how I learned that one.

The Cheat Sheet

Notice how much of that table is $0. That's not an accident, and it's not me being cheap. It's that the local-first option is genuinely good now, and the escalation only makes sense when a real constraint pushes you.

The exception I'll flag: if you're running something at home and need to reach it from outside, you don't need a cloud endpoint — you need a tunnel. I've written about reaching a home Ollama from the cloud, and it's a fraction of the cost of the equivalent hosted GPU.

Which Brings Me to the Robot

There's a reason I've been mapping every inference option from "free on my desk" to "$40 an hour," and it isn't purely academic.

I bought a Reachy Mini.

For the unfamiliar: Reachy Mini is an eleven-inch, three-pound desktop robot from Pollen Robotics — which Hugging Face acquired — and it is the most direct expression of "the Hub is a platform, not a museum" that I've seen. Six degrees of freedom in the head, full body rotation, two animated antennas that do more emotional work than they have any right to, a wide-angle camera, four microphones, and a 5W speaker. The Lite version is $399 and uses your Mac or PC as its brain over USB. The Wireless version is $499 and has a Raspberry Pi onboard, so it just… wanders off and thinks for itself.

The whole thing is Apache 2.0 — hardware, software, and simulation environments. The Python SDK lives at pollen-robotics/reachy_mini. And the part that made me actually pull out a credit card: its app store is Hugging Face Spaces. Robot behaviors are Spaces. You browse them from the robot's dashboard, install with one click, and publish your own the same way you'd publish any other Space.

Which means every single rung of that ladder up there is now a design decision about a physical object sitting on my desk. Does the conversation model run locally on the Pi? On my workstation over the network? On a Groq endpoint when it needs to be fast, falling back to local when the Wi-Fi is being the Wi-Fi? What happens to a robot's personality when its brain is 40 milliseconds away instead of on-board?

I don't know yet. That's the next post. There will be assembly, there will be a moment where I hold a very small screw and question my life choices, and there will almost certainly be a section titled "Why Is It Staring At Me."

Summary

• TGI is archived — maintenance mode since December 2025, repo read-only since March 2026. Use vLLM or SGLang. Update your mental model and your bookmarks.
• TEI is alive and excellent, and it does re-ranking and classification too — one container for your whole retrieval pipeline.
• Docker Model Runner is the new default local path. docker model run hf.co/<org>/<repo>, OpenAI API on port 12434, no CUDA archaeology required.
• Use your own hardware first. It's paid for. It's sitting there. Quantization makes it go further than you think.
• If you have no GPU, use Groq — 14,400 requests a day, free, no card. ZeroGPU is great too, but it's metered in minutes and they go fast.
• PRO's real value is 40 daily minutes of Blackwell, not the $2 of inference credits that everyone including me has been misquoting as "2 million."
• Inference Providers charges no markup — you pay the provider's rate through one API and one bill.
• Scale-to-zero has a cold start, and scaled-to-zero endpoints still consume your quota. Pause them, don't just idle them.
• The Hub is where you serve your models, not just where you get them. Load from /repository, serve a real /health, and it's a genuinely great deployment target.

The ecosystem moved fast enough in eighteen months to bury its own flagship engine and replace the local development story entirely. It'll move again. But the ladder holds: start with what you own, escalate only under duress.

Next up: unboxing a robot, and figuring out where to put its brain.

Happy Coding!!!