In a recent project I had 1,500 tweets and a codebook with 16 categories. There were three ways to get labels: code them by hand, send them to a commercial API, or run an open-weight model on our own GPU. I went with the third one, and this post is what I wish someone had told me before I started.

The reasons for staying local are not ideological. First, the data never leaves the machine. Second, I have already the hardware and there is no per-token cost, so you can iterate on the prompt fifty times instead of five, and prompt iteration is where the accuracy comes from. Third, a pinned model version at temperature=0 is reproducible, which turns the classification step into something you can ship in a replication package rather than a black box you apologize for in a footnote.

Which engine

If your mental model is “load the model with transformers and loop over the rows of a data frame”, that works, and it is slow. The reason is that a single request leaves the GPU mostly idle. vLLM solves this with continuous batching and paged attention: 24 requests running at the same time cost far less than 24 times one request, and new requests join the batch as old ones finish instead of waiting for a full batch to complete. The second reason to use it is boring but practical: vLLM serves an OpenAI-compatible HTTP endpoint. The client code is the same code you would write against a commercial API, so you can develop against one and switch to the other by changing a base URL.

Sizing the model to the card

Before comparing models on quality, do the arithmetic that actually decides the question. In bf16, weights take about 2 bytes per parameter. A 31B model is therefore roughly 59 GB before a single token of KV cache. On an 80 GB A100 that fits with usable context. On a 24 GB card it does not, and you either drop a size class or use a quantized checkpoint.

I tested two candidates, Gemma 4 31B and Qwen 3.6 27B. The useful advice here is not “use this one”, because that changes every few months. It is: pick the two largest models that fit your card, hand-code 100 to 150 items yourself, and run both against them. That comparison takes an afternoon and tells you more than any leaderboard, because leaderboards do not know your codebook.

Getting it running

Serving is one command. The flags below are the ones that matter for a classification job:

vllm serve google/gemma-4-31B-it \
  --served-model-name gemma4-31b \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 24 \
  --enable-prefix-caching \
  --generation-config vllm \
  --reasoning-parser gemma4 \
  --port 8000

A few of these are easy to get wrong:

• --generation-config vllm tells vLLM to ignore the model’s own generation_config.json. Without it, whatever top_k and top_p the model ships with are applied, and your runs are not reproducible even at temperature=0. For research use this is the single most important flag on the list, and the one I see discussed least.
• --gpu-memory-utilization is a fraction of free memory, not of total memory. Several GB are typically already reserved, so 0.95 can fail at startup with a Free memory on device error while 0.90 starts fine.
• --max-model-len has to hold the prompt plus the entire generated answer, including the reasoning trace if you enable thinking. A 1,100-token codebook prompt plus free reasoning does not fit in 4096, and the failure mode is a bare HTTP 400 that looks like a malformed payload.
• --enable-prefix-caching is close to free when every request shares the same long codebook prompt, which in a classification job is always.
• --reasoning-parser has to be named per model family (gemma4, qwen3, and so on). There is no sensible default, and see the traps section for what happens if you forget it.

The client side is the ordinary OpenAI SDK pointed at localhost:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

resp = client.chat.completions.create(
    model="gemma4-31b",
    messages=[{"role": "system", "content": CODEBOOK},
              {"role": "user", "content": tweet}],
    temperature=0,
    max_tokens=4096,
)

Making the model answer in JSON

Ask a model for JSON in the prompt and most of the time you get JSON. “Most of the time” is the problem: at 1,500 items, a 2% failure rate is 30 items you have to handle with a regex, and regex fallbacks introduce their own bias (more on that below).

Guided decoding removes the problem instead of patching it. At every decoding step the engine masks out every token that cannot continue a string valid under your schema, so the output parses by construction. There is no cleanup stage because there is nothing to clean up:

resp = client.chat.completions.create(
    model="gemma4-31b",
    messages=[...],
    temperature=0,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "classification",
            "schema": {
                "type": "object",
                "properties": {
                    "label": {"enum": ["everyday_life", "promo_live", "..."]},
                    "confidence": {"type": "number"},
                },
                "required": ["label", "confidence"],
            },
        },
    },
)

{“label”: “everyday_life”, “confidence”: 0.95}

So verify that the constraint is real, and do it again after every upgrade. The test that works is a canary schema no model would ever produce on its own:

{"zzq_verdict": {"enum": ["alpha", "beta", "gamma"]}, "zzq_score": {"type": "integer"}}

If the reply comes back with exactly those keys, the constraint is being enforced. A schema that happens to match what the model would have written anyway proves nothing at all.

There is a flip side, and it is the one that will actually bite a researcher. An enum forces a valid answer even when your prompt is wrong. In my case the 16 label codes lived in a single Python module, but they were rendered in three different places, and two of the three disagreed with the third. A hardcoded three-role enum in one output block silently overrode the 16-code codebook. Schema-constrained decoding hid this completely, because the enum forced plausible answers regardless. Only unconstrained decoding exposed it. If you support both modes, test the unconstrained one occasionally: it is the only one that reads your prompt literally.

Does thinking help?

Both models can produce a chain of thought before answering. The interesting question for a classification task is whether it is worth 20x the wall clock. I ran the same 150 tweets through five configurations at temperature=0 on one A100, sorted here from best to worst accuracy:

“Schema valid” is the share of replies that parse against the schema, whether or not the constraint was switched on. The 0% in the second row is a real measurement rather than a missing value: with thinking enabled and no constraint, not one of the 150 replies came back as bare valid JSON, because the model wraps its answer in prose.

Three things I take from this:

• Thinking buys roughly 6 percentage points and costs 15 to 25 times the output tokens. For 1,500 documents that is the difference between about 5 minutes and 1.5 to 3 hours. Schema-only decoding is the right choice when volume matters, thinking when accuracy does.
• Thinking and a JSON schema compose in a single request. The grammar is not applied during the reasoning phase; it engages only after the thought-end token, so the model reasons freely and is then forced into the schema. For Gemma the schema costs about one point of accuracy, which at 150 items is noise, while structural validity goes from 0% to 100%. A two-step “think first, reformat in a second request” pipeline doubles your inference for something one pass already gives you.
• Qwen thinks about 1.5 times as long as Gemma for an edge of two points, so the ranking between the two is a lot less stable than the gap between thinking and not thinking.

Traps that cost me the most time

• The response field is reasoning, not reasoning_content. vLLM renamed it and recent versions do not emit the old key at all, so reading the old one returns None on every single request. This makes a perfectly working parser look broken and leads straight to the wrong conclusion that thinking and structured output are incompatible. Print sorted(msg.keys()) before you believe a field is empty.
• Serving a model without its reasoning parser does not error. The thinking just lands in content as prose, which reads exactly like “this model has no thinking mode”.
• Too small a token budget gives you finish_reason=length with content=None. The model spent the whole budget thinking, the thought never terminated, and the parser discarded it. The symptom is an empty answer and the cause is the cap, not the parser. At max_tokens=1400 this hit 17% of my classifications; 4096 removed it entirely.

Further reading

• vLLM: structured outputs
• vLLM: engine arguments
• vLLM: OpenAI-compatible server