Call Me Maybe
Makes a small language model (Qwen3-0.6B) choose valid function names by controlling which tokens it is allowed to produce at every step. Argument values come from rules, and Python builds the JSON.
The problem
Small models often break the JSON format when asked to call a function. Instead of asking the model nicely, the decoder can make wrong output impossible by only allowing tokens that continue a valid answer.
What I built
- Logit masking. At each step, every token that cannot continue a valid function name gets a score of minus infinity, and the model picks the best of what is left.
- Function names are turned into token sequences and matched by prefix. When only one function remains possible, generation stops early. The token tables are cached.
- Arguments are extracted with patterns and the model picks between candidates, using the same masking.
- Pydantic validation of inputs and outputs, with clear errors for missing files and bad JSON.
- A step-by-step visual mode that shows which tokens were allowed at each step.
Decisions and trade-offs
- Guarantee the format, not the meaning
- Constrained decoding makes sure the output is valid. It cannot make the chosen function correct, so validation and tests still matter.
Limits and next steps
Argument extraction currently uses rules written for the given set of functions. The next step is to generate arguments token by token under a schema, so new functions work without new rules.