Demo
CP-Agent is a simple agent loop from Stefan Szeider’s work on agentic constraint programming. I wrote it in µNorman and ran it against Claude Sonnet 5 with a Python process.
Results
| Task | Budget | Spent | Result |
|---|---|---|---|
| Sum of multiples | $0.50 | $0.0027 in 4.7s | 233168 |
| Chickens and cows | $0.50 | $0.0041 in 4.9s | 23 chickens, 7 cows |
| Same task, tiny budget | $0.002 | $0.00 | OverBudget |
Traces
µNorman logs every model call (ask) and tool call (call) with its
tokens, cost and timing.
Sum of multiples · multiples of 3 or 5 below 1000
┌─ sum of multiples of 3 or 5 below 1000
│ ask step 469 in 50 out $0.001438 182ms → 2845ms ok
│ call py/exec 0 in 0 out $0.00 2845ms → 2916ms ok
│ ask step 524 in 24 out $0.001288 3037ms → 4695ms ok
│
│ 2 model calls, 1 kernel calls
│ spent $0.002726 of the $0.50 budget in 4695ms
└─ answer: "233168"
One piece of code, one run, then the answer. The second ask is bigger because it carries the code and its output.
Chickens and cows · 30 heads and 74 legs
┌─ chickens and cows
│ ask step 489 in 130 out $0.002278 150ms → 3336ms ok
│ call py/exec 0 in 0 out $0.00 3336ms → 3425ms ok
│ ask step 598 in 65 out $0.001846 3525ms → 4929ms ok
│
│ 2 model calls, 1 kernel calls
│ spent $0.004124 of the $0.50 budget in 4929ms
└─ answer: "7 cows and 23 chickens (7*4+23*2=74 legs, 7+23=30 heads)."
I asked it to check its work. It solved and checked in a single piece of code.
Same task, tiny budget · capped at $0.002
┌─ the same task, capped at $0.002
│
│ 0 model calls, 0 kernel calls
│ spent $0.00 of the $0.50 budget in 188ms
└─ failed: OverBudget
the cap held: spent $0.00 of $0.002
The “$0.50 budget” label in the summary line was a bug in my test report. The real cap was $0.002.
The model was never called. Each step reserves its worst case first, which is
about $0.04 here. That doesn’t fit in $0.002, so the run stops before sending anything.
The same check runs before every ask and call, so it can’t be
skipped.
Try it in your browser
This is the program behind the traces. Here a script stands in for the model and Python, so it runs free and offline. The last test shows a run that stops partway when the money runs low.
(grant claude (model)) ; defaults: Claude Sonnet 5
(grant py (kernel [exec (Text 4000)])) ; a real Python process in live mode
(datatype Step
[Exec (code (Text 2000))] ; "run this and show me the output"
[Done (answer (Text 500))]) ; "this is my final answer"
(val cp-prompt
"You solve problems by writing Python. Reply with Exec to run code, and you
will be shown its output. Reply with Done when you have the answer. Write
small steps, and print what you want to see.")
;; If the Python code fails, the error goes back to the model so it can fix
;; its code. Running out of money or time doesn't go back to the model: it
;; stops the program.
(define observe (k code)
(catch (call k exec code) e
(case e [(ToolError m) m] [_ (fail e)])))
(define react (model k ctx fuel)
(if (<= fuel 0)
(fail 'out-of-fuel)
(case (ask model Step ctx 'step)
[(Exec code)
(react model k
(append ctx (list (Message [role Assistant] [content (show (Exec code))])
(Message [role Tool] [content (observe k code)])))
(- fuel 1))]
[(Done answer) answer])))
(define cp-agent (model k task)
(budget ([cost $0.50] [time 5min])
(react model k
(list (Message [role System] [content cp-prompt])
(Message [role User] [content task]))
8)))
;; Scripted, the model and kernel are canned, so this is exact and free.
(script solving
[step (reply "{\"tag\":\"Exec\",\"code\":\"print(sum(range(101)))\"}") (out 20) (latency 3s)
(reply "{\"tag\":\"Done\",\"answer\":\"5050\"}") (out 12) (latency 2s)]
[py/exec (result "5050") (latency 1s)])
(under ([script solving] [cost $1.00] [time 5min])
(check-expect (cp-agent claude py "What is the sum of 0 to 100?") "5050")
(check-within (cp-agent claude py "What is the sum of 0 to 100?") ([time 6s])))
;; The first piece of code fails, the model sees the error, and the second works.
(script recovering
[step (reply "{\"tag\":\"Exec\",\"code\":\"print(undefined_name)\"}") (out 20) (latency 2s)
(reply "{\"tag\":\"Exec\",\"code\":\"print(42)\"}") (out 18) (latency 2s)
(reply "{\"tag\":\"Done\",\"answer\":\"42\"}") (out 12) (latency 2s)]
[py/exec (error "NameError: name 'undefined_name' is not defined") (latency 1s)
(result "42") (latency 1s)])
(under ([script recovering] [cost $1.00] [time 5min])
(check-expect (cp-agent claude py "Print 42.") "42"))
;; Stopping partway: each model call sets aside about $0.04 for its worst case,
;; so with a $0.041 limit two rounds run and the third call is refused unsent.
(script grinding
[step (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)
(reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)
(reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)]
[py/exec (result "") (latency 0s) (result "") (latency 0s) (result "") (latency 0s)])
(define capped (cap)
(budget ([cost cap])
(react claude py
(list (Message [role System] [content cp-prompt])
(Message [role User] [content "Keep going."]))
8)))
(under ([script grinding] [cost $1.00] [time 5min])
(check-fail (capped $0.041) OverBudget)
(check-within (capped $0.041) ([cost $0.001] [time 2s])))
Run it yourself
Scripted tests, free with no API key:
cargo run -- examples/cp-agent.nrm
Live runs need ANTHROPIC_API_KEY and Python on your PATH. Each costs a few cents:
cargo test --test live_cp_agent -- --ignored --nocapture cargo run -- --live --python python3 --trace examples/cp-agent.nrm
The kernel runs whatever code the model writes, on your machine. Use a container or VM. LIVE-SETUP.md covers API keys and spending caps.