µNorman

Demo

CP-Agent is a simple agent loop from Stefan Szeider’s work on agentic constraint programming. I wrote it in µNorman and ran it against Claude Sonnet 5 with a Python process.

Ask the model→ Run its Python→ Show it the output↺ Stop when it says Done

Results

TaskBudgetSpentResult
Sum of multiples$0.50$0.0027 in 4.7s233168
Chickens and cows$0.50$0.0041 in 4.9s23 chickens, 7 cows
Same task, tiny budget$0.002$0.00OverBudget

Traces

µNorman logs every model call (ask) and tool call (call) with its tokens, cost and timing.

Sum of multiples · multiples of 3 or 5 below 1000
┌─ sum of multiples of 3 or 5 below 1000
│  ask   step     469 in    50 out   $0.001438    182ms → 2845ms   ok
│  call  py/exec     0 in     0 out       $0.00   2845ms → 2916ms   ok
│  ask   step     524 in    24 out   $0.001288   3037ms → 4695ms   ok
│
│  2 model calls, 1 kernel calls
│  spent $0.002726 of the $0.50 budget in 4695ms
└─ answer: "233168"

One piece of code, one run, then the answer. The second ask is bigger because it carries the code and its output.

Chickens and cows · 30 heads and 74 legs
┌─ chickens and cows
│  ask   step     489 in   130 out   $0.002278    150ms → 3336ms   ok
│  call  py/exec     0 in     0 out       $0.00   3336ms → 3425ms   ok
│  ask   step     598 in    65 out   $0.001846   3525ms → 4929ms   ok
│
│  2 model calls, 1 kernel calls
│  spent $0.004124 of the $0.50 budget in 4929ms
└─ answer: "7 cows and 23 chickens (7*4+23*2=74 legs, 7+23=30 heads)."

I asked it to check its work. It solved and checked in a single piece of code.

Same task, tiny budget · capped at $0.002
┌─ the same task, capped at $0.002
│
│  0 model calls, 0 kernel calls
│  spent $0.00 of the $0.50 budget in 188ms
└─ failed: OverBudget

   the cap held: spent $0.00 of $0.002

The “$0.50 budget” label in the summary line was a bug in my test report. The real cap was $0.002.

The model was never called. Each step reserves its worst case first, which is about $0.04 here. That doesn’t fit in $0.002, so the run stops before sending anything. The same check runs before every ask and call, so it can’t be skipped.

Try it in your browser

This is the program behind the traces. Here a script stands in for the model and Python, so it runs free and offline. The last test shows a run that stops partway when the money runs low.

cp-agent.nrm
(grant claude (model))                       ; defaults: Claude Sonnet 5
(grant py     (kernel [exec (Text 4000)]))   ; a real Python process in live mode

(datatype Step
  [Exec (code (Text 2000))]      ; "run this and show me the output"
  [Done (answer (Text 500))])    ; "this is my final answer"

(val cp-prompt
  "You solve problems by writing Python. Reply with Exec to run code, and you
will be shown its output. Reply with Done when you have the answer. Write
small steps, and print what you want to see.")

;; If the Python code fails, the error goes back to the model so it can fix
;; its code. Running out of money or time doesn't go back to the model: it
;; stops the program.
(define observe (k code)
  (catch (call k exec code) e
    (case e [(ToolError m) m] [_ (fail e)])))

(define react (model k ctx fuel)
  (if (<= fuel 0)
      (fail 'out-of-fuel)
      (case (ask model Step ctx 'step)
        [(Exec code)
           (react model k
                  (append ctx (list (Message [role Assistant] [content (show (Exec code))])
                                    (Message [role Tool]      [content (observe k code)])))
                  (- fuel 1))]
        [(Done answer) answer])))

(define cp-agent (model k task)
  (budget ([cost $0.50] [time 5min])
    (react model k
           (list (Message [role System] [content cp-prompt])
                 (Message [role User]   [content task]))
           8)))

;; Scripted, the model and kernel are canned, so this is exact and free.
(script solving
  [step    (reply "{\"tag\":\"Exec\",\"code\":\"print(sum(range(101)))\"}") (out 20) (latency 3s)
           (reply "{\"tag\":\"Done\",\"answer\":\"5050\"}")                 (out 12) (latency 2s)]
  [py/exec (result "5050") (latency 1s)])

(under ([script solving] [cost $1.00] [time 5min])
  (check-expect (cp-agent claude py "What is the sum of 0 to 100?") "5050")
  (check-within (cp-agent claude py "What is the sum of 0 to 100?") ([time 6s])))

;; The first piece of code fails, the model sees the error, and the second works.
(script recovering
  [step    (reply "{\"tag\":\"Exec\",\"code\":\"print(undefined_name)\"}") (out 20) (latency 2s)
           (reply "{\"tag\":\"Exec\",\"code\":\"print(42)\"}")             (out 18) (latency 2s)
           (reply "{\"tag\":\"Done\",\"answer\":\"42\"}")                  (out 12) (latency 2s)]
  [py/exec (error "NameError: name 'undefined_name' is not defined") (latency 1s)
           (result "42") (latency 1s)])

(under ([script recovering] [cost $1.00] [time 5min])
  (check-expect (cp-agent claude py "Print 42.") "42"))

;; Stopping partway: each model call sets aside about $0.04 for its worst case,
;; so with a $0.041 limit two rounds run and the third call is refused unsent.
(script grinding
  [step    (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)
           (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)
           (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)]
  [py/exec (result "") (latency 0s) (result "") (latency 0s) (result "") (latency 0s)])

(define capped (cap)
  (budget ([cost cap])
    (react claude py
           (list (Message [role System] [content cp-prompt])
                 (Message [role User]   [content "Keep going."]))
           8)))

(under ([script grinding] [cost $1.00] [time 5min])
  (check-fail   (capped $0.041) OverBudget)
  (check-within (capped $0.041) ([cost $0.001] [time 2s])))

Run it yourself

Scripted tests, free with no API key:

cargo run -- examples/cp-agent.nrm

Live runs need ANTHROPIC_API_KEY and Python on your PATH. Each costs a few cents:

cargo test --test live_cp_agent -- --ignored --nocapture
cargo run -- --live --python python3 --trace examples/cp-agent.nrm
No sandbox

The kernel runs whatever code the model writes, on your machine. Use a container or VM. LIVE-SETUP.md covers API keys and spending caps.