↘ RabbitHoleA collection for curious minds.Search the catalog
← Catalog

Are Harnesses Worth the Cost?

article
Martin FowlerMartin FowlerOctober 13, 2026
Original source ↗
Frontier AISoftware engineering

Naresh Jain compares coding agents building the same API service with and without a behavioral harness. A frontier model, GPT-6 Sol, passed every external evaluation either way. But a cost-optimized model, GPT-6 Luna, succeeded in only 3 of 10 unharnessed runs and all 10 with the harness. While adding the harness increased token usage and runtime for Luna, the harnessed Luna runs achieved the same 10-for-10 reliability as GPT-6 Sol while costing 33% less per run ($0.273 vs $0.406). This experiment suggests a behavioral harness might be a practical way to get dependable results from smaller, co

This is a source preview. Continue at the original source to read more.

Open original source ↗

Follow a connection

Related by topics and source metadata.

  1. Fragments: October 21

    article
    Martin FowlerMartin FowlerOctober 21, 2026

    Gergely Orosz is frustrated at those that entered his explicitly “No AI” essay competition and used AI. He goes on to say: When I start to suspect some part is AI about someone I don’t know: how can I tell how much of it is? And how much time and effort do I want to invest to figure out? I fear most people do not realize this last part: that when it takes time and effort from others to figure out what % is AI, and what % is YOU, then I will simply assume it’s probably 100% AI and 0% you So when producing anything that is meant for other humans, if you use even a little AI, you risk the assumpt

    Original source ↗Frontier AISoftware engineering

  2. Fragments: September 29

    article
    Martin FowlerMartin FowlerSeptember 29, 2026

    Simon Willison: The more time I spend working with coding agents, the more convinced I am that they make software engineering even harder We can do amazing things with them, but unlocking their full potential requires extraordinary discipline and knowledge This has been a constant impression I get from following Willison’s writing. While things like vibe coding get a lot of attention, the real strength of agentic engineering relies on more sophisticated techniques - and these are not easy to learn or execute. It’s a reason I’m wary of extrapolating my own dabblings into firm opinions about how

    Original source ↗Consciousness & philosophyFrontier AISoftware engineering

  3. Fragments: October 4

    article
    Martin FowlerMartin FowlerOctober 5, 2026

    In response to my last fragments (probably the bit about us worrying if LLMs have consciousness when we when we should be wondering why they don’t have a conscience) “Metalanguage” replied : we shipped the id and forgot the superego. classic software lifecycle. I don’t know what was on their mind, but their post immediately made me think of the classic 1956 movie Forbidden Planet. Plenty of sci-fi, and other literature, have explored humans creating technology with unintended behavior, going back at least to Mary Shelly. But that movie was particularly influential on sci-fi film-making and in

    Original source ↗Consciousness & philosophyFrontier AISoftware engineering

  4. Battle Drones - a vibe coded RTS with audio-based multiplayer

    article
    Simon WillisonOctober 20, 2026

    My latest vibe coded game experiment is a Command & Conquer style RTS (Real Time Strategy) game called Battle Drones ( play it here ). Unlike my previous experiments , this one is actually fun to play . I'm somewhat embarrassed to admit that I got there by telling Claude to "Make it a LOT more fun". It also supports multiplayer between nearby devices using audio to synchronize game state! Here's what I've built so far, and some of the things I learned along the way. Battle Drones How I built this v1: the initial prototype v2: full sprites v3: varied terrain v4: the fun update v5: Battle Drones

    Original source ↗Frontier AISoftware engineering

  5. My obituary for programming

    article
    Simon WillisonOctober 19, 2026

    My obituary for programming Jeremy Scheff poignantly captures the key loss caused by coding agents for career programmers who enjoyed the puzzle of developing code: Programming turned into one of the greatest jobs you could imagine. You got paid to work on tricky little logic puzzles. The trickier the puzzle, the more you got paid. But even pretty simple puzzles were worth a lot of money because regular people just couldn't do it! No, they needed us. They would give us a problem and we would solve it the way we wanted. We set the strategy, we made the timeline, we defined what was and was not

    Original source ↗Frontier AISoftware engineering