Harness Engineering Pt. I (and the first blog post!)
Hi, there! This is Flávio and that is my very first blog post ever! In this blog, I will be sharing my professional experience - things I’ve learned along these 10 years of career, news on tech, as well as projects I might be involved with. Here, you will see content about Software Engineering, System Design, DevOps, AI/LLM, and whatever crazy new thing is going on in our industry! I truly hope you enjoy and can learn something out of here!
Yet another post about Harness Engineering
Yes, I know, this subject is everywhere now. At this point, you already know what Harness Engineering (HE) is about, what it solves, and how to build your own harness pieces. I’ll leave Addy Osmani’s post as a good overview and handy reference in case you’re new to HE. My goal on this series of posts is to deep dive on how I’ve been using HE on my day-to-day as a Software Engineer.
The AI-assisted coding vicious loop
Everyone says that ever since generative AI, we are becoming 5x, 10x, 100x engineers. To truly achieve that, one should not only open its coding agent of choice, start asking to “create a new page using React to manage company resources stock. Add necessary endpoints on backend too. Make no mistakes.” and hoping that it will work. Ok, let’s be honest, it will probably work - you’ll see hundreds of code lines being generated on your terminal/IDE, the agent will even execute docker compose to start the application, and voilà, here it is your request, materialized in front of you.
Generally speaking, you’ll get there. Even when it does not reach on the first time, if you are patient enough, you’ll drive your agent to finally deliver what you expected. And you will feel productive: a task that seemed to be 3 days long was delivered in one afternoon. That’s when things start to turn to the wrong side. After reviewing your agent’s output (or worse, when a teammate reviewed your PR in detail), you will figure out that, best case scenario, you have a prototype, not a final solution. Tests are missing, typing is not properly enforced, your application’s layering was completely disrespected, code is repeated everywhere, and so on.
Once again, you will feel like “hey, I still can use LLM to close those gaps, and I’ll be delivering much faster than before”. That’s the second strike: you’ll ask your agent to cover touched files with tests, to ensure typing, to follow project’s layering… but things don’t come out as expected. That’s because you did not give your agent the complete instructions to achieve 100% of your expectations, and this is probably because you have never needed to think how many split second decisions we do daily while writing code, so prompting with that depth is not easy on the first time.
Until you get there. You finally find a set of instructions that defines your way of coding, in a consistent, reproducible way, and now you know how to properly ask your coding buddy to do things. Repeatedly, over, and over again. Whenever starting a new task, you either see the agent making the same mistakes, so you need to correct it every single time; or you foresee its default behavior and proactively prompt your new API in detail. The problem is that it does not scale: you are really not that productive if you need to hand-hold the agent at every single task - you’re overseeing a Junior engineer, not training it to be a Senior+.
Weren’t we talking about Harness Engineering?
That’s exactly where Harness Engineering kicks in! Harness is defined as:
everything in an AI agent except the model itself
The harness goal is to constrain the model in certain ways that it would not do intentionally 100% of the time. The idea is to wrap around the model with context reminders (probabilistic - agent can reason over it and go on another direction) and self-correcting hooks (deterministic - impede agent of not respecting some constraints), leaning the results generated by the agent towards the direction intended.
This is where the 10x engineer can come true: once you’ve defined the set of mental models and quality gates we apply on our work, we can create harness pieces to make the coding agent aware of that, so that we don’t need to instruct them anymore - at least theoretically. Of course, models will evolve, the harness around your coding agent will evolve, and maybe the way you code and architect software will change, so new custom harness will need to be added, but this is the most reliable way of letting the LLM correct itself and reaching a defined objective.
What’s next
Given this introduction, in the next post of the series I’ll be talking about some skills I use on a daily basis. Next in the series, I intend to show how I use hooks to ensure a quality gate. Moving forward, I will use the series as a way of showing my advances in creating a workflow for long-running tasks, from conception to completion, ensuring that all intermediary goals were achieved.
See you in the next post!
AI-assisted review applied: grammar and syntax, subject-verb agreement.