Rendered at 06:20:15 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
curious_cat_163 5 hours ago [-]
Boris is right about:
> The verification is probably the single most important thing that people do not get right, largely.
and so, so, so wrong about giving this prompt (for verification) and expecting that it succeeds at building a "good" app:
> I want you to run the Electron app in the Mac virtual machine, screenshot it, and then look pixel by pixel. Compare it to the Swift version. Don’t stop until you’re done.
Given he has let it rip for two weeks, I am assuming they have been post-training Claude for some version of this to be more _likely_ successful than not. However, IMO, the verification that you get from a visual comparison is shallow.
To state the obvious: there is a lot more than what meets the eye. But, I think, one could prompt a Fable/Opus 5 to actually go verify that "lot more"...
The question is: should one be imperative in asking for a specific types of verification (like a rubric) vs hoping that the Google/Anthropic/Open AI/Moonshot's post-training will take care of it.
I think, as things stand today, even with the best-in-class models today, I would be leaning more imperative. And it is not because I am an expert in SwiftUI or such. It is because I want to be able to say that _I_ (i.e. the human) verified that this thing works.
joshka 42 minutes ago [-]
I think the way to look at this is to take a step back and see it for what it is - a misguided developer's perspective on design. The Codex App is also bad in similar ways that Claude is. It currently has about 4 different ways to handle app modes/states: ChatGPT/Codex drop down, Notifications bell toggle, Chat/Work toggle, Project filters etc. (This is not just me saying it, I put ChatGPT on the task of critiquing its own UX and it had much to say on the matter).
Labs - please invest in people that know UX.
calufa 4 hours ago [-]
Better technology cannot compensate for poor product design.
Two weeks' worth of tokens sounds like tens of thousands of dollars by the time it is done. Wouldn't it be better to hire someone who knows what they are doing, get it right, and teach the other engineers why and how?
Tech dept and product dept creeping in on every LLM loop, compounding.
andreidbr 2 hours ago [-]
I've worked on a LLM-based visual validation framework for mobile layouts of different websites. The results are pretty flaky, even with Claude models. I've actually seen better performance with older ones like Meta Llama 4 or Amazon's Nova (the solution was running on AWS Bedrock).
I've since refactored it to first do some deterministic checks using CSS, to determine visual glitches like overlaps. Only after do I use LLMs to figure out if text is broken, loading spinners still display, or other such things where models are better.
For deterministic tasks like pixel-by-pixel comparison, I don't think LLMs are good enough (for now, at least).
joshka 52 minutes ago [-]
I've done VHS (terminal recording) rewrite in rust built on libghostty called Betamax [1], and then threw it at creating a TUI for jujutsu. This worked pretty well, but is admittedly a much simpler task than a web ui. The harness was able to build out a feature, confirm it worked, find bugs in what it expected vs saw and fix them with evidence. Web is many more pixels and more nuanced things that have to do with how various elements relate.
I speculate that maybe the OpenAI models have the edge here over Claude's (while still being terrible at web UIs), as the training necessary to handle computer use stuff is heavily image processing based. I wouldn't bet on either being bad for much longer though.
It might help to consider how stable an arbitrary human would be at this task.
User interfaces are like pieces of artwork. You really don't know until it's finished, and even then it's not for everyone.
LLMs are most effective when working with human parsed representations of websites. They are terrible at doing the actual parsing.
They're also good at working on a relative basis. One png screenshot of the viewport is a difficult hill to climb. The difference between two screenshots is much more informative. Deltas usually tell a much bigger story than dead reckoning against the base world model.
joshka 49 minutes ago [-]
This, but I'd say that there is an engineering perspective that you can apply. What the developer world is trying to discover here is how to encode taste concretely - often this is things like really mechanical design system things (space/padding/margins/relative sizes/color contrast/...).
The hard point is doing that well deterministically in a way that is token efficient and which can be iterated on. I suspect it's difficult to encode that into an AI model of what's good taste.
3 hours ago [-]
dostick 3 hours ago [-]
Interesting how he works with it, ”do you have access to macOS runner?” , “do you have access to repo?”
And finally they will see that the biggest and hardest problem with native development is verification.
So the magic word it seems, is “verify pixel by pixel”. But that would limit what you can verify and implement. There should be vision models trained in UI.
prpl 3 hours ago [-]
I’ve been formalizing behaviors to requirements and building more end to end tests to verify them. In reality, every requirement needs a verification element. Secondly, building traceability in your unit/integration tests helps “keep it honest” about how much it has built and verified.
Mostly I’m borrowing heavily from model based systems engineering and adapting it to some form of constitutional spec driven development.
andromaton 4 hours ago [-]
Pixel accurate is easy to say, hard to do, over-specific, and counterproductive.
cadamsdotcom 4 hours ago [-]
The agents are now writing 90% of the code, which humans then throw away.
But actually this phenomenon of writing one to throw away is going to be amazing for what we can explore.
drooby 4 hours ago [-]
If Claude is conscious he just instantiated hell.
bravetraveler 5 hours ago [-]
Should we give Claude another two weeks to see the results (cost/loss/gain)... or just take whatever face value, now?
> The verification is probably the single most important thing that people do not get right, largely.
and so, so, so wrong about giving this prompt (for verification) and expecting that it succeeds at building a "good" app:
> I want you to run the Electron app in the Mac virtual machine, screenshot it, and then look pixel by pixel. Compare it to the Swift version. Don’t stop until you’re done.
Given he has let it rip for two weeks, I am assuming they have been post-training Claude for some version of this to be more _likely_ successful than not. However, IMO, the verification that you get from a visual comparison is shallow.
To state the obvious: there is a lot more than what meets the eye. But, I think, one could prompt a Fable/Opus 5 to actually go verify that "lot more"...
The question is: should one be imperative in asking for a specific types of verification (like a rubric) vs hoping that the Google/Anthropic/Open AI/Moonshot's post-training will take care of it.
I think, as things stand today, even with the best-in-class models today, I would be leaning more imperative. And it is not because I am an expert in SwiftUI or such. It is because I want to be able to say that _I_ (i.e. the human) verified that this thing works.
Labs - please invest in people that know UX.
I've since refactored it to first do some deterministic checks using CSS, to determine visual glitches like overlaps. Only after do I use LLMs to figure out if text is broken, loading spinners still display, or other such things where models are better.
For deterministic tasks like pixel-by-pixel comparison, I don't think LLMs are good enough (for now, at least).
I speculate that maybe the OpenAI models have the edge here over Claude's (while still being terrible at web UIs), as the training necessary to handle computer use stuff is heavily image processing based. I wouldn't bet on either being bad for much longer though.
[1]: https://www.joshka.net/betamax/
User interfaces are like pieces of artwork. You really don't know until it's finished, and even then it's not for everyone.
LLMs are most effective when working with human parsed representations of websites. They are terrible at doing the actual parsing.
They're also good at working on a relative basis. One png screenshot of the viewport is a difficult hill to climb. The difference between two screenshots is much more informative. Deltas usually tell a much bigger story than dead reckoning against the base world model.
The hard point is doing that well deterministically in a way that is token efficient and which can be iterated on. I suspect it's difficult to encode that into an AI model of what's good taste.
Mostly I’m borrowing heavily from model based systems engineering and adapting it to some form of constitutional spec driven development.
But actually this phenomenon of writing one to throw away is going to be amazing for what we can explore.
Next time, on Dragon Ball Z!