Reposted from original: https://crawshaw.io/blog/programming-with-llms
HN discussion of original a few days ago: https://news.ycombinator.com/item?id=42617645
I've been using LLMs the same way this past year. As a challenge to myself, I've implemented full-stack interview evaluations using languages and tools I've never worked with before and see what I can do. Using LLMs, it hasn't taken me more than 50% longer to craft a solution as compared to my crafting one using tools and technologies with which I'm intimately familiar.
Some of that extra time was a result of learning the craft of using LLMs in this manner. With each new technology stack, that excess amount of time decreased.
Now, let's be honest, you still have to have excellent development skills, and it helps immensely to have a good knowledge of software architecture and design. I say that to point out that non-developers could not achieve these results. In the right hands though, it can make you a lot more productive.
> I've been using LLMs the same way this past year. As a challenge to myself, I've implemented full-stack interview evaluations using languages and tools I've never worked with before and see what I can do. Using LLMs, it hasn't taken me more than 50% longer to craft a solution as compared to my crafting one using tools and technologies with which I'm intimately familiar.
The problem with this is that you don't know what you don't know. I've actually used Claude to build a project in a framework that I haven't worked with before and it got me pretty far. Once I got that reviewed by someone who actually knew the framework it was terribly structured, not using higher level constructs to organize things, wasn't using idiomatic approaches - stuff I could easily confirm myself after going through the official docs.
I think using LLMs to skip learning from official sources is a bad move, you're bound to hit a wall with LLM (at the very least the training data cutoff could be before something changed in the framework), you'll have to deep dive into the docs and figure it out.
That is to say - sure you can pass the interview with LLM only - but you'll suck at the job. If you invest some effort upfront to learn from traditional sources and then use LLM to speed you up it's going to pay off really fast IMO. The more I know about what the output should be the more useful LLMs are.
Right. I've had that same problem when trying to build a new app with Claude / 4o. I essentially have to go through the docs myself so I understand the framework structure enough to write up an outline for the llm to follow when developing a new component or backend structure.
With Claude I've had great success with first uploading the docs so that it has them as context for the rest of the session. Without doing that I got all kinds of answers, sometimes using a different framework or an older version of the framework I was using. Once it had the documentation to reference it started giving a lot better answers
Great point. Great suggestion.-
Very interesting outcome. Shows how far it’s possible to get without being idiomatic.
You say the docs contain guidance on structuring, high-level constructs, and how to make things idiomatic. It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”.
If it picks up the same things that’s great news for novice programmers and anyone new to a framework!
LLMs will improve at making these fixes over time - even if they’re currently bad at it that won’t last.
> It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”.
For your info, a few days ago, I was trying to migrate Tailwind 3 to 4 in a codebase where I had just set up the boilerplate, and I went through hell and eventually gave up. I used the Tailwind 4 documentation and Claude 3.5 sonnet in Windsurf IDE, and the part related to migration in my codebase was probably less than 50 lines. All of those tasks would take just a few minutes if a person did them directly (excluding reading the documents)
But that's kind of my point - when I was originally working on it - I didn't know there's a more idiomatic approach - why would I copy paste the docs to ask it to improve the results ? Especially given the LLMs tendency to hallucinate and gnerate noise when asked to "improve code".
Also without reading the docs would I be able to spot why the idiomatic approach is better from single use case ?
You're basically depending on LLM to make choices for you - and in my experience so far it makes very suboptimal ones.
Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems.
The focus on "terrible structure" misses the point - what matters is whether the system meets its requirements efficiently and can be maintained effectively. Have you measured any actual negative impacts on system performance, maintainability, or customer value? My experience suggests that starting with working code and iteratively improving it as patterns emerge often leads to better outcomes than delaying development for complete framework mastery.
The interview analogy is particularly misleading - success in a software engineering role isn't measured by framework knowledge, but by the ability to deliver value to customers effectively. Learning framework idioms can happen in parallel with productive development, especially when using LLMs to accelerate the initial learning curve.
> Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems.
Except that using the original approach would make it hard to navigate the project after a few weeks of development - duplicate logic all over, inconsistency in duplicates, bugs. Rewriting the code to actually use the framework as intended let me progress much faster and reuse the code.
And as someone who has had gigs with 6 different languages/stacks at this point, and played with probably as much on the side - that's a nice sentiment in theory but doesn't reproduce in practice. There's definite learning curve when using a new stack/language - sure your experience makes that curve different to a newbie, but in my experience it's going to be a few months until you can move as quickly as you can with a stack you're familiar with.
While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems.
To a first approximation, you will never be the person maintaining the code you wrote. So if that person who takes over from you, who is well versed in the idioms of the framework in question, cannot easily understand what you have done and why, then it isn't maintainable software.
> While framework idioms are important, they're secondary to delivering working, *maintainable* software that solves real problems
Emphasis maintainable
The truth is that if you are not following framework idioms, you are very likely not delivering maintainable software
Is it too much to ask for a "How I use LLMs to program" article not come from someone who is actively marketing their AI startup?
Or at least call it out as a disclaimer? I read this whole article a few days ago when it came by on this site and that would've been real nice to know.
Really puts the whole thing in a different perspective, knowing he's FUCKING SELLING ME SOMETHING.
I'm so sick of this!
You have touched upon something important here. I have spent the last six months thinking about the role of disclaimers in how written text is perceived. Disclaimers, those seemingly innocuous statements meant to protect the author from liability or misinterpretation, often have the unintended consequence of undermining the credibility and impact of the work itself. By preemptively acknowledging potential flaws or limitations, authors can inadvertently signal a lack of confidence in their own arguments. This can lead readers to question the validity of the information presented, even if the disclaimer itself is minor or irrelevant to the core message.
The very act of issuing a disclaimer can create a sense of distrust. It suggests that the author anticipates skepticism or potential backlash, raising questions about their motives and the integrity of their work. This is particularly problematic in academic settings, where disclaimers can undermine the authority and rigor of research findings. While acknowledging limitations is crucial for responsible scholarship, the way these limitations are framed can significantly impact how the research is received and interpreted. A carefully worded disclaimer can acknowledge potential weaknesses without undermining the overall strength of the argument, while a poorly constructed one can create unnecessary barriers to engagement and understanding.
For this reason, I created Craft Your Disclaimer, an AI-powered service to help creators craft disclaimers tailored to their goals and audience preferences. You can register for free at: CraftYourDisclaimerOrSomeOtherBS.ai
Just a quick note that the title is "How I Program With LLMs", this is currently missing "How" which changes the meaning a bit, at least for me.
If i am not mistaken, HNs automated, somewhat pedantic filter/formatter at work.
The rationale being, it tries to reduce any click-baity content by making titles a bit more serious/boring.
Me as well. But sometimes I like the hallucinations of aider and other tools and just let it iterate over and over and over. Kind of fun to see. But by no means a drop in replacement for actual knowledge.
It works much better when you already know what you're doing-- so you can ask for exactly the right thing, spot the LLMs errors, and know what you need to do to fix them rather than iterating forever with the hallucinating machine.
I can't help but feel that while these tools speed everyone up they actually increase the difference between an expert and a novice rather than leveling the playing field.
Steve Yegge has laid this thesis out: https://sourcegraph.com/blog/the-death-of-the-junior-develop...
He also has some good content out there on how to program using LLM's. He calls it Chat Oriented Programming, aka CHOP. That's a good term to Google for.
I enjoyed the examples.
My previous company's lawyers really had success writing complex contracts in LLMs.
The writer one really hits home: a friend who's a writer is able to make amazing stuff with ChatGPT, because he knows how to do it himself. I can only make stupid shit, because I while can recognise quality writing, I don't really know why.
Similarly, I see the same thing happening with junior devs: a lot of them just can't get the answers I get out of Copilot or ChatGPT. I have to sit next to them and tell them what to ask the LLM, or when to say no to a proposed solution.
Similarly, I can get started myself on Computer Vision and Machine Learning projects with LLMs, but I need to hand over to a proper CV or ML Engineer to get the last 10% done.
> Similarly, I can get started myself on Computer Vision and Machine Learning projects with LLMs, but I need to hand over to a proper CV or ML Engineer to get the last 10% done.
Unless by "Computer Vision and Machine Learning" projects, you mean you are trying to develop new approaches to computer vision or ML, I don't see why you would need to involve a CV/ML engineer? The stuff is all generally packaged as re-usable libraries like everything else, no?
Exactly. These tools test whether we're brilliant managers, understanding the potential and limitations of everyone we manage, and understanding the work well enough to do it ourselves.
I too am using AI to play with new languages and problem domains, and to write tools I've long wanted but never found time to implement. I find it helpful to ask other LLMs about best practices, with no conflict of interest where they're trying to write me code.
For my initial experiences with Windsurf Editor, the AI got ahead of me and then into trouble it couldn't resolve, thrashing in ways my daughter learned to avoid in her first semester of programming. I've found Claude 3.5 Sonnet understands the problems I work on best, so I'll use Cursor for access to this AI. Workflow enhancements are secondary to the underlying intelligence.
I think I agree. It might take me 30-60 min to look at a Figma mock and transfer a form with several different HTML elements then wire them up to an API service on the front-end. I can do that now in 5 minutes by simply having AI look at the mock and generate the code for me, where I spot the errors right away due to my level of experience. It looks exactly as professional code should look from a senior+ engineer. I've got things honed down now to where it maybe takes 2 iterations to get exactly what I'm asking the LLM to do, but I'm often getting it on the first prompt now or maybe making a couple of one line changes in manual tweaks.
I think junior devs could easily get lost in the output if they're not familiar with what it's doing and why it's doing it.
- [deleted]