I help startups roll out tools for their go-to-market teams. These days, I keep coming across different products with small teams, the backing of notable VCs, and lots of potential, but then when I go to use them the UIs are just littered with bugs which prevent them from functioning. And often it has nothing to do with prompting, hallucination, or anything like that. It's simple things like "when I hit save, my data disappears."
I'm accustomed to working with buggy tools, nothing I do is mission-critical, so things aren't as thoroughly tested as a car might be before hitting the road. But it seems things are getting released with more and more bugs. Am I nuts?
Seems like there are three possibilities to me:
1. This is just what happens with products that are new to market. 2. People creating these products are relying too much on tools like Cursor that don't work right. 3. The pressure to keep up is getting faster and faster, so companies are releasing products that are less and less thoroughly tested.
My gut tells me it's a combination of 2 and 3, and this is a sign we're reaching a new stage in the AI bubble. But maybe I'm wrong and being overly cynical.
The number of times you are going to run through the new functions of your software during QA is in the hundreds. Automated testing only counts as one test because it does the same test over and over... it is a solid practice, but it is a safety net against regressions, it is not actually putting the software through new environments, scenarios, and usage.
But get that software out in the field, grow its userbase, run it for years, and you have thousands if not millions of different uses of it. Take all the bugs that come out of that, fix them, and you have software that is less buggy.
So sure, inexperienced teams put out bugs. I do, too... more than I should. Hopefully the egregious one's where it simply does not work never get to production. But the more nuanced bugs and their eventual fixes come from the software running in the wild.
So if you are the pilot user for new products, you will see all those early bugs.
Writing good software takes a good amount of time - you can catch bugs like this in so many ways but ultimately they should be being caught, tests should be failing in their deploys or their commits or whatever, but they will only be caught while companies invest in good software practices. We're in a midway point between humans being undesirable to do this work, and AI not quite being good enough by itself (yet), so I think buggy software will have a golden age the rest of this decade lol!
Haha, I liked that "the golden age of buggy software"
The tools are great such as Cursor, Windsurf, . but they only work upto a point of complexity. Beyond that, developer expertise comes into play. The tiny nuances of the application like which version of API to use, how to use them extra is dependent on the developer prior experience.
Tools like replit are idea to product and feel like it is prompt to a fixed typescript kind of stack based product.
The argument for the vibe coding side could be that best developers are able to write better prompts. But based on my experience, that stuff works only till mvp or medium complexity code and not for complete framework level code.
Also, regarding microsoft, I use Windows 11 home Edition of Microsoft and it crashed two times via updates. Although, i was able to restore it via restore point. A paid software product that doesn't cover user data protection in its warranty is not really a great product.
A lot of this is also recreating tooling. A mature framework like Ruby on Rails solves things like
> "when I hit save, my data disappears."
out of the box, because it has evolved to make these things automatic and easy to implement as part of its convention. Whatever full stack JS framework is hot for startups these days doesn't do this out of the box because it hasn't been around long enough and focuses on other things.
I m more surprised by old programs that worked being updated with obvious bugs. Right now , when you open any youtube video on a web browser, the video settings menus pop up for a second or so (probably some kind of css not hiding it). This is one of the most visited sites on the planet.
Years ago, before the Internet, software was expected to be rather buggy. It took a long time to research and fix. Releasing software was expensive, you had to put it on physical media and ship it. QA was critical role, it was way less expensive to test the hell out it and fix it than it was to ship new code. The idea that computers could be trusted to perform tasks well was easily shattered when things went wrong, and the additional optics of losing customer trust went a long way into driving QA.
Fast forward to today, and it's way cheaper to ship code with bugs to prove out an idea works than it is to spend even a few minutes writing test cases and doing even a modicum of QA.
Ultimately, it's a not just a combination of all 3 things you've mentioned, which are all contributing factors; the real problem is any level of QA before proving an idea is seen as a waste of time and money. As someone who started in tech support & QA 30 years ago, it's really tough to see.
It just seems like the idea its actually cheaper to ship broken code than invest in QA is, well, wrong.
And I'm just thinking about the amount of time people spend paying me to work through these bugs with the team at the startup building it.
Putting aside the whole "it's easier to keep a customer than it is to acquire one" and "easier to keep your good reputation that overcome a bad one" side of things (and there is one tool a client asked me to use that I told them either the tool goes or I do),
Your argument certainly holds water for a company that is thinking long term. But too many individuals within organizations are thinking short term instead.
Ship out the product, even if it isn't ready. Meet your performance goals for shipping on time. Collect your bonus and exercise your stock options. Bail to another gig before the consequences for the buggy product hits the fan.
Aligning incentives is a very delicate art.
What’s mind blowing to me is not the tiny startups, but the megacorps like Microsoft putting out mission-critical cloud infrastructure code that’s just plain broken.
A recent one was that Azure Backup and Azure Update don’t work in combination! If you restore a VM, it can never again be patched by the update system. (It won’t recognise the OS as supported and just refuses to do anything.)
It has been reported, but nobody cares. I was told by support that this is a “will not fix” because it’s not a requirement.
There is no way anything this catastrophically bad would have made it past QA back when Microsoft used to have an actual QA team.
I swear there is a significant portion of development teams out there that are secretly pleased as punch that they can just push whatever garbage they want straight to production without the pesky people in QA “just complaining” and being “obstructionist”.
Seems like an example of software being written for a powerpoint bullet list / sales meeting, rather than real world use, which is often/generally the case.
Backup: check!
Update: check!
Backup + Update both working at the same time - not on the list.
“We met the requirements!”
The culture of startups is to put out slop then validate it as having demand before taking it seriously.