When IDEs first emerged in the world of web development, many developers proudly claimed that a simple text editor was all they needed. For them, saying "I'm using Notepad / VIM" became a badge of honor, symbolizing their minimalist approach.
Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.
> When IDEs first emerged in the world of web development, many developers proudly claimed that a simple text editor was all they needed. For them, saying "I'm using Notepad / VIM" became a badge of honor, symbolizing their minimalist approach.
Whenever some new technology is being sold, someone always tries to delegitimize any criticism by citing some superficially similar criticism of some past technology that was adopted. Or, fairly frequently, they just reflexively cite Socrates's/Plato's OG critique of writing.
> Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.
So your message is: "Who are you going to believe, the sales pitch or your own lying eyes? You should focus on selling yourself on the new thing."
I think you should listen to the criticism, and try to understand and empathize with it instead of trying to dismiss and disregard it. There's something there. Maybe the people who critique it are the careful and thoughtful developers, and the ones who sing its praises are the sloppy ones who aren't careful enough to see the problems or just don't care (e.g. the kind who see their job as closing tickets not writing working software)?
Maybe this thing will get adopted despite its shortcomings and change everything, like NFTs did, but that doesn't mean those shortcomings aren't important.
That's an interesting comparison because it seems to me that there are still a significant fraction of experienced developers using text editors like vi and sublime, without any apparent loss in productivity. Also, I use VS almost all the time but have had to learn how to bypass it when there are errors that aren't flagged up by its particular build process. I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good practises for refactoring, debugging etc... until suddenly they do. Not saying it's a bad thing, just that you may not need to worry too much about "losing out" if you're not on the hype train.
I think there might be a few things to consider here:
1. Is it without any loss in productivity? Or are they so productive/skilled that they can still outpace most other devs even without the IDE's additional functionality?
2. Is there a loss in productivity, and they just don't realize it? Just because they can still do a good enough job without an IDE doesn't necessarily mean that they couldn't do a better job as experienced IDE users.
This is similar to the arguments people make against using Rust. "I can code C just fine and never have any issues, I don't need Rust"
It's also similar to this Copilot discussion. The people who say "I don't see any benefit from LLMs when coding", could it be that they just... don't have enough experience using LLMs? We've -just- had that come up in HN[0], where everybody in the comments was showing how they got the answer just fine out of their LLM of choice, while the OP was all about how AI couldn't possibly understand that bit of code.
My intuitive understanding is that a lot of an IDE's functionality - e.g. running builds, common refactorings, or searching for instances of a class - can be performed at a similar speed by someone who has a toolset of commands and shortkeys at the tips of their fingers. And if you do grok those commands, you can probably do even more flexible things than your IDE will easily allow. I also find that using an IDE means you can easily forget about how e.g. config files factor into the build process, which can make it harder to on the fly diagnose certain kinds of issues that depend on configuration.
But I guess you're right that we can't trust word of mouth and don't know for sure until/unless we have an agreed measure of productivity and can run with/without or before/after type experiments.
> I also find that using an IDE means you can easily forget about how e.g. config files factor into the build process, which can make it harder to on the fly diagnose certain kinds of issues that depend on configuration.
This is a fact that needs more attention, and it's why I prefer to onboard people with setup documents they work through rather than a canned working development setup, even though the former takes "longer" to get people up and running. The long term benefits in forcing people to touch and familiarize themselves with as much as possible far outweigh any short-term time gain.
Give competent developers - i.e. people trained to solve optimization problems in their sleep - a metric to optimize for. Call it "productivity". See what happens.
For sure, productivity is easy to define loosely but extremely difficult to measure exactly. Thinking for two weeks and then making a one-line change might be more profitable than churning out any amount of high quality code.
Yes, Vim and Emacs have been "IDEs" in all but name for decades. They just do things in a slightly different way which is arguably less "integrated" and more "modular", but feature-wise it's pretty much identical, and has been for a long time.
>I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good
I can't think of a worse idea than giving copilot to a junior dev... I use chatgpt a lot, but I've got lots of experience. One has to understand how the big picture works very well to not find yourself led down the wrong path.
Most non-technical business stakeholders don't really understand what we do. You don't see shows about awesome devs doing awesome shit, like you do for lawyers, doctors, business people, etc. All they know about us is that we're nerds who work like two hours a day and can't be bothered to go into the office.
They do, however, understand ChatGPT and how it can be used to create whatever you want (see also: Eric Schmidt's missive to Stanford business undergrads that made the front page here some time back).
Given this, they are EXTREMELY incentivized to give/force upon ChatGPT to juniors. If they can give this to juniors and maintain feature velocity with an acceptable amount of quality degradation, then they can eventually move software development to lower-cost countries with weaker worker protections wholesale (and use good old fashioned protectionism to make sure that those countries don't try to steal our slice of the pie).
Quality issues can be fixed by expert consultants who know their shit, but that market won't be big enough for all of us.
> Most non-technical business stakeholders don't really understand what we do. You don't see shows about awesome devs doing awesome shit, like you do for lawyers, doctors, business people, etc. All they know about us is that we're nerds who work like two hours a day and can't be bothered to go into the office.
I was there when IDEs first started becoming popular amongst my cohort. Yes, there were people like that who prided themselves on their hard-earned skills.
But there's a key difference here.
I don't remember an IDE trying to autocomplete something which didn't exist. Similarly, an IDE never pointed me to documentation which it had just invented. If it had, I'd've been one of those people living out my days in Nano.
If my IDE autocomplete sometimes generated invalid syntax, I would throw that IDE in the trash and never do business with the people who made it ever again.
Writing the code is not where I spend most of my time: it's in planning, optimizing, and debugging. AI only helps modestly with the second and third of those right now, and it certainly doesn't need to be in my code editor to do that. In fact, I spent a lot of mental cycles when I tried copilot removing potential bugs it was hallucinating into the code in the first place.
IDEs made it easier to navigate a huge codebase with things like autocomplete and one-click refactor. This made it easier to make huge codebases that _require_ these QoL add-ons over time. I think this was a factor that made outsourcing software development (to countries with weak or non-existent worker rights) possible at scale.
In a similar vein, highly paid software engineers are using LLMs to generate more and more of the code they ship.
If highly paid SWEs can write code by shoveling a sentence into Copilot, _literally anyone can._
We seem hellbent on destroying our profession for the sake of "productivity" (i.e. using less of our brainpower to churn our features faster).
But what do I know? I'm just a simpleton that uses vim and writes code the old-fashioned way.
Exactly. As a solo entrepreneur, using ChatGPT to produce an acceptable MVP to test PMF makes all of the sense. People used to pay big money for that.
Whether you scale that strategy once your business becomes viable is dependent on whether you give a shit about software quality or not. That's the long-term concern I have with this flavor of AI.
I find IDEs add complexity to development, even today. They're very bad at hiding irrelevant information. There are so many buttons I've never used in Visual Studio that are always displayed, it makes it very visually cluttered and harder to find the buttons I actually need.
And configuring tools through IDEs is very opaque. How do I add a flag to the compiler command line in this IDE? Who knows. How do I change the CMake config? Maybe I can edit the CMakeLists directly, or maybe that will make the IDE blow up, so I have to use some arcane GUI built atop an already arcane text UI.
Plus, you can usually forget about scripting your workflow. Unless the exact workflow you need is built in to the IDE already, you'll be clicking the same sequence of buttons over and over ad infinitum. An IDE is one big complex tool in place of a handful of simpler composable tools, and the only way to extend an IDEs functionality is to make it more complex instead of adding another simple tool or composing existing tools in a different way.
AI is. a tool. a power tool for sure but ultimatly it multiplies what the developer brings to the table. I personally have found it of limited use outside a few situations such as knocking out a bunch of boilerplate.
That said, I would keeo this thing the hell away fro junior engineers and I wouldn't want a junior who has used AI his entire career anywhere near my codebase. AI is gasoline and without someone with experience behind the scenes, they are just going to be 10x faster at knocking out unmaintainable spagetti code.
When you run a saas company, code is only ONE factor among many you are maintaining. The other arguable more important one is the domain specific knowledge that exists in the heads of your talent. Give me the nerd that uses emacs with 5+ years of experience thinking though and solving problems without AI. I'll let him have copilot because he'll be the one writing and maintaining the code. More importantly, he'll have teh experience to know when copilot is leading him astray.
The biggest speed up was when I was refactoring some code and it “learned” the pattern I was moving to. It felt great!
. . . Except it got one so wrong I couldn’t trust it and had to be just a meticulous as I would’ve been in the first place, so that was a net loss of time.
As a solo full-stack developer with 20 years of experience in PHP, I’ve found that using Copilot for my latest project (Laraval app) boosts my productivity by at least 50%. The multi-line autocomplete often nails exactly what I’m trying to achieve. I can easily verify its accuracy as I've written the same code multiple times over the span of my career and it is way faster than typing the same lines.
In addition to Copilot, I frequently use ChatGPT to get quick answers or examples, even beyond Laravel's already excellent documentation. It’s also incredibly useful for quickly learning the basics of setting up the infrastructure for my projects, saving me time and effort in the process.
I am an also an experience developer on the Python side and find it to be the same. Its hard to quantify the amount of time but the autocomplete does a really great job most of the time.
What is it auto completing? Most of the common things I share between projects are in the form of libraries, there is really no need for ai auto complete. Don’t get me wrong, ai is incredibly useful, but more for understanding things than autocomplete.
It's auto completing everything. Not just words or lines, but multiple lines. And it is way faster to read and see if it's correct than to type it all yourself.
That’s what I’m struggling with. I might write a micro service over and over again with spring boot but they are all different. The frameworks take care of all the boiler plate already. Why are we having ai generate boiler plate ? It seems on the surface to be a bad practice or a code smell.
I'm very optimistic when it comes to AI. I use chatgpt all the time at work and for personal code, but copilot just feels much worse. Perhaps I just gotten used to prompting and guiding (and copy pasting code) to the model in a way that gives me exactly what I want. Would I love an Ide that automated it? Sure, but none does it well!
The single biggest use I've found for LLMs has been search or lookup functionality. Instead of a carefully crafted google search + "reddit" I can just ask a question and get a specific answer with references. Some times it's wrong, generally not.
Runner up is copilot integration in the IDE that I ask to transform data or reorganize lines of code. It's grunt work and having smart automation around it is helpful.
Anything requiring "thinking" generally falls over pretty quickly. For example I've found broader feature code generation has generally been crap or worse and often using APIs that just don't exist. Catch is that LLMs have been sold as thinking machines that will replace people and VCs have thrown all their money into that fire.
P.S. VIM has a growning following and vscode user counts are astronomical, so I'm not sure your point stands.
>
The single biggest use I've found for LLMs has been search or lookup functionality. Instead of a carefully crafted google search + "reddit" I can just ask a question and get a specific answer with references. Some times it's wrong, generally not.
This is the use case that I struggle with the most.
Let's say that I want to know how to reheat a baguette. If I throw that into ChatGPT, it will give me an answer based on...who actually knows? It could be from the set of responses it found most often in its training. It could be from reddit.
It could be mashing these up into something completely incorrect.
Perplexity and Kagi Assistant are better in that they give you some of the sources they used in generating an answer, but now we're back to a search engine with extra steps.
With a search engine, I can go through each source and know (a) that the answers aren't altered and (b) who wrote the answer and whether I should trust them.
To wit: I once used ChatGPT 4 to find me good restaurants in Houston based on /r/houston subreddit, with sources. (This is the best way to find new restaurants in my opinion.) It gave me five recommendations that (a) were not the best recs and (b) had fake Reddit URLs for sources.
Umm, hi? I still use emacs, and I'm one of the top developers in my field. Never figured out how one of those IDE would be more useful for what I do, which is to develop novel low-level algorithms.
Also, neither vim nor emacs are simple text editors.
Quite frankly, IDEs at the time were really bad. They were slow and clunky. Intellisense didn't always work. People who relied too much on IDEs didn't understand their project's build process and couldn't debug it. The code templating stuff spit out awful code.
I kept using Emacs for a long time because I was more productive with it. I had a lot of customizations and add-ons that made it a dream to browse the codebase and churn out good code.
I finally switched to VSCode some time after it was open sourced because the plugins for my languages had gotten really good, and emacs development felt like it had slowed down and everything was starting to get bitrotten.
I don’t know any web developer who have ever used vim. And the IDE alternatives were table generators which made the worst code ever by ignoring every CSS rule, so I don’t really understand your argument.
As for IDEs in general, they only propose functions that do exist.
-> GitHub's Copilot lies about its documentation. Why would I trust it with my code
This is the biggest misconception people have about AI. You shouldn't trust a Copilot by default, but more often than not it will be very handy if you know what you do.
You can use it as a "better" rubberduck and bounce ideas back and forth. Just with like a coworker, wich also is not 100% right.
Or sometimes I use it for languages which I'm not god at, but still can read it. I'm a 10+ year Java programmer. But sometimes I want some small bash / python / powershell script.
For example a powershell script to change power state profile. It spits something out in seconds, I can read it, understand it and ask for changes.
With a large java project I don't use it much. But sometimes, and then I also read the code the check if it does what I want to happen. Sometimes I can just describe a problem in English, copilot will create something that I think is wrong, but it gives me an idea on how to solve it myself.
The coolest thing copilot or whatever did in vscode for me was generating a block of boost::regex code for c++. I started typing it, and it generated a block of 10 or so lines that would have been 10 minutes or so of deciphering boost docs, writing a simple test, failing, recompiling, etc. There is nothing worse than dealing with regexes, and it wrote the function for me, and then I wrote tests and verified it did what I needed.
You don't have to use their brand name, you can call it whatever you want. I similarly like to use the term "ClosedAI" because I find it is easier to understand than the closely named brand.
I think the same. it very often has some reasonable suggestion, then I mostly need to fix some minor things and it's good to go.
overall it definitely helps to reduce the amount of time required to write code
Is it surprising that asking a coding tool that writes code to manipulate its own settings in a way it was not built to do produces bad output? Can you ask your Google Home to delete your Google account? Or tell Siri to turn off Hey Siri?
This is specifically the version of Siri that now _uses_ semantic retrieval over learned embeddings of existing corpus of documentation. There's no reason Github couldn't implement something similar (except that it's not a high priority for them to do so).
Its not that surprising. These are two very different technologies. Alexa/Siri point to explicit functionality and the LLM chat in this example is open ended. Luckily I have been using the internet for a few years, I saw the triple dots and clicked those and immediately saw one option was to hide the Copilot bubble.
To each there own, I have been using these tools for a while and have a grasp of what is possible and not.
Not really. GH advertises that its Copilot can go through documentation and fetch relevant results. I don't see anything wrong in asking the copilot to fetch any kind of results from its own documentation.
I mean, even if you live coding assistants and use them often (I frequently consult sonnet on random things I can’t remember how to do off the top of my head)…
…you have to admit that this is just outrageously shit.
Come on
You’re sending a response from a help bot to a user with a url in it and you didn’t bother to add a check to make sure the links it returns actually exist.
That’s just lazy. I wouldn’t want to use it either.
I don’t.
Or any product where the developers have been in such a rush to pour AI into their products that they’ve failed to integrate it in a way that is useful and meaningful.
It doesn’t seem like such a big ask.
If you’re integrating AI and you’re not doing the most basic output sanitation… you deserve responses like this.
From github? With their resources and experience? It’s a fuxking disgrace.
This is a bad faith argument, OP is looking for reasons to distrust Copilot. Let's say Copilot returned a correct link. Would OP then decide it's worth trusting Copilot with their code?
I doubt it. That's a horrible proof of trust. It really feels like the only outcome of this 'test' is that copilot can prove itself untrustworthy.
It will be interesting to see as years go on how the aging population that get so worked up over LLM coding tools are doing. The new generations are for the most part embracing it with open arms and no discrimination. I see the many flaws in these tool but I still find them useful. Maybe there is a plateau but if they continue to improve these tools can be incredible force multipliers.
Why do we focus on the promise of a technology instead of its state as is right now ? How many time should we be duped before we learn from our mistakes?
The burden of proof is on the new generation, not on old timers. And this is coming from a new generation software engineer.
I also find their new UI changes devoid of taste. Really? Putting a stupid chatbot button where the notifications button usually is? Fuck you.
I will some day in the near future migrate my projects off of GitHub. It was nice having a centralized place for open source projects for a while, though; a lot more people saw stuff I was working on than otherwise would if I just had a cgit instance running somewhere. But that's okay: the point of me publishing it wasn't for other people to see, necessarily.
And I'm sure eventually there will be another anyway. I certainly remember how Sourceforge went downhill.
Bing chat does something similar. It suggests going to the settings for the unrelated, now-defunct product also called bing chat. These settings no longer exist and there is no way to turn it off.
Does it predict the next most likely word or sample proportionally from the space of possible next words? I don't get consistent results, so I assumed it generated a sequence by sampling. I guess I also don't understand how these things work.
As Open Ai notes here are the limitations of ChatGPT
Limitations
ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows(opens in a new window), rather than what the human demonstrator knows.
ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.
The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.1, 2
Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.
While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.
...and the AI zealots will insist that there is nothing wrong with these systems spouting disinformation like this, as it's just a speed bump on the road to progress.
“One of my intern developers can’t tell me the URL of their own latest blog post, why would I trust a different intern to write code without supervision or review?”
If one of my interns lied about something I asked them, why would I trust the same intern with literally anything else?
A key skill is learning when to say "I'm sorry, I don't know that."
AI is currently trained to be overly friendly and helpful - to the point of deception. They often "hallucinate" entire software packages. At which point, they become worse than useless.
If your intern delivered false information or wrote broken code as Copilot is wont to do, at the rate Copilot manages to, you would fire that intern. Especially if they argued with you that their completely made up solution was, in fact, real.
How much of this could be measured as a user using the tool incorrectly? We hear these complaints and while I have definitely experienced incorrect code, they feel more like blips that I overcome in seconds.
If I bought a hammer that occasionally instead of striking and driving the nail instead turned itself back-to-front and clawed the wood, I'd return it because it's broken. The how or why it does it is immaterial. It's a tool. If it doesn't perform it's task, it's very Reason for Being, then it's worthless to me.
If you tolerate broken tools, that's fine, more power to you. I don't. If shit doesn't work it goes in the bin.
I guess I have the advantage of having used the majority of these various tools and found workflows that work quite well. Sorry you are incapable of attacking people in your responses and having a reasonable dialogue.
Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.