Hit or miss in my experience so far.
It's more like a junior dev with an encyclopedic knowledge of GitHub, and that took a sabbatical in 2021 until now so anything new is completely out of its grasp. This last point is a huge deficit when it comes to quickly moving subject matter, like JavaScript libraries and ironically AI itself, eg. langchain.
Some examples:
- Incorrectly creates AWS SDK code in v2 examples instead of v3, makes a *huge* difference. But then if you ask it to convert some v2 code into v3, it gets it right the first time.
- Exceptionally detailed, well-commented, apparently correct code that is completely (or worse, subtly) wrong. And then even after a laborious debugging session back and forth in the chat, still incapable of getting it right.
- The output is only as good as the input, so you have to know what you're doing to ask it correctly.
- "Rewrite this Python in TypeScript" -> Hell yeah, worth the price right there.
- Writing unit tests for anything beyond a single function will be hit or miss.
- Shell scripting: particularly good at this for some reason, but it will occasionally need some help. I used it to write a set of shell functions for assuming AWS roles and manipulating Parameter Store, and it took care of all the BS shell garbage that I can't be arsed to remember like argument parsing and the like.
All development shops are very resource constrained. Always. GPT is turning into a tool that relieves some of that constraint, but we still need people behind it.
Now, give it a few years, and the more rote stuff could disappear. No one is going to be doing unattended development with it for some time though.
It's not great for operationalizing stuff yet either.
Some examples:
- Incorrectly creates AWS SDK code in v2 examples instead of v3, makes a *huge* difference. But then if you ask it to convert some v2 code into v3, it gets it right the first time.
- Exceptionally detailed, well-commented, apparently correct code that is completely (or worse, subtly) wrong. And then even after a laborious debugging session back and forth in the chat, still incapable of getting it right.
- The output is only as good as the input, so you have to know what you're doing to ask it correctly.
- "Rewrite this Python in TypeScript" -> Hell yeah, worth the price right there.
- Writing unit tests for anything beyond a single function will be hit or miss.
- Shell scripting: particularly good at this for some reason, but it will occasionally need some help. I used it to write a set of shell functions for assuming AWS roles and manipulating Parameter Store, and it took care of all the BS shell garbage that I can't be arsed to remember like argument parsing and the like.
All development shops are very resource constrained. Always. GPT is turning into a tool that relieves some of that constraint, but we still need people behind it.
Now, give it a few years, and the more rote stuff could disappear. No one is going to be doing unattended development with it for some time though.
It's not great for operationalizing stuff yet either.
