My mental model for ChatGPT is that it’s an entry-level engineer that will never be promoted to a terminal level and will eventually be let go.
However, this engineer can type infinitely fast, which means it might be useful if used very carefully.
Anyway, letting such a person near financially important code would lead to similar issues, and in both cases, I’d question the judgment of the person that decided to deploy the code at all, let alone without much testing.
This is kind of how it works with Real Engineering™ and other licensed professions - it's the non-licensed people doing most of the grunt work, the PE/architect/licensed professional reviews and signs off on it. But then, by virtue of their signature, they're still on the hook for any problems.
However, this engineer can type infinitely fast, which means it might be useful if used very carefully.
Anyway, letting such a person near financially important code would lead to similar issues, and in both cases, I’d question the judgment of the person that decided to deploy the code at all, let alone without much testing.