What scares me is that con men and delusional idiots are the ones making the decisions about AI. Like biological weapons development, this is an area where unintended consequences have the potential to destroy mankind. And it is in the hands of people who have demonstrated that they will fire anyone who wants to slow them down by examining the risks and the underlying ethics of what they are doing.
Altman is the most obviously terrible example of someone who should never be allowed near this technology, but his counterparts at Google, IBM, Apple, and the other tech giants are nearly as bad. They want the fame, money, and power this could bring them. None of them are looking out for the good of humanity as a whole.
I firmly believe that our best hope, at least for the moment, is that general AI is going to take longer than they think. We are not going to achieve it by building more powerful versions of what we have now. It will require something new and different. By the time that breakthrough happens, we need to have responsible people managing it.
I saw an article a year or two back that talked about this very thing. It was actually management people at Amazon saying that they predicted they would be “out of employees” before the end of this decade.