Why do you think moral realism is the major problem here? Anti-realism is hardly a novel position, especially among the scientific materialists that I assume most AI researchers are.

I believe that they are still moral realists. I was a moral realist without knowing it for a long time, even when I considered myself very science-minded.

Scientific materialists anthropomorphize. They also reify morality, though the extent to which they do varies significantly. They fall prey to the Mind Projection Fallacy. People like me and you still fall prey to it all the time without even knowing it, even though we’re familiar with the fallacy and have studied it. The human mind is terrible at representing things fuzzily. (Representing them with the rationally justified degree of uncertainty.)

Personally, I would bet on the most important philosophical error being (object-level) underestimation of the complex fragility of value, thinking that greater intelligence, ‘complexity’, or whatever is desirable without qualification, not wanting to seem parochial.

As you point out below, this is intimately connected to moral realism. I believe that the reason why many scientific materialists can’t see the fragility and complexity of morality, even though it’s right in front of them, is due to moral reification, Mind Projection Fallacy, and moral realism.

I also suspect philosophical errors are less important than simply not taking the problem seriously enough. However, without any data I’m not confident in any of this. Has anyone ever tried to survey the field and catalogue reasons for non-concern?

No, they haven’t.

It wouldn’t be any fun in a game if the AI didn’t have the ability to give misleading signs of its intentions, i.e. lie. It would lack the unpredictability of a human opponent who employs all tricks of deception.