Criteria for developing principles of ethics governing AI research
AI is both an existential threat and the future of human civilization.
Whether our future is apocalyptic or utopian may depend on the parameters of the very first self-improving AI, as that system will have the potential for superintelligence, which is likely to rapidly evolve beyond traditional governance mechanisms (i.e., laws) and perhaps beyond all human control. (By "AI", I include possibilities such as the augmentation of biological humans, uploaded human consciousness, and artificial general intelligence.)
It is essential to develop a set of ethical principles for AI research that have a good likelihood of being considered by the teams likely to first develop such technologies, and it is necessary to reach a consensus on these principles before such AI is actually developed. This is a much more ambitious goal than current "AI ethics" guidelines, which are limited to current technologies, politically motivated, and do not attempt to guard against existential threats.
More important than the principles themselves are the goals that the principles should further. It is important to be realistic and market-oriented about what is practical. A principle such as "do not build AIs that control killer robots" is counterproductive since AI in weapons is highly effective and likely, and a ban will either fail or direct AI weapon progress to rogue nations and groups, thus contributing to existential risks of AI.
With the above in mind, I recommend the following *criteria* for developing principles of ethics governing AI research:
1: Should not go against strong market incentives, as pushing AI innovation underground or to rogue jurisdictions is worse than no guidelines at all.
2: Should identify which party has the moral responsibility for the actions or decisions guided by AI, so that AI researchers can be held accountable when and only when appropriate.
3: Should respect the moral rights of human and non-human sentient life, so that the rights of humans, uploaded, and synthetic minds will be respected.
4: Should be generic enough to encompass new technological paradigms so that when a new AI paradigm is developed, the existing principles will still apply.
5: Should facilitate the identification of when an AI system deserves moral recognition. For example, non-sentient AI systems shouldn't identify themselves as sentient, as Bing Chat currently does.
What do you think of these criteria? Given consensus on why AI ethics are needed, the ethical principles for AI research themselves will become clear.