# An unethical optimization principle

If an artificial intelligence aims to maximize risk-adjusted return, then under mild conditions it is disproportionately likely to pick an unethical strategy unless the objective function allows sufficiently for this risk. Even if the proportion *η* of available unethical strategies is small, the probability *p*_{U} of picking an unethical strategy can become large; indeed, unless returns are fat-tailed *p*_{U} tends to unity as the strategy space becomes large. We define an unethical odds ratio, **Υ** (capital upsilon), that allows us to calculate *p*_{U} from *η*, and we derive a simple formula for the limit of **Υ** as the strategy space becomes large. We discuss the estimation of **Υ** and *p*_{U} in finite cases and how to deal with infinite strategy spaces. We show how the principle can be used to help detect unethical strategies and to estimate *η*. Finally we sketch some policy implications of this work.

## Stay in the loop.

Subscribe to our newsletter for a weekly update on the latest podcast, news, events, and jobs postings.