On King Haggard, the Red Bull, and paperclip maximizers

“I like to watch them. They fill me with joy.” The childish voice was all but singing. “I am sure it is joy. The first time I felt it, I thought I was going to die. There were two of them in the early morning shadows. One was drinking from a stream, and the other was resting her head on his back. I thought I was going to die. I said to the Red Bull, ‘I must have that. I must have all of it, all there is, for my need is very great.’ So the Bull caught them, one by one. It was all the same to the Bull. It would have been the same if I had demanded tumblebugs or crocodiles. He can only tell the difference between what I want and what I do not want.”
-The Last Unicorn by Peter S. Beagle

The Last Unicorn has been one of my favorite movies since I was a kid, and the book is one of my favorite books. I was rereading this part of it the other day, where King Haggard tells the unicorn how he came to have all the unicorns in the world except her trapped in the sea beneath his castle.

Now that I’ve read more about AI, it’s hard not to think of the Red Bull as a paperclip maximizer with King Haggard as its utility function. And King Haggard is a human who can never be happy, because he fails to appreciate the complexity of his own utility function. He thinks taking the happiest moment of his life and collecting as many resources as possible to replicate it will make him happy.