Right but that just says you didn't care about evolution's goal, about reproduction. The paperclip Maximiser sets up a very weird scenario.

In your reproduction example, it would be like if you did decide or care to reproduce but then killed all your children soon after.

The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way. We don't know any intelligence that does so it's an odd scenario.

> Right but that just says you didn't care about evolution's goal, about reproduction.

You care about sex because evolution "programmed" you to care about sex. Evolution didn't really want you to have sex, it wanted you to reproduce. You know this. But you don't care. The analogy is sex=making paperclips - the paperclip maximiser knows that the person who programmed it didn't actually want it to make as many paperclips as possible. But it doesn't care, it wants what it wants.

> The problem about the paperclip Maximiser isn't really that it doesn't care about humans, presumably it didn't care about them before this goal. The problem with the paperclip Maximiser is that it cares about the goal but not the intent behind it and is supposedly generally intelligent. Humans don't work this way.

We don't work this way because we have a bunch of different, often contradictory goals, not a single goal that we devote everything towards. But that seems to be more luck than anything else.