I agree with you. But I don't trust myself with the power honestly, I am pretty sure that if I could do things like this, I would end up misusing it for reasons which would in the moment seem perfectly reasonable.
So, it's sort of a meta argument, if I would have this power, and even if I had resolved too use it only to do good, it is *still* more likely to do harm.