I assume it's that while Hoid has amassed a large deal of Physical Realm power and apparently travels the Cognitive Realm at will, he has yet to weasel his way into the Spiritual Realm directly; this would probably bother someone going around swiping every form of magic he comes across.