I wonder if there's some kind of intent involved with what Taravangian was capable of doing in terms of the memories.
It'd be pretty hard to argue that e.g. removing all of Hoid's memories wouldn't harm him, but maybe just removing the previous conversation and rerunning it was a small enough change (and for some motivation that we don't know about) that Taravangian could convince himself it wouldn't harm Hoid?