There seem to be three possibilities.
1. Taravangian could only read Hoid's memories.
This would have been what I thought was the only possibility because how could messing with someone's memories not be considered harming them?
Moreover, it would be quite strange if Hoid's short-term memories were instantly stored in breath. Taravangian suggests that he needed to store memories for the reason that he had lived so long. So short-term memories wouldn't need such storage. Then again, it might be an automatic function of relying on breaths. Still, Hoid doesn't have enough breaths to have reached agelessness with them, only perfect pitch.
And yet, Hoid's short-term memory is clearly altered. It is possible, as some have speculated, that Hoid (partially) did this to himself (through feruchemy, and the missing coin). But wouldn't that raise Taravangian's suspicion, if Taravangian didn't also manipulate those short-term memories?
2. Taravangian could only alter or remove memories, but not read them.
But then how would he know which memories to take? The fact that Hoid's perfect pitch is off suggests that, at the very least, some (but not all) of the breaths are gone/destroyed. I suppose it is possible that only some of the breaths had memories, but that sounds implausible to me.
3.Taravangian could do both: read the memories and alter/destroy them.
In this case, why didn't Taravangian turn Hoid into his own instrument? Or destroy all the memories?
It just doesn't make sense to me that Taravangian could mess with Hoid's memories without doing harm. As other people have noted, the very idea of someone messing with our memories is one of the most terrifying thoughts imaginable. It goes to the very heart of what makes us...well...us. And clearly Hoid was terrified.
Bonus question: Why would Hoid expose this most vital part of himself to the most dangerous being in the universe? We, as readers, already went through this when Sazed found he couldn't trust his own metalminds. It was extremely scary. Is Hoid really dumb enough to expose that most essential aspect of himself to manipulation?