Exactly. Remember how he was actually surprised when he found out he could beat Kelsier up?
How has all of Hoid's life, he had never interacted with a Cognitive Shadow he could beat up, if it was just because they were a Cognitive Shadow?
Plus, I like to think Hoid used to carry the same Dawnshard, "Survive", as we know his time holding a Dawnshard has something to do with:
- his longer than normal life (coughs because it was literally the Dawnshard of Survival, maybe? coughs)
- his inability to harm others (coughs because harming is the opposite of Survival, maybe? coughs)
- his inability to eat meat (coughs does anyone have a cough drop? coughs)