I figured it out early on when they were on the smuggler’s ship. He mentioned thinking about telling the smugglers he had nothing interesting to say but thought better of it. That reminded me of Charlie as his whole defense against the would be brides was to not have anything interesting to say. From then on, everything else just confirmed it.
It was almost painful how Tress didnt notice until the end (Of course, then where’d the story be?). Like when she wasn’t feeling well after Crow kept her from leaving the ship and Huck talked to her about random things on the island to make her feel better. If there was ever something iconic to Charlie, it’s exactly that.