Nah, can't be. This would not explain why there's so much variation within the same gender, since hormone levels are generally similar. Like it might contribute somewhat if that's the case (personally I've never heard of estrogen causing growth plates to fuse, but I'm not an expert) I'm not sure that we actually have a ton of info as to why men tend to get larger. And it's possible that it has something to do with hormones, but if that's the case, I would lean more towards it being a difference in the regulation of growth hormones (for example it's quite common for girls to grow way faster than boys, but boys just keep growing for a lot longer).