That doesn't change the fact that it doesn't take that many people from each Shardworld to establish a kind of worldhopping community. Also, once you've got a few thousand people hanging out in Silverlight, you're probably going to get new additions from children born there (does it count as Worldhopper if you're born in Silverlight?), not just "new" worldhoppers coming from the Shardworlds. A few thousand is enough to start a sustainable population, even without any influx from outsiders. And with people from the different Shardworlds joining them, I see no reason why Silverlight shouldn't be growing in population (assuming they've got enough soucres of food and water and so on, and since they established a city, I guess they do).
As for groups like the Ire, well, why wouldn't they exist? All it takes is one person discovering Worldhopping and instead of going off by themselves, starting a group.
But how they all manage not to be noticed by the people of the Shardworlds, I have no idea. Maybe most of them stay in the Cognitive Realm most of the time, and only a few people actually travel to different worlds (these would be the Worldhoppers we see), and the majority of Silverlight's inhabitants don't go running around on different Shardworlds, just like most inhabitants of Shardworlds never leave their home world.