Magic on Sel is related to its geography in some way -- that, at least, is the general assumption. In TES, Shai fancies that the setting mark that finishes all soulstamps looks like her homeland MaiPon, similar to how all Aons are built around Arelon.
Shadesmar reflects the geography of the world that you enter it from in some way. (The one map we have of Shadesmar is from the perspective of one entering it from Roshar, and the map is Roshar inverted: land becomes sea, sea becomes land (and in some cases, mountains).)
This probably has a great deal to do with why it is difficult to travel to Sel from Shadesmar.