In Warbreaker the first Return was around 400 years before the story, and the first one was where the Edgli flowers grow.
At the time of the story there are Returned all over the world, but so is the color from the flowers...
So the flowers colors are needed to make a Returned or the Shard arrived 400 years before...