So wouldn't that be a perception thing rather than a fast removal of stormlight?
Kaladin drew upon really low numbers of spheres, sometimes even one, and yet when he did things like survive the Highstorm, or pull every arrow in a volley into his sheild at once, he never comments on his gems breaking, only going dun.