Could you not build a sampler that pitch shifts across they keys rather than effecting the sample timing?
Could it not be the sort of thing that, when you load a sample, made heigh res calculation first to avoid latency problems? Depending on the size of the sample this could take a few but it would be better than live pitch shifting which calculates on the fly.
Could this not be left optional (toggle on for calculations) and chucked into the long awaited granular sampler?
Sorry if this has been mentioned already but the longer we wait for new, more powerful samplers the more my mind churns stuff like this out.