Here's my guess:
How fast is your CF card? 1x, 2x, 3x? You may have one of the slower ones...
Since the Epod is scanning the directory structure when you tell it to do something, it may be looking in folders on the compact flash card each time. If the card is slow, this will slow down all accesses.
Sometimes cheap cards can be more hassle than the "bargin price" is worth...