Juice allows you to download and manage your various podcast subscriptions. It has a large built-in directory of podcasts (though some are no longer active). You can play a downloaded podcast directly in Windows Media Player.
By default, Juice will save your podcasts to JuicePortable\Data\Downloaded Podcasts. You can alter this directory in the options. It should be noted that while you are downloading and saving files to a USB flash drive, performance of other portable apps will slow.
Give Juice Portable a try to see how easy it can be to download and manage the podcasts!