Seems that I found another application that more or less breaks entirely on Windows Server Codename Longhorn: SharpReader. It does load the feeds and cache correctly, however, one thing does not work at all - refreshing the feeds. I'd say that this pretty much renders a RSS reader useless. Darn. Especially now that I got everything else up and running, including #develop 2.0 Codename Corsavy Build 510.
Update PEBKAC. I didn't install SP1 for .NET Framework v1.1. Now SharpReader works as expected. Note to self: having a WSUS server up and running for your other boxes really can spoil your Beta experience.