It's a plugin that caused the problem. Which plugin, it's the one that I made up of few small ones, see this post elsewhere. I suspect it's one of the little plugin called "wordcount" with whitespaces that caused the problem. In theory my way should have worked, I'll prove that later on.
It's fun, isn't it. Creating problem yourself, tracking it down and solve it yourself. Pain is the mother of joy.
I prove my assumption by activating the default classic 1.5 theme in WP Playground and I could successful subscribe to the feed from Windows Live. Now I'll be busy tidying up the mess.
PS: It's now done