The wife was asking for one so rather than buy her one, I waited a few months and am food chaining my ipad 2 to her. Unless you are an Apple Junky (or @WayneSchulz ... :) ) , if you have an Ipad 2, unless the display resolution is killing you, I don't think its worth firesaling your old unit for $300 to spent $600+ on a new one.
I don't notice a speed difference, but I never felt any slowness in the ipad 2. I haven't tried any heavy graphics games yet and only a handful have been optimized.
I will say, when the app is properly retina updated, its damn crispy looking. I don't have any hires photos on it to see the screen in that regard. Anybody know of an app that will show nice hires photos? Most of the stuff on flickr doesn't seem that hires.
Maybe I'm crazy, but it just feels noticeably heavier and thicker. I do need to get one of those peel on wraps on it, especially the back. It's kind of slippery 'naked' and is uncomfortable to hold up for a long time.
With Retina, I think there could be a market for a 7"" unit that's alot lighter.