Yeah the move to iOS4 has not been good:
1) The install process sucked; The 'backup' stage of the upgrade took forever - so instead I ended up using the 'restore' function from iTunes (once I'd got iOS4 downloaded in iTune 9.2). It still took HOURS.
2) The number new features on the 3G is hardly worth it (Folders, Spell checker, Threaded mail,...)
3) It seems less stable and slower than iOS 3.2.x - have to reboot it more
4) A number of my apps crash ALOT like MobileRSS, OffMaps - not handy. (though I guess they probably need to be updated...)
5) Quite a few Apps aren't iOS4 ready - e.g. Skype
Maybe 4.x will help but who knows when that's due......
[5nov10]: 4.1 definitely improves things (and the relevant apps have mostly been updated)