If you check one of your users account settings, look for profile section, if you see server share (\\servername\user) then you are using roaming profiles.
If you are using roaming profiles, that IE fav. list will take some time to load since its loading from the server.