What version did you use? I know from the author performance has been improved lately and remains a hot issue for future development. It is known that importing is slow, but that is (according to the author) because it imports more than any other DAM software. To be honest, I use IdImager in a multi-user environment and, though not "fast" by any standard, don't find it painfully slow. We have about 200k images in the database...
Anyway, check out http://www.damroundup.com/ for a good overview...