I'm having exactly the same problem, everything seems ok except for a few 3D games. In-game the system will freeze and I have to reset, this seems to happen mainly with Half-Life 2, Halo and a couple of others, Call of Duty and XIII have been fine.
I ran memtest86 and it's fine with the ram in individually, but comes up with errors when both DIMMs are in - in any configuration.
Here's my spec:
AMD Athlon XP 3200+ Barton
2x 512mb 3200 DDR
Asus A7V600 SE
Nvidia GeForce Ti 4200
Anyone got any ideas?