When you buy a CPU it typically comes on the bottom of the stock cpu cooler. This however is not the best stuff, it will most likely come in the form of a grey patch on the bottom of the heat sync.
Like This..
Some after market coolers also come with thermal paste.
You might want to buy some ArctiClean Thermal Material Remover... sometimes that greyish patch can be hard to get cleaned off.. but if you take it off before you use it, it shouldn't be to difficult, because it hasn't melted yet.