You could think of it as the buffer for your CPU, between the RAM and the CPU. Technically it's now in front of the North Bridge now, but nontheless, it's a <I>very</I> high-speed buffer for the CPU. You can now see how having a bigger cache on the CPU is better. Other than speed, this is usually a driving cost behind any CPU. It is extremely fast memory. The more threads you have (pipelines) they more you may need. Normal chips come with L1, L2, and L3; each one is bigger, and due to this size, is slower, also. Data flows from RAM > North Bridge > L3 > L2 > L1 > CPU.
For example, on the new Intel Core i7: L1 (32KB <I>per core</I> Instruction, 32KB <I>per core</I> Data), L2 (256KB <I>per core</I> Instruction and Data), L3 (Up to 8MB used for all applications).
Hope that helps.