When you go ahead and download an installer it has a load of important files, such as icons, libraries, sounds, etc. When you run it, it will go ahead and extract all these files from one .exe or .msi installer file and allow the files packaged up inside to be static files the program can access.
You may also find if you're on Windows (not 100% sure about what happens on Mac and Linux) some programs install registry keys which store bits of information, for example this could be the colour code you want the background of the application to be.
Running big programs from a .exe would be slow as when an application wanted to access a file (such as an icon) it would have to extract it and load it. Instead when you install, all of the files are ready to load straight away on your PC.