With plain Windows® DLLs all the system had to go on was the 'name' of the DLL in order to load it. These are listed in the import section of an executable. That is the main cause of the problem - this is just not much to go on if the system has to guess what it needs - no version information is included in the import section of the PE information at all. But when the program was built it does know a lot more about the DLL it will need to run.
This is where Manifests come in, they describe the executable itself and the environment needed to run on. Each program component (EXE or DLL) should have a manifest to describe it. Each assembly typically has an 'assembly identity' that describes the name, author and version of the program and more importantly 'dependentassembly' to define which system components the program needs - and in specific details.
InspectExe display of Manifest
Side by Side assemblies
To manage all these different versions of a component Microsoft now put potentially sharable DLLs in a special set of folders (Winsxs 'Windows side by side'). As there are different versions of the same DLLs with the same name they live in separate folders and the folder name is encoded with the manifest's version number for the component. This allows a component to be shared and multiple versions of the same component to be active at the same time.
InspectExe is an easy to use tool that lets you view the manifests contained in executable files. This lets you work out why programs will not load correctly.