One question that seems to come up among people new to the 40k universe is "what happened to the technology?" Especially when there is actually so much there, what we have left are hand crafted artisan work as the height of technological mastery other than what is covered by a STC. My theory is that genius comes rarely to the human race and the "spark" that makes for the technologically innovative mind is related to the "spark" that makes a psyker. Not that it is new, but I've seen it a lot in fiction. Steampunk makes the connection close (as well as the Gothic connection) and I think that is simply what has happened in the 40K universe. At the time of the Emperor's being placed on the Golden Throne, the only Imperium approved psykers were those related to the Navigators and astropaths(?) and possibly those who used "faith" powers that may or may not draw on the warp. Not being stupid, either prior to or in the aftermath of the Heresy, it was noted that there was a relationship between general "modern" and "technological" learning and warp taint.
"Modern" and "technological" training contains not only the rote elements we see in use in the 40k universe, as represented by things like building codes and standards, but it also includes training in thinking "outside the box" to apply previously known procedures and information to new situations and solve them. Open, inquisitive minds seeking to know how things work could have been the prey of warp predators. Maybe not as true psykers, but as very, very marginal ones that gave just a toehold to the warp, especially at the end of the Dark Age of Technology and beginning of the Age of Strife. Consider the tales of sudden onset of powers, insanity and possession linked to "raw" psykers manifesting. Without any real psychic powers, these poor technologically trained humans would have been no match for the warp.
It might even be that the more intelligent of the warp creatures hunted out these kinds of minds because they are the ones that would be heading down the path most closely related to the C'Tan/Necrons and their mastery of the physical realm that could cut the material universe off from the warp.
At any rate, once the connection was noticed (or just feared) the suppression of large scale learning had to follow, at least in the short term. Perhaps there was the 40k version of the Butlerian Jihad to go along with it, there seem to be legends of such. Thirdly, the sheer mass of knowledge was probably getting so huge that it couldn't be grasped easily. So now we have rote learning to keep people ignorant of learning how to learn. Thinking big theoretical thoughts = thinking be heretical thoughts. Science and technology stagnate. Information becomes trapped in the form it is in and a closed mind becomes free from doubt. The ability to cross-pollinate ideas and technology is lost, so when things finally break, nobody knows how to fix them or reverse engineer them.