From the section on movement:
"A base movement of 120' in ten minutes may seem slow, but it assumes that the players are mapping carefully, searching, and trying to be quiet. It also takes into account the generally "dark and dingy" conditions of the dungeon in which characters are adventuring."
Consider a hallway with no windows and no lights, or perhaps a natural tunnel/cave. What if you could only see 30' in any direction by a flickering torch or oil lantern before darkness obscures everything? Now imagine that you have never been in this place before. You have no idea what lies ahead. Now imagine you are worried about booby traps, possible cave-ins, and monsters. You're testing the floor with every step to make sure it's safe to step on (possibly pressing with a long pole). Even the guys in plate armor are trying not to make too much noise as they clank along, keeping weapons at the ready. All the while someone in the group is trying to sketch a map with parchment and a quill to find your way back out again.
Are you really moving all that fast?
It's also small wonder that such a group –especially carrying loads of equipment, armor, weapons, and eventually (hopefully) treasure– would need to rest every hour or so, just to take a sip of water, a bite of dried meat, catch their breath, and sheathe that sword for a minute after holding it at the ready for so long.
Now, once you've been up and down the entrance corridor a few times to and from base camp, it makes sense that you'd go a bit faster. Moldvay doesn't disappoint on this score:
"The DM may wish to allow characters to move faster when traveling through areas they are familiar with."
Overall, I find these rules elegant and quite "realistic" enough to emulate what I feel they were after; the experience of fortune-seekers exploring and searching through ancient subterranean ruins and lairs.