Everyone loves a good story. Stories, after all, are probably the reason why you're roleplaying in the first place. Tolkien, Dunsany, Vance, Cook, Martin, Howard, here are your reasons to pick up a copy of Dungeons and Dragons and play it. They all weave such good stories of the fantastic that one can hardly resist them. Who among us does not wish to emulate the fantasy greats and tell stories of our own? I certainly do. Every time I read some good fantasy I curse that I did not write it, that my own fantasy is not near as engaging or powerful. But here's the thing: I don't take that desire with me into my Dungeons and Dragons games. They just don't seem to be compatible.
"Why not?" you would be very well in asking. What prevents me from writing out the plots that I would otherwise insert into my fiction as adventures? What prevents me from involving my players in the self-same stories that I want to write myself? There are a myriad of reasons, but they can all be summed up with a single idea: the railroad.
That is not truly enough to build a story. But it can be reduced to something more simple even than this. The proper attribute of pen and paper roleplaying games like Dungeons and Dragons is freedom. Freedom to act whatever way you want, to take any action you want within the game-world. This is the thing that no other media can do, this limitless and absolute freedom. Books most certainly cannot do it: their characters are confined to a single pre-determined set of actions that never change. No matter how many times you read the Lord of the Rings, Boromir never kills Frodo and dooms Middle Earth.
And that's the way it should be! That is the form most proper to books. It is why they are books and not roleplaying games. But to impinge upon player freedom in a game like Dungeons and Dragons, to force them down a linear path (or to give them the illusion of choice, as in the problem of the Quantum Ogre), is an inherent violence to the medium. People say this all the time to the point where it seems to have become a meaningless cliché that is easily brushed off by the opposition but I will venture to say it again and perhaps explore what I mean by it: At that point, you may as well be playing a video game. By which I mean: video games can prepare for multiple contingencies, but in the end they are more like a story than a sandbox. The stories you make are rarely emergent simply because video games cannot plan for every contingency. They cannot let things unfold as they may. If you're playing a game with a limited number of options for your players, then you are playing a video game on paper. You have taken away from them the very thing, the very attribute, that sets roleplaying with pen and paper apart.
This is true for all types of pen and paper roleplaying games that I can imagine. Certainly, some have more restrictions than others; you cannot, for example, decide to ignore the horrific mystery in Call of Cthulhu. To do so would be to deny the game itself, to negate the reason for playing. It is understood when you begin that game that you will try to solve the mystery and not drive off to find a speakeasy in Manhattan. However, within those limits your actions are still infinite. They are a form of bounded infinity; you must adhere to the rules of the story (solve the mystery, investigate the murder, whatever) but you will be rewarded with the option to solve the mystery any way you like or to fail completely and be annihilated or driven insane.
Stories that come from Dungeons and Dragons or Call of Cthulhu or any roleplaying game should have a strong element of improvisation within them. As a game master you cannot reasonably prepare for every action the PCs might undertake; indeed, it is an accepted maxim of gaming that you will never be prepared for whatever hairbrained scheme the PCs think up. "No battle plan survives contact with the enemy," writes Helmuth von Moltke, and the same is true of adventure outlines. Your players are not you. They will hardly ever approach a situation in the way you think or hope they will, simply because they will come to different conclusions about the most effective way to do so.
Remaining flexible, allowing your coveted "story" to shift and change with the actions of the players, is vital. On the railroad model, no answer but your answer is correct. If the players answer a different way, you must either tell them NO without cause or you must warp their path to get them back on your rails so the story doesn't careen off track. Rare is the day that players want to be train cars. No one is particularly satisfied by living out someone else's idea of good fantasy. If they had wanted to do that, they could read a book (which is, as we discussed above, meant for that kind of thing).
When you abandon the novel-model of story and approach roleplaying with an emergent model, it is more exciting for everyone. After all, you're all finding out what happens together! As a game master you are adjudicating the interaction of the rules and determining the reactions of the NPCs but you are also responding to player input as the plan for your NPCs shifts. You need to think on your feet. And not every player solution has to be right; you are the arbiter, after all. If you determine that it is foolhardy or stupid after strong consideration (done in a manner that has no bias for the "story" you built) then you can penalize them. Bursting down the front door of a wealthy nobleman's house, for example, will be penalized with shouts for household knights and 2d4 of the bastards arriving every 1d6 combat rounds or something like that. They should have tried to sneak in, you will think to yourself. But that is alright; they didn't. If they survive, so much the better and so much more the glory heaped up on them. If they don't, they'll try to be more clever next time.
Surrender your conductor's hat, my friends, and get off the express.