Holy days and feast days were amazingly important in pre-modern and pre-industrial society. Our own holidays pale in comparison. I've missed many in the last few years, due to work or travel. For us, holidays aren't really that special. They're important when we are children, and then they become less and less so as we age. I would make the argument that the same is not true of medieval people, and the same need not be true of the folk of your fantasy game-world either.
From my own research and readings, calendrical rights played an all-important role in the medieval mind. They sometimes marked days of major labor (for instance, those rights associated with the harvest feasts wherein even freemen had to work on their lord's lands) and sometimes instead marked days of major remittance. Either way, they were almost always accompanied with drinking and eating beyond what was considered the acceptable amount. These were the feast-days, the days when social barriers were relaxed, food was abundant, and a festival atmosphere reigned.
A third type of holiday was the common and secular market day, which generally happened as often as once a week and as rarely as once a month in large towns and villages in England and France. Market days, while not of the same power as a regular holiday, still caused many hundreds of outsiders to pour into a center of distribution and engage in a festival-like expression of commerce. These gatherings were not as liminal as the true holy days, but they still provided an outlet. Indeed, a yearly market (such as the Champange Fairs) might last for several days and have the feeling of a true holy day! In the yearly market towns the arrival of foreigners from every corner of the globe was certainly a chance to trade with and speak to merchants from regions that one would otherwise know nothing about.
While festivals, feasts, and holidays serve the important function of marking the passage of the year (hence, calendrical), they had other purposes as well. Many holy days allow for the release of social pressures, none so clearly as the festival called the Feast of Fools (known by various other names also). Other festivals draw clear links between the nobility and the common man* by the distribution of food and ales from the local lords.
No one would miss these festivals and think little of it; they were important aspects of life that, in a sense, kept the world turning. These holidays were not for children alone, and they rarely included the kind of supernatural commercialism (I honestly can't think of any that did off the top of my head) that we see in modern incarnations of holidays such as Christmas and Easter.
As holidays helped mark the turning of the year, they were of momentous importance. Farmers, merchants, and even lords kept them almost without fail. There could be a certain danger in repressing or failing to celebrate a holiday: since they act as outlets of social tension, stopping the holiday would be clogging up one of those outlets. They relieve pressures between lord and common man, between neighbors, and between family members. When that system of pressure-relief is taken away, otherwise unimportant incidents may blossom into violence, rebellion, and treachery.
So don't forget your festival days, don't forget your holidays. No fantasy setting (other than maybe Eberron) has industrialized to the point where the year has been mechanically chopped up by a clock and calendar. Everyone needs their holidays both to tell what time of year it is and to feel better about themselves and their position in society.
*it's important to note that nobilis was a quality that you possessed in most of the middle ages, not a class that you belonged to. The nobility was a semi-permeable class.