Medieval themed fantasy would have you believe that every land has a lord, and that every lord has but one land. Each town or vill has its baron, its great man with heraldry and shields and knights and all that jazz. Of course, this couldn't be farther from the truth and no lord would ever have been able to raise enough men to do battle with if every holding in the land was subdivided into a thousand thousand petty lordlings. Certainly, there were subdivisions, but it was a rare enough case when each lord owed fealty to both a higher and lower lord. The real questions are: If a lord owns several towns, vills, and manors, who administers all of them? Where is he, if that manor house has no lord living in it? Who DOES live in it?
The answer is fairly simple. Every little town might have a manor house, and it is true that there were lords that owned multiple manors. The man living in that manor, then, was less likely the lord (by the law of numbers, since there were more manors than lords) and more likely his local bailiff. Bailiffs in England were drawn from the populace of the town they administered and were often wealthy or highly placed farmers, knights, or lesser sons of other houses. The bailiff generally served as the ultimate legal authority save the lord himself and was sometimes attended by a serjeant of his own. It was highly unlikely that a baron would have barons of his own to attend him, something important to remember once your baron rules over several towns. His arm of the law there would be the bailiff.
The bailiff was also in charge of the collection of rents and overseeing of pretty much all the towns functions. This included, of course, supporting the asshole miller (everyone always hated the miller, who was part of the lord's monopoly on gristing and milling), making sure roads and bridges were kept up and collecting market tolls (if there were weekly or monthly markets in the town or vill).
The steward was his other administrative ally (apart from the reeve, not addressed here) who was generally a literate knight (or one with a secretarius clerk) that would perambulate to check the lord's holdings and ensure that all was being dealt with in a manner that would bring him increase and wealth. If tubs of butter or barrels of malt were lacking in a villages taxes, the steward would go and investigate not the lord. Bailiff (and town reeve) would be responsible for reporting to the steward and helping him solve the matter until he returned to the seat of the barony.
So, when you're designing small manorial vills and outlying villages, there's no need to give them their own heraldry (though the bailiff might have his own arms) and their own baron. Indeed, a baron would generally be very insulated from the regular populous on most occasions save feast days. Whether or not the nobles delight in meeting the adventurers they hire is another question, but I'd bet that most would at least make preliminary contact by their local bailiff or steward. The same goes on a grander scale with important nobles maneuvering against one another: they are unlikely to make contact directly or even indirectly through their spy masters, likely preferring some intermediary agent that could never be tied back to them.