When people use the term “hornet’s nest” as an idiom, they are generally referring to the consequences of creating controversy or changing the status quo. In typical circumstances, someone might warn someone away from dealing with a controversial subject by saying that he might possibly stir up a hornet's nest. It can also be technically used to describe any situation with significant opposition and it is especially applicable when the opposition to something is likely to come from several simultaneous directions.
The term “hornet” is actually a generic term for wasp, although it can also describe certain specific types of wasps. These are generally large stinging insects which often live in huge nests. They are known to sometimes have prickly dispositions, and if someone goes near their nests, there is a great possibility of causing the insects within to become agitated, potentially leading to a whole swarm of bees chasing and stinging someone. The metaphor of the idiom relies on the danger of the nest if it is bothered and its relative harmlessness when left alone. Usually, a veritable hornet’s nest in the idiom will be something that might be better left alone because of potential consequences.
Oftentimes, the reason something would be described as a hornet’s nest is because of controversy. Political reporters might often use the term to describe subjects that politicians need to avoid if they want to win a political contest. These are often subjects where there is really no obvious way to speak about them without making someone angry. In many cases, a veritable hornet’s nest might be some kind of old tradition or custom that needs to change, but can’t be dealt with because of resistance, so it might be avoided until time passes and opinions change.
One aspect of a hornet’s nest in terms of the idiom is that opposition is generally multi-faceted. If someone stirs up a hornet’s nest in real life, he will usually face a whole swarm of insects and the idiom works the same way. Stirring up a figurative hornet’s nest will generally involve causing a lot of opposition from a wide number of people, potentially leading to a losing situation.
Most of the examples mentioned previously have dealt with politicians and reporters and people who deal with the public, but the hornet nest metaphor can work on a smaller scale too. For example, saying something that causes a big family controversy at home could also be described with the hornet nest idiom. The size and scale aren’t nearly as important for the idiom as the circumstance of being swarmed with opposition.