A mare's nest is an extremely messy or complicated situation, akin to a rat's nest. This phrase is not widely used, and it appears to be most common in Britain; outside of England, people may find it a bit confusing. However, this is not the original meaning of this phrase, and tracing the history of the mare's nest reveals a fascinating array of twists and turns, demonstrating how much a language can mutate in a short period of time.
The first recorded instance of “mare's nest” occurred in 1576, and it referenced a false discovery. Because mares notably do not nest, the discovery of a “mare's nest” would have been viewed with some skepticism and perhaps criticism. In this sense, the slang term also referred to any sort of hoax or fraud, made with intent to mislead people. People could also use the phrase to refer dismissively to illusory discoveries, as in “Mr. Jones thinks he's made a breakthrough, but really he's found a mare's nest.”
By the 1800s, the term had shifted slightly, coming to be used to refer to confusion or misconception. In this sense, a mare's nest wouldn't necessarily be intentionally deceptive, but rather a confused or unclear circumstance. A mare's nest could be a form of false reality, created through a lack of complete understanding and awareness.
In the mid-1900s, the meaning of a mare's nest realigned, coming to be used to describe a chaotic situation. People could use the term both literally, in the sense of a physically messy environment, and figuratively, to refer to a confusing situation. This modern sense of a mare's nest is certainly a far cry from the original meaning, and for anyone who stops to ponder the literal phrase, it can seem a bit confusing or mysterious. Some etymologists have suggested that the modern meaning of a “mare's nest” may be closely linked with the common slang term “rat's nest,” with people drawing a connection between the two because both include the world “nest.”
Modern authors appear to have contributed greatly to the new meaning of a mare's nest, especially in the 1930s and 1940s. Agatha Christie was one notable author who used the term “mare's nest” to refer to a mess, and the popularity of her books undoubtedly contributed to the evolution of the new meaning of the phrase. Given the fact that her books were often layered with meaning, she may have in fact chosen “mare's nest” deliberately, and her readers may have simplified the meaning into its modern sense.
