Genre conventions are all the little elements, from archetypal characters to repeating plot points, that separate one genre of fiction from another. Most genres have a set of very particular elements that readers or moviegoers expect, and artists often try to make sure their works include enough of these to keep the audience happy. Genre conventions tend to evolve over time as artists work to break out of the typical mold and create fresh new content. Sometimes people eventually get tired of certain genre conventions, at which point these conventions might even be labeled as cliché.
Essentially, genre conventions are the defining aspects of any genre or sub-genre. There are tons of these conventions for each genre, and any given work doesn't necessarily have to include more than a handful of them; if there aren't any at all, the work may not ultimately fall into the intended genre. For example, a horror movie without a few recognizable elements — things like monsters, creepy settings, isolation, or darkness — wouldn't necessarily be considered a "horror" film anymore by most fans and critics.
In many cases, genre conventions begin through repetition. A writer, moviemaker, or playwright will create a famous work that the public loves, and other artists will try create something similar. Over time, all the artists borrowing from the original writer will use certain elements again and again, to the point where they spread across the whole genre and become conventions. For example, Bram Stoker's novel "Dracula" created a host of genre conventions about how vampire fiction should work, and many of these have persisted to this day in works like "Interview with a Vampire" by Anne Rice and "Salem's Lot" by Stephen King. These conventions have also continued into many famous vampire films over the years, and at this point, almost any vampire story will include at least a few ideas that go all the way back to Stoker's original novel.
Genre conventions also tend to evolve over time. As new artists provide their contributions to the genre, they will often experiment with new things, and their more popular works will lead to the creation of new conventions. Sometimes a work will be created that drastically changes a genre in one fell swoop by bringing in many new elements. For example, fantasy author Robert E. Howard's Conan character wasn't the first fantasy hero ever created, and fantasy fiction was already popular when the Conan stories were written, but the popularity and success of the character had a huge impact on the genre. Tolkien's "Lord of the Rings" trilogy had a similar impact on the fantasy genre later on, and more recently, works by writers like George R. R. Martin might have the potential to change the genre by bringing a new style and focus.