If someone claims to be "in the know," that person is generally suggesting that he or she has access to information that isn’t available to the rest of the public. This kind of access is often claimed for government information, company information, or in regards to secret tips and tricks in a certain area of skill. People can sometimes be skeptical when they hear this term used by a random person, because it’s generally well-understood that people sometimes claim to be "in the know" as a way of garnering prestige and making people believe they are more knowledgeable than they actually are.
Generally speaking, using the term requires a situation where there is some level of secrecy. This is why government information is a common context, since people normally believe that there are top-secret government projects and other possible areas where some facts may be kept from the public. The idiom is also usually applied to situations where the secret-keeper is a large and monolithic entity, like a government or a large company, but this isn’t always the case.
Some people claim to be "in the know" because they have information on the practice of a prestigious skill that hasn’t been shared. For example, a particular athlete might have a special workout routine that allows him to outperform other people, and if he were to keep this routine mostly secret, the few people he told about it would be "in the know." In a case like this, the list of people who actually knew something might include the athlete’s closest friends, his coaches, and maybe his teammates. If those people started sharing the information, it would eventually become common knowledge, and once it stopped being a secret, using the term wouldn’t generally apply very well.
When someone claims to be "in the know" there can sometimes be a lot of skepticism because it is often very easy to make the claim without having to back it up until much later. This sort of fraudulent utilization of the idiom has generally become even more common since the Internet became widespread because online communities often provide an additional layer of anonymity. On the other end of the spectrum, some reporters and other people in the information business are especially famous because they actually are "in the know," and these people may go to great lengths to protect their sources so that they can maintain that status.