Social-media battles. A cease and desist. Lawsuits and threats. The real-estate industry is tearing itself apart over a question that home buyers haven’t had to ask in decades: When you search for a home online, are you actually seeing every property for sale?

For the better part of two decades, the answer to that question has largely been yes, with the vast majority of listings available on sites like Zillow and Trulia. But Compass—now the nation’s largest brokerage—is changing the game. Compass offers its sellers the option to market their homes first as “private exclusives” through the company’s internal website before putting them on listings sites or the local multiple listings service. In fact, it’s been actively encouraging its agents to take their listings off those portals. In New York, for instance, Compass recently offered a $1,000 credit to agents who agreed to pull their listings off the website StreetEasy.

While Compass once controlled only a small percentage of listings, that changed in January with the company’s controversial acquisition of the behemoth Anywhere Real Estate, parent company of Coldwell Banker, Corcoran, Century 21 and Sotheby’s International Realty. Now a much higher percentage of properties for sale could be hidden from potential home buyers.

Compass says its model offers sellers more choice on how to market their homes, but critics say it is harming the consumer. Now the long-simmering industry dispute is spilling over into public view, with lawmakers and civil-rights advocates taking notice.

“This is all coming to a head,” said Errol Samuelson, chief industry development officer at Zillow Group. “This issue is going to have a real impact on people’s lives and how they buy and sell, where they live.”Compass has grown quickly since its founding in 2012, but its $1.6 billion acquisition of Anywhere took it to the next level, more than tripling its U.S. agents from about 40,000 to more than 200,000.

Justice Department antitrust enforcers wanted to investigate the merger before it took place, but were overruled by senior officials; Compass worked with Trump-aligned lawyer Mike Davis to make its case to the Justice Department.

Now Compass International Holdings, the umbrella company, handles an enormous share of listings across the country. In Manhattan, for instance, Compass and its affiliates now handle more than 80% of residential sales by dollar volume, according to some estimates, as well as more than 30% of listings in other major markets. Compass’s spokesperson said those figures are incorrect, but said the company doesn’t keep track of the numbers itself.

Since the merger, Compass’s stance on private listings has made it a lightning rod for litigation and scrutiny. The company is now involved in litigation with Zillow as well as the California Regional MLS, one of the country’s largest listing services. In July, Rep. Scott Fitzgerald, the Wisconsin Republican who chairs the House Judiciary Committee’s antitrust subcommittee, asked Compass CEO Robert Reffkin for a briefing on its private-listings network, warning that private networks can create “velvet ropes” around homes.

In an August essay in TIME magazine, NAACP President Derrick Johnson said private-listing networks are akin to “digital redlining.”A Compass spokesperson said the company follows all laws, including fair-housing rules, regardless of how or when a home is marketed, and declined to comment on Fitzgerald or pending litigation.

Off-market sales aren’t new. Agents have long kept a small number of homes as private “pocket listings,” which aren’t advertised publicly but instead shopped mostly by word-of-mouth among a select group of agents and their clients. MLS rules allow certain exceptions for sellers who don’t want the public to know their homes are on the market, such as celebrities or divorcing couples.

Many homeowners, however, want their listings to reach as many potential buyers as possible in hopes of selling quickly, for the highest-possible price.

“There’s 40 years of white-paper research on how the more eyeballs you put on the property, the higher the probability of a higher price,” said appraiser and market analyst Jonathan Miller, who has been a critic of Compass.

Compass’s acquisition of Anywhere, however, has dramatically increased the number of private listings.

“[Pocket listings] were the exception,” said Bess Freedman, chief executive of Compass rival Brown Harris Stevens. “Compass is making them the rule.”In addition to “private exclusives,” Compass also offers sellers the option of marketing their properties as a “coming soon,” which means they are on the Compass website but not on most other listings sites.

Critics say Compass is keeping listings in-house to increase the likelihood that one of its own agents will bring the buyer, thereby earning double the commissions for Compass.

“Sellers could get hurt because the brokerage has this self-interest,” Miller said.

Compass, however, said double-ended deals aren’t its goal and that it doesn’t track them. Instead, it said private listings provide sellers with more options for how to market their homes.

“Homeowners have the right to decide how their own home is marketed so long as they comply with the law,” Compass International Holdings said in a statement.

Citing internal data, Compass said when homes it represented were initially listed privately, they sold for 4.6% more on average than comparable homes, as well as going under contract 34% faster once active on the MLS. These homes were also 29% less likely to cut their price, the company said.

Eric Johnson, a Compass agent in Boston, said he uses private listings as a sort of test run for homes for sale, to gather feedback on the price and presentation before listing them publicly. He says 20% to 30% sell before they ever reach the MLS, and some buyers pay more than the asking price to snag a property before it goes on the open market, in hopes of avoiding a bidding war. “They’ll say, ‘Here’s my crazy price. I want to lock this up today.’”He pointed to a waterfront home in Quincy, Mass., that sold while being marketed privately for $1.63 million, above the seller’s target range of $1.55 million to $1.6 million.

But Miller said sellers who make a deal before going mass market will never know if they could have gotten a higher price. He said Compass’s strategy recalls the 1970s and 1980s, when agents hoarded listings information on cards in their offices and chose who to show them to. “Except that in this case, there’s only one real-estate agent that’s the gatekeeper, and that’s Compass,” he said. Meanwhile, the consumer may not be aware of how the stakes have changed.

“The consumer, when they sit down at the keyboard, they assume they’re seeing the whole market, but they’re not.”Freedman recalled an instance where one of her agents was showing an apartment on Park Avenue, when a Compass agent mentioned she had an apartment with the same floor plan available on another floor.

“[My agent] said, ‘Well, why don’t I know about it? I have the buyer for that,’” Freedman said. “And [the Compass agent] said, ‘Oh, it’s a private listing.’”Freedman said she emailed Reffkin about the incident, asking him: “Why are we playing detective?”A Compass spokesperson said all of its listings are viewable either on its website or by contacting Compass, whether or not a buyer is working with a Compass agent.

Houston real-estate agent Nancy Almodovar of Nan & Company Properties said keeping a listing off the public MLS for weeks or months shortchanges buyers by limiting the available information about it.“They want to know, ‘how long has this home been on the market?’”Almodovar said. “They deserve to ask those questions and get answers. What are you trying to hide?”Freedman, Miller and others have been outspoken in their opposition to Compass’s practices, and Reffkin is pushing back hard.

Last month, he told a room of several hundred MLS executives at a Florida conference that he was prepared to sue any MLS that fines or punishes its agents for marketing listings outside of the MLS.

Freedman said Compass sent her a cease-and-desist letter after she wrote a string of op-eds criticizing its practices. She also claims Compass retaliated against her by filing an ethics complaint against her with the Real Estate Board of New York.

“We wrote back, ‘Good luck, we’re never shutting up,’” Freedman said.

A Compass spokesperson said the company believes Freedman has made false and defamatory statements about Compass, but denies retaliating against her or others.

For his part, Miller said the dispute has turned personal, with Reffkin blocking him on LinkedIn after Miller repeatedly commented on the CEO’s posts about private listings. Compass declined to comment on Miller.

Zillow, meanwhile, has fought back by blocking listings that were marketed outside of a traditional MLS for more than 24 hours. Zillow isn’t an impartial observer; it makes money from the traffic that listings bring to its platform and by charging agents for leads.

“I think two things can be true at once,” said Samuelson of Zillow. “I think it’s good for sellers and buyers to have transparency in the marketplace, and I think it also happens to be good for Zillow.”Write to Katherine Clarke at Katherine.

Clarke@wsj.com