PARIS—The French Open lost two of its biggest stars in a matter of hours on Sunday afternoon, as Serena Williams crashed out in the fourth round and Roger Federer withdrew in order to nurse his knee ahead of Wimbledon.

Williams and Federer had arrived at Roland-Garros on parallel missions to add to their record Grand Slam title hauls. Both are 39 years old. Both are struggling for full fitness. And neither has won a major tournament in at least three years. For Williams, who lost 6-3, 7-5 to No. 21 seed Elena Rybakina of Kazakhstan, the drought is the longest of a professional career that spans more than two decades.

“I’m so close,” Williams said after Sunday’s defeat. “There is literally a point here, a point there, that could change the whole course of the match. I’m not winning those points. That literally could just change everything.”

Federer, seeded eighth here, had hinted at a potential withdrawal following a four-set grind to beat Dominik Koepfer of Germany on Saturday. After undergoing two surgeries on his right knee and sitting out three majors during the pandemic, he wasn’t expecting his stay in Paris to last this long. He said that he hadn’t imagined feeling strong enough to win three matches.

“After two knee surgeries and over a year of rehabilitation it’s important that I listen to my body and make sure I don’t push myself too quickly on my road to recovery,” Federer said in his announcement on Sunday.

Roger Federer celebrates after winning his third-round match against Dominik Koepfer on Saturday.

Roger Federer celebrates after winning his third-round match against Dominik Koepfer on Saturday.

Photo: christian hartmann/Reuters

Federer and Williams, who have 43 major titles between them, had looked on course to reach the second week of a Grand Slam tournament together for the first time since the 2019 U.S. Open. But Sunday afternoon dashed those hopes, as it became clear that neither was fully ready to start winning championships again.

Williams, who last won a major at the 2017 Australian Open, had come to Roland-Garros with just three clay-court matches this season—and two of them had ended in defeat. But she said on Sunday that her struggles to be ready for the French Open wouldn’t change her preparation for Wimbledon. And for the next few weeks, her plan is to have no plan: Williams won’t play any grass-court warm-up events before returning to the All England Lawn Tennis Club, where she has won seven times.

“I’m kind of excited to switch surfaces,” she said here. “But historically I have done pretty well on grass. I have done pretty well on clay too. Just not this particular season.”

Write to Joshua Robinson at Joshua.Robinson@wsj.com