Jesse Tyler Ferguson Jokes “Modern Family ”Cast Couldn't 'Get Out of the Building' After Emmy Loss Following Winning Streak (Exclusive)

The ABC sitcom won the Emmy Award for outstanding comedy series five consecutive times before losing in 2015 to 'Veep'

The Modern Family family cast were feeling lost — literally — after their Emmy-winning streak came to an end in 2015.

In PEOPLE’s exclusive first look at Jesse Tyler Ferguson’s appearance on Busy This Week, the 49-year-old actor discusses the ABC sitcom’s five consecutive Emmy wins for outstanding comedy series, which tied the record for most wins in the category with Frasier.

When asked if the cast ever got sick of winning, Ferguson responds, “No, but the audience got sick of us being there.”

Related: The Cast of Modern Family: Where Are They Now?

“We won five years in a row, which is some sort of a record,” he adds. “I think we tied with Frasier for the most consecutive Emmy wins in a row and I remember the year we won for the fifth time, there was a collective groan in the audience.”

After the win streak, which came to an end in 2015 when Veep was announced as the category's winner, Fergurson jokingly recalls how the cast had difficulty finding their way out of the awards ceremony as they previously exited through the backstage area following their wins.

“I hope this doesn't make us sound too gross, but when we didn't win the sixth year, we didn't know how to get out of the building. We went to the exit and they were like, ‘This is not the exit, it's always been over there.’ We're like, ‘well, we've never had to go out that door,” he explains with a laugh. “A group of like 10 actors [saying] ‘Well, that's the way we went last year.’”

“That’s kinda iconic,” host Busy Philipps responds before joking that the group still wanted to go backstage and do press.

Related: Jesse Tyler Ferguson Explains How Playing a Gay Character on Modern Family Created a 'Coat of Armor' in Real Life

“I thought you’d want to hear our massive disappointment that we did not win a sixth [Emmy], no, we were very lucky to get five,” he jokingly responds. “But it was like, enough is enough.”

Ferguson jokes that over the years, the cast — which also included Sofía Vergara, Ed O’Neill, Julie Bowen, Eric Stonestreet, Sarah Hyland, Nolan Gould, Ariel Winter and Rico Rodriguez — got “closer and closer to the side and the exit” while his costar Ty Burrell, who played Phil Dunphy on the sitcom, remained front and center as a nominee. Burrell was nominated for outstanding supporting actor in a comedy series from 2010 to 2017 and took home the prize in 2011 and 2014.

“Ty Burrell kept being nominated. So he was always in the center, and we just kept moving farther away from him,” he recalls before adding, “He was like Kate Winslet on the door at the end of Titanic. He's like, ‘You can try it all fit on this’ and we’re like, ‘No, we're fine.’”

Related: Jesse Tyler Ferguson Makes Modern Family Costar Aubrey Anderson-Emmons 'Really Nervous' at Her School Plays

Modern Family came to an end in April 2020 after 11 seasons.

“It’s something we’d all been dreading for a long time,”co-creator and executive producer Steve Levitan told PEOPLE at the time. “This has been a dream experience for everybody. I can’t tell you the number of people that have said this is the best job of my career. When you’ve had such a wonderful experience, it’s very sad to see it end. There was a lot of love on that stage and in that room for the table read. People dreaded saying goodbye. The emotion was built right into everything we did towards the end.”

“This is as close a cast I’ve ever seen,” he added. “Everybody was just grateful to be there. Everybody was so appreciative. It really is a family.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.

Busy This Week airs Wednesdays at 10:00 PM EDT on QVC+.