The couple shared at the time that they left filming of The Block Tree Change because Joel's mother Trish fractured her neck during a fall in her home in Adelaide.
Now, Scott's saying the influencer and former AFL star left because the show wasn't "on brand" for them, TV Week reports.
Apparently, Scott also claims the pair complained that "the toilet paper was too scratchy".
Joel and Elle have slammed the claims, saying they left the show after the first challenge due to a genuine family emergency.
They claim they tried to stay on the show, but "couldn't get a definitive answer" from production about whether Joel could "come and go" from the construction site in Victoria to his mum's house in South Australia.
At the time, when they left the show, Elle shared a post on her Instagram that read: "Wherever you are in the world. Whatever you are doing. Family always comes first."
Scott told TV Week he still wasn't convinced by their explanation, saying he never heard anything about Joel's mum when the pair left.
"Of course, family comes first on The Block - we've stopped production before when people have had family emergencies," he said. "And of course, I wasn't referring to somebody going to visit their sick mother... It was because they'd given us no reason."
Scott had previously said at Nine's Logies brunch that he thought Elle and Joel quitting after two days was "unAustralian" and "p*ss poor".
"Forty-five thousand people applied to be on The Block and win life-changing money. These guys got on and they just couldn't handle the pace after 48 hours," he said.
"Which to me is a bit unAustralian! Have a go! It's p**s poor."
When the news broke in April that the couple would be leaving the show, a Nine spokesperson told Yahoo Lifestyle they were able to quickly secure a replacement: "Over the weekend, we were surprised to have one of our new contestant teams depart The Block a few days into filming for the upcoming season."
"We wish them all the best for the future and we’re excited to have cast two new Aussies for the opportunity of a lifetime on The Block. The Block Tree Change will air as scheduled later this year."
Elle, who has almost 680,000 followers on Instagram, and her fiance Joel, a former GWS Giants player, were set to be two of the biggest names on this year’s show.
The 2022 cast includes five couples, with the other big name being Neighbours star Sharon Johal.
Never miss a thing. Sign up to Yahoo Lifestyle’s daily newsletter.
Or if you have a story idea, email us at email@example.com.