She was featured in Sunday's debut episode where she completed a skydive, but on Monday (7 November) a show spokesperson said she had to leave the programme's set for precautionary medical checks and was told it was not safe for her to return.
As Attwood left, a message on her Instagram account said she was "heartbroken".
Referring to the former health secretary, who is due in camp, the post teased: "Who's going to roast Matt Hancock now?"
Watch: 'I’m A Celeb' Star Olivia Attwood Quits Show After Just 24 Hours
Viewers also said they were disappointed that Attwood wouldn't be there to grill the politician, who has been widely criticised for signing up for the programme.
Several posted messages on Twitter saying they felt as if they'd been "robbed".
"I was very much looking forward to the Olivia Atwood vs Matt Hancock showdown," said one.
"That’s a big miss, she’s brilliant television."
"Honestly gutted we won’t get to see Olivia Attwood and Matt Hancock share the same living space," tweeted another.
One posted: "I feel like we’ve been robbed of showdown, Olivia Attwood annihilating Matt Hancock in less than a day!"
"Can’t believe Olivia has had to leave, been robbed of her having to live with Matt Hancock," said someone else.
The statement - which was apparently shared on Attwood's Instagram account on her behalf - said: "To say Olivia is heartbroken would be an understatement, she dreamed of doing 'I'm a Celeb' for years, and was absolutely loving every second of the show and throwing herself into jungle life feet first (just as we knew she would).
"However, for reasons beyond her control her journey has been cut short. Your support and kind words have meant the absolute world to her, now as much as it always has.
Read more: All the campmates in I'm A Celebrity 2022
"You will hear the truth from Olivia in due course and she will be back on your screens in the New Year."
It added: "P.S Who's going to roast Matt Hancock now?? :((."