Olivia Rodrigo Shocks Trick-or-Treaters for Halloween on 'Jimmy Kimmel Live': 'Am I That Scary?'

The pop star surprised kids on Thursday, Oct. 31 and sang some duets with them

Olivia Rodrigo made some trick-or-treaters "happier" this Halloween.

On the Thursday, Oct. 31 episode of Jimmy Kimmel Live, the pop star, 21, surprised a handful of trick-or-treaters who happened to show up at her door and were just expecting candy.

At the start of the bit, Kimmel Live star Guillermo Rodriguez appeared in costume as Rodrigo but with a mustache, sporting glitter hot pants, an “eat, drink, sleep” crop top and a brown wig.

Related: Olivia Rodrigo Opens Up About 'Terrifying' Moment She Fell Through a Hole While Onstage in Australia

“I’m Olivia Rodrigo!” the comedian, 53, said as he removed a paper mask of the musician. However, the real Rodrigo, wearing an identical outfit to the comedian, but with a red heart necklace and a white tank with red bra straps emblazoned with the message “Live, Laugh, Love.”

“And I’m Olivia Rodrigo!” she exclaimed while holding a bowl of candy. “Let’s mess with some kids!”

The first set of trick-or-treaters, a young girl and boy, guessed who Rodriguez was supposed to be for Halloween. After he asked the girl if she liked Rodrigo and she said the "Vampire" singer was one of her favorite artists, Rodrigo appeared and the girl became nervous, before revealing she attended one of the "good 4 u" artist's concerts.

“Do I look just like that?” she asked pointing to Rodriguez, to which the girl replied: "Yes."

Another group of kids appeared, and one girls was flabbergasted after she was asked if she recognized Rodriguez's costume.

A young boy in the group asked her, “Why are you not talking?”

“It’s like if you met whoever you like the most,” she replied.

“Wait, you’re the real Olivia Rodrigo?” he asked.

“You don’t believe that it’s me?” Rodrigo asked, teasing that she could show her driver’s license as proof.

“It doesn’t say Rodrigo,” the boy replied. “Oh wait, it does.”

One girl started crying once she saw Rodrigo and told her friend: “I’ve never happy cried in my life.”

After a boy dressed as a grumpy old man with a mustache looked shocked, Rodrigo asked: "Sir, are you having a stroke?"

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Related: Olivia Rodrigo Reveals Hilarious British Phrase She Repeated Before Every Guts Tour Show: 'Very Superstitious' (Exclusive)

One group of girls alternated screaming with Rodrigo and another fan played a game with the "deja vu" hitmaker where she earned a piece of candy for each correct song title of Rodrigo's that she named.

At the end of the bit, Rodrigo sang “Vampire” with several of the trick-or-treaters.

Rodrigo released her Netflix concert special Olivia Rodrigo: GUTS World Tour on Tuesday, Oct. 29.