The TV star said Alexander Armstrong, his co-host on the quiz show, played Pointless with the late monarch at the Sandringham Women’s Institute.
He said Armstrong told him that Her Majesty was really competitive - and admitted he was "fuming" that he didn't get to go too.
"I note I wasn’t invited!"
He went on: "He went up to Sandringham WI and he played a game of Pointless. There were two teams, the Queen was captain of one of the teams against a local woman from the village.
"The Queen’s team won.
"He said she was very, very forthright in her opinions as to what answers they should give and at the end he presented her with a Pointless trophy."
Osman told host Lorraine Kelly: "She was apparently very competitive and very funny."
"I was fuming the next day because he kept it all a secret, I was fuming that I hadn't been invited!" he confessed.
Osman had hosted Pointless for 13 years since 2009 but announced earlier this year that he was stepping down.
"Pointless has been a joy from start to finish, working alongside my friend Alexander Armstrong, backed by the most wonderful team, and for the best viewers in the world," he said at the time.
From now on the star will only appear in celebrity specials.
Watch: Richard Osman doesn't think Alexander Armstrong would leave 'Pointless' if he quit show