Armie Hammer's Life 4 Years After Shocking Downfall: 'Zero' Family Money, a Small L.A. Apartment and ‘Time in the Doghouse'

Armie Hammer is “trying to be at peace” navigating the fallout from allegations of sexual and physical abuse, sources tell PEOPLE

Hahn Lionel/ABACA/Shutterstock  Armie Hammer in 2019

Hahn Lionel/ABACA/Shutterstock

Armie Hammer in 2019

What does Armie Hammer’s life look like years after allegations of sexual and physical abuse derailed his Hollywood career?

Following the launch of The Armie HammerTime Podcast, sources tell PEOPLE that the actor, 38, is living a humble life in Los Angeles and navigating the fallout from a series of claims against him that began in January 2021.

“He still gets frustrated at specific sensational allegations but he’s trying to be at peace about it,” says one source.

After Los Angeles police first identified Hammer as a suspect in an investigation into an alleged 2017 sexual assault, the Los Angeles District Attorney found in May 2023 that there was not sufficient evidence for a case, resulting in no charges for Hammer. Exes Courtney Vucekovich and Julia Morrison have also claimed that the Call Me by Your Name star sent messages detailing violent and cannibalistic fantasies about them. A lawyer for Hammer said in a statement at the time, "All interactions between Mr. Hammer and his former partners were consensual."

Related: Armie Hammer's Exes Speak Out About His Alleged Cannibalism Fantasy: 'All He Wanted to Ever Talk About'

These days, Hammer is “not living in some lap of luxury,” says the source, noting that he “got zero money from family” and lives in a “very small” L.A. apartment.

“He wants to show people he’s actually working for it… He feels like he’s admitted to the wrongs he made and he is determined to start over. He’s doing it in what he thinks is ‘the right way,’” they add. “He definitely wants to have his career back and he’s taking it step by step."

Todd Williamson/E! Entertainment/NBCU Photo Bank via Getty Armie Hammer in 2020

Todd Williamson/E! Entertainment/NBCU Photo Bank via Getty

Armie Hammer in 2020

As a result of the allegations, Hammer was dropped by his talent agency, William Morris Endeavor, in February 2021. Following his last acting credit, 2022's Death on the Nile, he next stars in an upcoming Western titled Frontier Crucible.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

A second source tells PEOPLE that Hammer has a lawyer “handling everything” as he continues to have no Hollywood representation. “He is super stoked about everything happening and really thinks his career is about to have a second wave. He has the podcast and two or three projects coming up.”

Those close to Hammer “are being cautiously optimistic,” adds the insider. “They’re basically telling him, ‘Yes, this is all great, but we don’t want you to get your hopes up and you get let down when people aren’t ready to take you back yet.’”

Related: Armie Hammer Says He 'Kind of Likes' Allegations of Cannibalism Levied Against Him Now: 'It's Wild'

Although “skeptical” about the Armie HammerTime Podcast, his inner circle is “happy that Armie’s happy and very much leading with the ‘we’ll see’ mentality… He thinks he’s paid his dues and spent his time in the doghouse.”

Mike Coppola/Getty Armie Hammer in 2018

Mike Coppola/Getty

Armie Hammer in 2018

Through it all, Hammer continues to co-parent his two kids, daughter Harper, 9, and son Ford, 7, with ex-wife Elizabeth Chambers. The couple announced their separation in 2020 amid the allegations and finalized their divorce in 2023.

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

“He really is a great dad,” says the first source. “He’s very hands-on and he and Elizabeth are in a great co-parenting routine now. They still consider each other ‘family’ and support each other and only want the best for the other.”

Following the series of allegations, Hammer entered rehab for addictions to drugs, alcohol and sex, as he detailed in a July interview on Bill Maher’s Club Random. The actor continues to be “committed to his sobriety” as he co-parents, notes the source. “There’s a lot of grace, and it’s because their kids are the priority.”