Stephen Colbert’s Late Show interview with Prince Harry got off to a lighthearted start Tuesday night. 

But when the self-exiled Duke of Sussex started discussing his raw new tell-all memoir, Spare, things got fairly serious fairly quickly. “Writing this book has been a cathartic experience for me,” and it’s nice to have it out finally for people to read, Harry said. He slammed the British press, his nemesis, for leaking fragmented, out-of-context bits to “turn it into a salacious headline.”

“Do you think that right now there is an active campaign by the rest of your family, by the royal house as it were, to undermine this book and you as you support it?” Colbert asked. “Of course,” but the effort is “mainly by the British press,” Harry said.

“Without doubt, the most dangerous lie that they have told is that I somehow boasted about the number of people that I killed in Afghanistan,” Harry said. That’s “a lie,” and “it’s really troubling and very disturbing that they can get away with it, because they had the context. … My words are not dangerous. But the spin of my words are very dangerous,” because it puts a bigger target on his back. Veterans carry shame about their service, Harry explained, and “my whole goal and my attempt with sharing that detail is to reduce the number of suicides.”

Prince Harry discussed his decision to exit Britain and distance himself from his royal family, which the royals perceived as a threat, he said. “They always knew that my wife was going to leave, because the way that they were abusing her. But I think the most embarrassing thing is that I decided to leave with her.” The idea of him and Prince William growing up close, brought together by the traumatic death of their mother, is a royal fabrication, he added. “The fracture of the relationship between me and my brother has very much been pinned on my wife.”

Finally, about 30 minutes into the interview, and right before their second glass of tequila, Colbert brought up the part of the book about Harry getting “frostnip” on his “todger” during a veterans’ walk at the North Pole, and their long discussion of that topic lightens the mood. You can watch that, and learn whether Harry fact-checks The Crown, below.