I know people whose lives have been blighted by racism, but I’m not among them. One of my earliest childhood memories is of hiding, with tens of other Sikh families, in the local temple, as far-right gangs terrorized the Midlands, but our homes remained untouched and I’ve never experienced racial violence. My mother wouldn’t let us out on match days because of racist football fans. But even this has changed and Wolverhampton Wanderers now have one of the most racially diverse fanbases in Britain.

It reflects well on Britain an immigrant factory worker who couldn’t read or write could father someone who became a professional writer. And my experience echoes that of my community, Sikhs in Britain being widely embraced as an immigrant success story. My parents’ generation faced brutal discrimination in work, housing and leisure, but there are now prominent Sikhs in the media, the Commons and the Lords.

It’s surreal, therefore, that, at the age of 44, vicious racism is suddenly a regular feature of my life. What has changed? Well, I’ve written a book, Empireland: How Imperialism Has Shaped Modern Britain, and talking about it has attracted endless abuse. After discussing the topic on BBC Radio 5, presenter Nihal and I were trolled for days, being told so often that as brown people we didn’t belong in Britain, that I stopped using Twitter for a while. I’m now receiving handwritten letters every other day, telling me, among other things, that I’m racially inferior and should head back to India if I loathe British history.

It’s particularly bewildering as even critics who rejected my arguments in Empireland, have conceded it’s a balanced introduction to imperial history. So what the hell is going on? At first I blamed the topic. Let’s face it, the so-called debate on British empire is characterised by more emotion than facts, there being a popular view that it’s possible to look back at 400 years of events and somehow conclude whether empire should make us proud or ashamed. It’s no sensible way of looking back at complex history. You can’t review such a massive thing as if it were an air frier you’ve purchased online. Nevertheless, it’s a popular view and anyone trying to be nuanced gets caught in the crossfire.

But in the  last few days I’ve been struck by another possibility, as Meghan Markle’s remarks to Oprah Winfrey have inspired a national crisis about racism. Maybe the intense response I’ve got is due to the fact that conversations about empire are actually conversations about race. After all, when you talk about British imperialism, you’re talking about white people colonising, and sometimes enslaving, brown people. And I’m probably only getting a taste of what every prominent person of colour gets whenever they mention racism. Whether you agree with them or not, Meghan Markle, David Olusoga, David Harewood, black footballers taking the knee, have all been the subject of routine, vicious racist online trolling after their statements.

In terms of basic logic, it’s absurd. If your response to a British person of colour observing racism as a fact of life is such intense anger, that you tell them to get back to India or Africa, you’re demonstrating their point. But then this idiocy reflects a wider national conversation on race that is habitually stupid. What has actually been achieved since the royal couple made racism in Britain a global debating point? The executive director of print trade body The Society of Editors has resigned after a row about his reaction. Prince William has insisted his family isn’t racist. An online sportbook published odds on who in the royal family might be the royal racist, making “royal racist” sound like a formal title.

In days of fuss, of people throwing the accusation “racist!” around and people we don’t know resigning from organisations we don’t know, I’ve heard no serious conversation about how racism actually affects people’s lives. About how, for example, black people are treated more harshly when it comes to school exclusions, criminal justice, press coverage, or sectioning under the Mental Health Act. If we keep failing to have this serious conversation about race, it’s surely partly because of poor understanding of imperial history.

There’s a popular view that because we beat the evil racist Germans in World War II, and abolished slavery, that Britain is somehow beyond racism. This narrative glossing over the fact we also ferried 3m Africans across the Atlantic, and that British Empire in the 19th Century was proudly white supremacist. We also fail to get anywhere because race is yet another topic, alongside empire, trans rights, Brexit, that has become a culture war. You’re not allowed to explore these issues with innocence or curiosity: you need to take sides and despise the opposition in the way you despise any football team playing your own. Everyone loses as a result.