Harry, the younger son of King Charles, is due in London and Birmingham next week for a series of charity engagements and events promoting the 2027 Invictus Games.
Meghan, seven-year-old Archie and five-year-old Lilibet were expected to travel with the prince but his spokesperson said they had now dropped plans to travel to London.
The Telegraph newspaper reported that the decision was taken after a request for police protection was denied.
The spokesperson added, however, that the family had not ruled out travelling elsewhere in the UK, including to Birmingham where Harry is due to promote the 2027 Invictus Games.
Harry has repeatedly expressed a desire to bring his children to the UK, which they have not visited for several years, but has said concerns about security have complicated those plans.
"The Duke continues to explore every available option to enable the visit to proceed safely and to give his children the opportunity to enjoy the UK," his spokesperson said earlier this week.
The prince has long argued that security concerns put his family at risk when visiting the UK.
Last year he lost a legal challenge against a government decision to remove the automatic police protection afforded to him after he stepped down from royal duties in 2020 and moved to the United States with Meghan.
The planned visit had raised the prospect of a rare meeting between Archie and Lilibet and their grandfather, King Charles, who has seen little of the children in recent years as relations between Harry and the royal family remain strained.
Archie and Lilibet last saw their grandfather in person during Queen Elizabeth II's platinum jubilee celebrations in 2022.
Harry said in a BBC interview last year that he wanted a reconciliation with his family after years of public disputes, including criticism of royal life in television interviews and in his memoir Spare.
His trip next week also coincides with a closely watched court ruling in his privacy claim against Associated Newspapers, publisher of the Daily Mail.
Harry is expected to attend engagements linked to the Invictus Games, the sporting competition for wounded and injured military veterans that he founded and which Birmingham will host in 2027.
with PA
