About 110 migrants were crowded onto the small boat on Tuesday when panic broke out, the local prefecture said.
Neither the prefecture nor coast guard could say how many people had been rescued or could be missing.
"The provisional death toll is five people died: three men, a woman and a child," an official at the prefecture said.
"After becoming initially stuck on the shore, the boat set out to sea again.
"A crowd movement apparently occurred in the overloaded boat, causing several victims."
The coast guard was still carrying out search-and-rescue operations at sea, a spokesperson said, after what the official called a "busy" morning, with several crossing attempts.
The attempts took place after Britain's upper house of parliament passed legislation on Monday night that will allow the UK government to send asylum-seekers to Rwanda rather than stay in Britain for processing.
"These tragedies have to stop," British Home Secretary James Cleverly said of the five new migrants deaths at sea.
Stopping the flow of migrants is a priority for British Prime Minister Rishi Sunak's government, which says the Rwanda plan will act as a deterrent.
Human rights groups and other critics say it is inhumane.
Tens of thousands of migrants - many fleeing wars and poverty in Africa, the Middle East and Asia - have reached Britain in recent years by crossing the Channel in small boats on risky journeys organised by people-smuggling gangs.
The Channel is one of the world's busiest shipping lanes and currents are strong, making the crossing on small boats dangerous.
The people smugglers typically overload the boats, leaving them barely afloat and at risk of being lashed by the waves as they try to reach British shores.
11°C