At least 38 people were killed in the accident and a search is continuing for five who are still missing, according to Vietnamese state media.
The Wonder Sea embarked early Saturday afternoon for a three-hour excursion in the popular bay, carrying 48 passengers and five crew.
But a storm quickly covered the sky, bringing high winds and pouring rain, survivor Dang Anh Tuan told The Associated Press.
Tuan said the passengers asked for the boat to turn back to shore, but the crew reassured them they were almost at their destination and kept the boat moving forward.
"It rained for about 15 minutes, and then the boat started to shake vigorously, tables and chairs were jostled around and seconds later the boat overturned," the 36-year-old fire extinguisher salesman said.
"Water gushed in and I lost all orientation.
"I tried to breathe. But more water came in. I took a deep breath, got rid of my life vest and dove down.
"I saw a streak of light and followed it to swim out, escaping the boat, and then I climbed on the overturned boat to look for help," he said.
Tuan and three others survived by clinging to the capsized boat and its propellers, waiting another two hours until the rain stopped and rescuers arrived.
Rescue workers saved 11 people, but one died in hospital due to injuries, VNExpress newspaper said.
The boat turned upside down because of strong winds, the newspaper said.
A 14-year-old boy was rescued after four hours trapped in the overturned hull.
The newspaper said most of the passengers were tourists from Hanoi, including about 20 children.
Tuan was a holiday with 11 university friends, only three of whom survived.
Tuan has only minor cuts, but one of his friends suffered multiple head injuries and the other's tendons were cut by broken glass as he escaped the boat through a window.
The other nine members of the group were killed, including one who was travelling with his wife and three-year-old son. The wife and child were also drowned.
A tropical storm is also moving toward the area.
A national weather forecast said Storm Wipha was expected to hit Vietnam's northern region next week, including Ha Long Bay's coast.
7°C