Emergency service crews rushed to the Royal Daylesford Hotel, northwest of Melbourne, on Sunday evening after an SUV crashed into the beer garden.
The white BMW mounted the kerb and struck patrons, killing two men, a woman, a boy thought to be six years old and a girl aged between 12 and 16.
At least six others were injured, including the driver of the car, a 66-year-old Mount Macedon man.
Superintendent John Fitzpatrick said police were waiting to speak to the driver, who suffered non-life-threatening injuries.
Crash reconstruction specialists are trying to piece together what happened but Supt Fitzpatrick said diners were eating when the car came down the hill along Albert Street and left the road.
"We don't believe that speed was excessive," he said.
A three-year-old boy was flown to the Royal Children's Hospital with critical injuries while a man, a woman and a boy had non-life-threatening injuries.
"Families have got loved ones that are no longer going to be around," Supt Fitzpatrick said.
Victoria Ambulance regional director Trever Weston said initial emergency crews were confronted with a chaotic scene.
He commended the efforts of people who came to the aid of the injured until ambulance paramedics arrived.
First responders to the Daylesford crash were confronted by chaotic scenes.
"(Injuries were) quite traumatic for a number of those patients, particularly the four that were airlifted to hospital," he told reporters on Monday morning.
He said Daylesford was a close-knit community and some of the initial crews were from the area.
"You never want to respond to any incident like this but certainly not in your home town," Mr Weston said.
Paramedics will be offered welfare support in coming days and weeks.
"Any incident involving children is that next level of distress for our paramedics but the crews that responded last night did so professionally and provided the best care they could for all of those patients," Mr Weston said.
Hepburn Shire Mayor Brian Hood said Daylesford would have been crowded on Sunday, with many people enjoying the warm evening over the unofficial Melbourne Cup long weekend.
"This will send shock waves through the community for some time," he told ABC TV.
Police have asked the public to avoid Daylesford so emergency services can access the scene.
Infrastructure Minister Catherine King, whose federal Ballarat electorate includes Daylesford, asked the community to follow advice from emergency services as crews carry out their work.
"My thoughts are with all those at the scene, their friends and families and the broader Daylesford community," she said in a statement on X.
Anyone who witnessed the crash or has dash cam or CCTV footage is urged to contact Crime Stoppers.