"We got to Townsville ... right up to the top of Cape Yorke Peninsula and Erin chose to fly just herself back to Perth," Simon told a jury of 15 people on Thursday.
He drove himself and their baby back to the west coast.
It would become the first of many separations across their "up and down" relationship, before they separated permanently in 2015, the estranged husband told day three of Erin Patterson's triple murder trial.
"I was always keen to have a good relationship, have a good marriage and a good strong family to bring up the kids in," Simon said.
"She decided to leave anyway?" prosecutor Nanette Rogers SC asked.
"Yeah, that's fair," Simon replied.
Erin Patterson felt their relationship was "toxic", the jury was told.
Simon glanced at his former wife twice across the morning, as she stared at him dressed in a pink shirt from the court dock.
One year before they married, Patterson inherited about $2 million from her grandmother's estate and was gradually giving the money over eight years, Simon said.
She was generous with his family and gave several loans to Simon's three siblings, including a $300,000 loan for his brother Matt to build a house, he said.
Simon cried as he recalled the moment he saw his mother and father, Don and Gail Patterson, after they consumed the death cap mushroom meal.
Don had eaten an entire portion of the deadly beef Wellington, plus half of his wife's meal, at the lunch in July 2023 at Patterson's Leongatha home.
"Dad was substantially worse than mum, he was really struggling," Simon said.
"He wasn't right inside. He was feeling pain."
Patterson, 50, has pleaded not guilty to three charges of murder and one of attempted murder, and claims she did not intend to harm or kill.
Simon's parents and aunt Heather Wilkinson all died in hospital between August 4 and 5.
His uncle Ian Wilkinson was sick but recovered.
Simon had been invited to the lunch, on July 29, 2023, but declined in a text the day before because he felt "too uncomfortable" about going, the jury was told.
"That's really disappointing, I've spent many hours this week preparing lunch for tomorrow," Erin Patterson replied within five minutes.
"And spent a small fortune on beef eye fillet to make beef Wellingtons because I wanted it to be a special meal, as I may not be able to host a lunch like this again for some time.
"It's important to me that you're all there tomorrow."
Simon did not reply and was not at the lunch.
He instead offered to collect their two children from the Leongatha cinema later in the afternoon, as Patterson wanted them out of the house to discuss a "serious matter" about her health, he said.
She told his in-laws she had ovarian cancer, and ask for advice on how to break it to her children, however her defence team has admitted she had never been diagnosed with cancer.
The morning after the lunch, Don called Simon to say he and Gail were vomiting and had diarrhoea and were waiting for emergency services to arrive.
He went to the Wilkinsons' home where he found Ian looking sick, "grey and struggling" while Heather was inside looking "pretty crook" on the couch.
"She had a container as a spew bucket," he said, about Heather.
Later that afternoon, Simon said he spoke to Erin on the phone and told her his family were "all crook and they'd all been admitted to hospital".
Erin told him she had been suffering diarrhoea and he said she was "worried that she'd poo her pants".
The trial before Justice Christopher Beale continues on Friday.
9°C