The proposed merger, first announced in January, would have created the world's largest mining company, with a market value exceeding $US200 billion ($A286 billion).
It is the second round of failed discussions in just over a year, following an earlier approach by Glencore in late 2024 and the third in total.
Talks late last year were also initiated by Glencore, according to a source familiar with the matter.
Glencore's shares closed 7 per cent lower at 467 pence.
Rio Tinto's London-listed shares were down 2.6 per cent at 6,820 pence.
Rio Tinto told shareholders on Thursday that it "is no longer considering a possible merger or other business combination with Glencore".
Attempts to combine the companies have repeatedly fallen short.
The British-Australian company also rejected a merger approach from Glencore in 2014, saying it was not in the best interests of shareholders.
However, the latest round of discussions marked a departure from past efforts.
The source described it as "the first time there has ever been a really serious, rigorous due diligence process".
Although transition metal copper was an obvious motivation for a deal, Rio Tinto was seeking to acquire Glencore in its entirety, including its coal assets and marketing business.
"We concluded that the proposed acquisition ... does not reflect our view on long-term, through the cycle relative value, including not adequately valuing our copper business, and its leading growth pipeline," Glencore said in a statement.
Glencore talked up its copper assets at an investor day in December, when it said it aims to reach 1.6 million metric tons by 2035 through new and restarted mines and streamlined operations, from 852,000 tons last year.
Global copper demand is expected to rise 50 per cent by 2040, benefitting from the energy transition and artificial intelligence demand, and global miners are racing to bulk up.
Analysts at HSBC had estimated an average deal premium of 30 per cent, which would have given Glencore's shareholders 38 per cent of a combined company.
The companies did not reveal the terms proposed and rejected.
The abandoned talks echo other ambitious mining deals that have faltered, including BHP's $US49 billion approach for Anglo American, which unravelled over concerns about the structure of the offer.
The only deal still proceeding is a plan for a $US53 billion all-stock, nil-premium merger between London-listed Anglo American and Canada's Teck Resources that would create the world's fifth-largest copper producer.
with PA
20°C