In a match between the fifth and sixth ranked players in the world, McDowell jumped out into an early lead when McIlroy three-putted the first and then fell two down after hitting into the lake on the long third.
McIlroy won the fifth to cut the gap, but McDowell was two up again thanks to a birdie two on the short next.
Casares - The younger of the two Irishmen McIlroy then three-putted again at the long eighth to fall three down, but he pulled out an invaluable 18-footer on the next to avoid turning four behind.
There was no way back from there for McIlroy as McDowell held firm to clinch a 3 and 2 win that puts him into a quarterfinal matchup against in form Belgian Nicolas Colsaerts who edged Venezuela's Jhonattan Vegas at the par-five 18th.
"It's hard to get into the mood to play against one of your best friends, but you have got to, It was great to win a quality match," McDowell said.
"I got bragging rights till the next time but Rory will be winning multi tournaments in the next few years."
Match of the round as it turned out was between Ryder Cup European stars Lee Westwood and Ian Poulter.
Poulter led for most of the way but world No 1 Westwood hung on to get all square at the 15th.
The match turned at the par-three 17th where Westwood hit to 12 feet only for Poulter to get inside him to four feet. Westwood missed his putt and Poulter duly sunk his to move back to one up with one to play.
Both were on in three at the par-five last, and Poulter clinched the win by sinking another four footer after Westwood had secured his birdie.
World No 2 Martin Kaymer and No 3 Luke Donald both reached the last eight, the German defeating Soren Kjeldsen of Denmark 3 and 2 and the Englishman edging Johan Edfors of Sweden at the first extra hole.
Home hope Alvarao Quiros hit back to defeat South Korea's Noh Seung-yul at the 18th, with Masters champion Charl Schwartzel winning likewise against England's Ross Fisher.
Francesco Molinari of Italy completed the quarterfinal line-up by comfortably defeating Aaron Baddeley of Australia 3 and 2. He next goes up against Poulter.
Results:
Spain Alvaro Quiros (Spain) beat Noh Seung-Yul (South Korea) by 1 hole
Martin Kaymer (Germany) beat Soren Kjeldsen (Denmark) 3&2
Charl Schwartzel (South Africa) beat Ross Fisher (Britain) by 1 hole
Luke Donald (Britain) beat Johan Edfors (Sweden) on 19th hole
Nicolas Colsaerts (Belgium) beat Jhonattan Vegas (Venezuela) by 1 hole
Graeme McDowell (Britain) beat Rory McIlroy (Britain) 3&2
Francesco Molinari (Italy) beat Aaron Baddeley (Australia) 3&2
Ian Poulter (Britain) beat Lee Westwood (Britain) by 1 hole