The two-time winners shot out of the blocks to lead 22-3 after 28 minutes only for the Chiefs to storm back and level the scores at 25-25 with under 30 minutes left.
But the Brumbies hung on with a push-over try to flanker Jarrad Butler on the hour.
All Blacks flyhalf Aaron Cruden had the chance to send the game into extra-time but missed a sideline conversion attempt after replacement Gareth Anscombe's try three minutes from time.
The Brumbies now await the winner of Saturday's other play-off match in Durban between the Sharks and the Highlanders.
Should the Sharks win, the Canberra-based side will have a mouth-watering showdown with the Waratahs in a semi-final in Sydney next weekend.
"We knew it was going to have to be a full 80-minute performance, you just cannot knock a side like that over easily," Brumbies captain Ben Mowen said.
"We're very proud of that effort and you've got to give the Chiefs a lot credit on what they've done over the last three years. It's been an outstanding effort to be there."
"We've been working hard on our skills not only over the last few weeks but probably over the last couple of months, getting good transfer into the game and it's exciting to play," Mowen said.
"If we keep getting results like that you have to keep backing it," he added.
The home side had the Chiefs firmly on the back-foot with exhilarating passing rushes leading to tries to Nic White, Robbie Coleman and Jesse Mogg.
They could do little wrong as Waikato struggled to stay in the contest, but the Kiwis, who beat the Brumbies in last year's final in Hamilton, began to find some rhythm and centre Bundee Aki scored a try five minutes before half-time to trail 22-10.
While it was all the Brumbies in the opening half, it was the opposite in the second term with the Chiefs playing disciplined controlled rugby and working through the phases to put the home side under pressure.
The Chiefs capitalised on Brumbies wing Henry Speight's yellow card in the 48th minute to score two tries in his absence from scrumhalf Tawera Kerr-Barlow and centre Tim Nanai-Williams to tie up the game.
But as the Chiefs looked as though they would go on and win the game, the Brumbies got their big break with Butler's try off a push-over maul close to Waikato's try-line.
Christian Lealiifano's conversion gave the home side a seven-point cushion but the Chiefs again showed their attacking potency with Anscombe scoring in the corner, only for Cruden to send the difficult conversion attempt wide of the posts with minutes left.
The Brumbies killed off the game with pick and drives inside the opposing quarter as the clock ticked down and Matt Toomua kicked the ball into the touch with time up to ensure victory.
"We started pretty slowly and gave a quality side like the Brumbies a lead like that," Chiefs skipper Cruden said.
"We did really well to peg it back and showed a lot of character, but in these playoff games you need to be clinical for 80 minutes."
Scorers
Chiefs - Tries: Bundee Aki, Tawera Kerr-Barlow, Tim Nanai-Williams, Gareth Anscombe; Conversion: Aaron Cruden (2); Penalties: Cruden (2)
Teams:Brumbies
15 Jesse Mogg, 14 Henry Speight, 13 Tevita Kuridrani, 12 Christian Leali'ifano, 11 Robbie Coleman, 10 Matt Toomua, 9 Nic White, 8 Ben Mowen (captain), 7 Jarrad Butler, 6 Scott Fardy, 5 Leon Power, 4 Sam Carter, 3 Ben Alexander, 2 Josh Mann-Rea, 1 Scott Sio
Substitutes: 16 Ruaidhri Murphy, 17 Ruan Smith, 18 Allan Alaalatoa, 19 Fotu Auelua, 20 Tom McVerry, 21 Michael Dowsett, 22 Joe Tomane, 23 Pat McCabe
Chiefs
15 Dwayne Sweeney, 14 Asaeli Tikoirotuma, 13 Tim Nanai-Williams, 12 Bundee Aki, 11 James Lowe, 10 Aaron Cruden (captain), 9 Tawera Kerr-Barlow, 8 Liam Squire, 7 Tanerau Latimer, 6 Liam Messam, 5 Brodie Retallick, 4 Mike Fitzgerald, 3 Ben Tameifuna, 2 Mahonri Schwalger, 1 Jamie Mackintosh
Substitutes: 16 Nathan Harris, 17 Pauliasi Manu, 18 Nick Barrett, 19 Matt Symons, 20 Tevita Koloamatangi, 21 Augustine Pulu, 22 Gareth Anscombe, 23 Tom Marshall