Grace, a triple winner on the tour in 2012, clipped two strokes off Lee Westwood's previous Kingsbarns record of 62 set in 2003 after finishing with five straight birdies.
Frenchman Victor Dubuisson was two shots off the lead on 62 after breaking the course record at St Andrews, one of three layouts being used this week.
Grace became the 14th player to shoot a 60 on the European Tour.
Northern Ireland's Darren Clarke, the 2011 British Open winner, is the only man to card two 60s on the circuit, in 1992 and 1999.
Long-hitting Frenchman Dubuisson broke the previous St Andrews best of 63 held by world No 1 Rory McIlroy, former top-ranked player Luke Donald and fellow Briton Simon Dyson.
Young Dane Thorbjorn Olesen was also on form at the famous Old Course, returning a 63 in the $5m Dunhill pro-am tournament.
Ryder Cup hero Martin Kaymer (70) played at the more testing Carnoustie layout and missed a par putt at the 18th of a similar length to the one he holed in Chicago last Sunday to win the trophy for Europe.
"The putter was not really hot but in general you've got to play solid at the toughest course we play here," the 2010 US PGA champion told reporters.
"It's a very nice feeling to come here after Sunday. A lot of people told me it was their best sporting experience on TV," added the German.
Europe came back from 10-6 down to score a stunning 14 1/2 - 13 1/2 win over United States in the biennial team event.
Kaymer's Ryder Cup team-mates Paul Lawrie (75) and Peter Hanson (72), both playing at Carnoustie, also found the going tough.
British Open champion Ernie Els returned a 73 at Carnoustie where the best score midway through round one was a 67 by England's Oliver Wilson.
At Kingsbarns
60 - Branden Grace (RSA)
65 - Joel Sjoholm (SWE), Anton Haig (RSA)
66 - Magnus A Carlsson (SWE), Alejandro Canizares (ESP)
67 - Richard Bland, Kristoffer Broberg (SWE), Andrea Pavan (ITA)
68 - Craig Lee, Anthony Wall, Jorge Campillo (ESP), Fabrizio Zanotti (PAR), Simon Khan, Graeme Storm
69 - Damien McGrane, Estanislao Goya (ARG), Paul Waring, Matthew Baldwin, Jose Manuel Lara (ESP), Gary Boyd, Shane Lowry, Lorenzo Gagli (ITA), Bradley Dredge
70 - Peter Whiteford, Kieran Pratt (AUS), Peter Lawrie, Josh Geary (NZL), Emiliano Grillo (ARG), Andy Sullivan, Keith Horne (RSA), Gregory Bourdy (FRA)
Selected
71 - Terry Pilkadaris (AUS), Nick Cullen (AUS), Darren Fichardt (RSA), Adam Crawford (AUS), Christiaan Basson (RSA), Richard Green (AUS), Edoardo Molinari (ITA)
72 - Jaco Van Zyl (RSA), Matthew Griffin (AUS)
73 - Mohd. Siddikur (BAN), David Gleeson (AUS)
74 - Allan Versfeld (RSA), Himmat Singh Rai (IND)
78 - Ryan Haller (AUS)
At St Andrews
62 - Victor Dubuisson (FRA)
63 - Thorbjorn Olesen (DEN)
64 - Alexander Noren (SWE), Ashley Hall (AUS), Fredrik Andersson Hed (SWE)
65 - Richie Ramsay, Jamie Donaldson, Thongchai Jaidee (THA)
66 - David Drysdale, Tommy Fleetwood, Danny Willett, Richard Sterne (RSA), Oliver Fisher
67 - Daniel Denison, Stephen Gallacher, Michael Campbell (NZL), Pablo Martin (ESP), Rikard Karlberg (Swe), Lee Slattery
68 - James Kingston (RSA), Joost Luiten (NED), Thomas Bjorn (DEN), Benjamin Hebert (FRA), Dylan Frittelli (RSA), Scott Jamieson
69 - George Murray, Soren Kjeldsen (DEN), Chris Wood, Richard Finch, Sam Little, Raphael Jacquelin (FRA)
70 - Gary Lockerbie, Marc Warren, Ross Fisher, Anders Hansen (DEN), David Lynn, James Morrison
Selected
71 - Brett Rumford (AUS), Shiv Kapur (IND), Jbe' Kruger (RSA), Andrew Dodt (AUS), George Coetzee (RSA)
73 - Miguel Angel Jimenez (ESP)
74 - Matthew Zions (AUS), Peter O'Malley (AUS)
At Carnoustie
67 - Oliver Wilson, Gregory Havret (FRA)
69 - Alex Cejka (GER)
70 - Steve Webster, Rich Beem (USA), Martin Kaymer (GER)
Selected
71 - Colin Montgomerie, Louis Oosthuizen (RSA), Padraig Harrington
72 - Hennie Otto (RSA), Thomas Aiken (RSA), Peter Hanson (SWE), Tjaart Van Der Walt (RSA)
73 - Pariya Junhasavasdikul (THA), Ernie Els (RSA)
75 - Charl Schwartzel (RSA), Darren Clarke, Rohan Blizard (AUS), Paul Lawrie, Shaun Micheel (USA)
76 - Paul McGinley