Martin completed the 32.5km time trial from Villars-les-Dombes to Parc des Oiseaux in just under 37 minutes.
Rohan Dennis of Australia took the overall lead from David Veilleux of Canada by finishing second, 47 seconds behind Martin.
Froome wound up third, 2:45 ahead of Contador.
A two-time Tour de France winner, Contador was also overtaken by Australian rider Richie Porte, who started two minutes behind him.
Froome, the runner-up behind Bradley Wiggins at last year's Tour, climbed to second place in the overall standings.
Thursday's fifth stage is a tough 139km ride from Gresy-sur-Aix to the ski resort of Valmorel.
Results from the fourth stage of the Criterium du Dauphine, individual time trial from Villars-les-Dombes to Parc des Oiseaux:
1. Tony Martin, Germany, Omega Pharma-QuickStep, 36 minutes, 54 seconds.
2. Rohan Dennis, Australia, Garmin-Sharp, 47 seconds behind.
3. Christopher Froome, Britain, Sky Procycling, 52 seconds behind.
4. Jonathan Castroviejo, Spain, Movistar, 1:08.
5. Michal Kwiatkowski, Poland, Omega Pharma-QuickStep, 1:13.
6. Edvald Boasson Hagen, Norway, Sky Procycling, 1:19.
7. Richie Porte, Australia, Sky Procycling, 1:20.
8. Jan Barta, Czech Republic, Team NetApp-Endura, 1:36.
9. Marco Pinotti, Italy, BMC Racing Team, 1:38.
10. Geraint Thomas, Britain, Sky Procycling, 1:42.
Overall Standings (After 4 of 8 stages):
1. Rohan Dennis, Australia, Garmin-Sharp, 12:40:00.
2. Christopher Froome, Britain, Sky Procycling, 5 seconds behind.
3. Michal Kwiatkowski, Poland, Omega Pharma-QuickStep, 26 seconds behind.
4. Edvald Boasson Hagen, Norway, Sky Procycling, 32 seconds behind.
5. Richie Porte, Australia, Sky Procycling, 33 seconds behind.
6. Geraint Thomas, Britain, Sky Procycling, 55 seconds behind.
7. David Veilleux, Canada, Team Europcar, 1:09.
8. Leopold Konig, Czech Republic, Team NetApp-Endura, 1:11.
9. Stef Clement, Netherlands, Blanco Pro Cycling, 1:14.
10. Andriy Grivko, Ukraine, Astana, 1:26.