His Orica GreenEdge teammate Simon Gerrans held on to the yellow jersey finishing 15th in the stage won by British sprinter, Mark Cavendish.
Impey was part of the nine-man team which won the 25km team time trial by less than one second on Tuesday, the fourth-stage of the Tour, which was a mainly flat course ride around Nice.
This result pushed him to second on the leaderboard -- the highest ever position in the overall standings by a South African.
The team, which made its race debut last year, picked up its first Tour stage win on Monday after Gerrans won the third stage in a sprint finish and took over the overall lead from Belgian rider Jan Bakelants.
Impey was instrumental in that victory as he flawlessly led out Gerrans to his first win on the tour.
The 28-year-old from Johannesburg started his campaign on Saturday with an impressive 11th place finish.
He finished eighth in Sunday's stage-two, 156km ride across Corsica from Bastia to Ajaccio, putting him fourth on the standings.
Results of the Tour de France fifth stage, a 228.5km ride from Cagnes-sur-Mer to Marseille on Wednesday:
1. Mark Cavendish (GBR/OPQ) 5hr 31min 51sec
2. Edvald Boasson Hagen (NOR/SKY) at 0:00.
3. Peter Sagan (SVK/CAN) 0:00.
4. Andr Greipel (GER/LTB) 0:00.
5. Roberto Ferrari (ITA/LAM) 0:00.
6. Alexander Kristoff (NOR/KAT) 0:00.
7. Juan Jos Lobato (ESP/EUS) 0:00.
8. Ramunas Navardauskas (LTU/GRM) 0:00.
9. Cyril Lemoine (FRA/SOJ) 0:00.
10. Jos Joaquin Rojas (ESP/MOV) 0:00.
11. Samuel Dumoulin (FRA/ALM) 0:00.
12. John Degenkolb (GER/ARG) 0:00.
13. Daryl Impey (RSA/ORI) 0:00.
14. Danny van Poppel (NED/VAC) 0:00.
15. Simon Gerrans (AUS/ORI) 0:00.
16. Egoitz Garca (ESP/COF) 0:00.
17. Fabio Sabatini (ITA/CAN) 0:00.
18. Gert Steegmans (BEL/OPQ) 0:00.
19. Wout Poels (NED/VAC) 0:00.
20. Julien El Fars (FRA/SOJ) 0:00.