The rising Greek star, expected to reach the latter stages at Roland Garros, fought back to win 4-6, 6-0, 6-3, 7-5 on Court Simonne Mathieu.
Tsitsipas will face either Spaniard Roberto Carballes Baena or Serbia's Filip Krajinovic in the last 32.
The 20-year-old started strongly with an early break, but appeared unsettled following a medical timeout for Dellien after the world number 86 appeared to twist his ankle.
Dellien quickly sealed the opening set, but Tsitsipas raced through the next two sets and seemingly towards victory.
Despite letting a 4-2 lead in the fourth set slip, the 25-year-old Dellien just would not go away, saving three match points from 0-40 down in the 10th game to level at 5-5.
But Tsitsipas did secure a third-round spot in Paris for the first time in his opponent's next service game, curling a forehand up the line on his fourth match point.
Results from the fourth day of the French Open on Wednesday (x denotes seeding):
Stefanos Tsitsipas (GRE x6) bt Hugo Dellien (BOL) 4-6, 6-0, 6-3, 7-5
Laslo Djere (SRB x31) bt Alexei Popyrin (AUS) 6-4, 7-6 (7/4), 6-4
Juan Ignacio Londero (ARG) bt Richard Gasquet (FRA) 6-2, 3-6, 6-3, 6-4
Corentin Moutet (FRA) bt Guido Pella (ARG x19) 6-3, 6-1, 2-6, 7-5
David Goffin (BEL x27) bt Miomir Kecmanovic (SRB) 6-2, 6-4, 6-3
Rafael Nadal (ESP x2) bt Yannick Maden (GER) 6-1, 6-2, 6-4
Sloane Stephens (USA x7) bt Sara Sorribes Tormo (ESP) 6-1, 7-6 (7/3)
Polona Hercog (SLO) bt Jennifer Brady (USA) 6-3, 6-7 (8/10), 6-4
Garbine Muguruza (ESP x19) bt Johanna Larsson (SWE) 6-4, 6-1
Elina Svitolina (UKR x9) bt Kateryna Kozlova (UKR) -- walkover
Carla Suarez Navarro (ESP x28) bt Shelby Rogers (USA) 6-1, 6-7 (4/7), 6-3
Elise Mertens (BEL x20) bt Diane Parry (FRA) 6-1, 6-3
Anastasija Sevastova (LAT x12) bt Mandy Minella (LUX) 6-2, 6-4
Veronika Kudermetova (RUS) bt Zarina Diyas (KAZ) 7-5, 6-1
Kaia Kanepi (EST) bt Shuai Zhang (CHN) 6-7 (4/7), 6-0, 7-5
Karolina Pliskova (CZE x2) bt Kristina Kucova (SVK) 6-2, 6-2