The victory in the provincial derby lifted Stars five points above bottom club Maritzburg United in a league where only the last team automatically drops to the second division.
Just six points separate the half a dozen clubs closest to Maritzburg, setting the scene for a dogfight until the season ends in May to avoid coming second last and going into the play offs.
Harris Tchilimbou from Congo Brazzaville put visiting Stars ahead on 28 minutes with his fifth Premiership goal this season and Sinethemba Jantjie bagged a brace within four minutes.
Seventh-place Celtic rattled Stars in the final 20 minutes with goals from Mbhazima Rikhotso and Sepana Letsoalo setting up a tense finish as the home side sought an equaliser.
Elsewhere on Sunday, Black Leopards had a player sent off and struck the woodwork twice as they drew 0-0 at home against Highlands Park in remote northeastern town Thohoyandou.
Khuliso Mudau was red-carded on 63 minutes after a second booking while Lefa Hlongwane, who was voted man of the match, saw shots come back off a post in each half.
Relegation-threatened Leopards rose one place to 13th while Highlands remained ninth, 12 points below leaders Wits and 12 above Maritzburg.