"If [Chad] Le Clos' win did not make us believe, then the gold won by our rowing team must have done the trick," Mbalula told reporters in Johannesburg.
"We acknowledge that it is still early to celebrate, but when you have been starved of success for so long you cannot help it but celebrate every little achievement made."
South Africa has won three Olympic gold medals since the start of the games last weekend.
Swimmer Cameron van der Burgh was the first to scoop the gold medal when he won the 100m breaststroke with a world-record time of 58.46 seconds.
A few days later, Chad Le Clos beat acclaimed US swimmer Michael Phelps to win the men's 200 metre butterfly final.
The lightweight four men's rowing team secured South Africa's third gold medal of the Games on Thursday.
Mbalula said: "When you win this big, you can't remain quiet".
The minister arrived back from London on Thursday, where he had attended the opening ceremony of the games.
While there he also attended the commonwealth sport ministers' meeting, officiated at the flag-raising ceremony at the Olympic village and officially opened the Ekhaya Hospitality Centre.
Mbalula said the Olympics was an opportunity for SA to market itself.
"As we market ourselves, we do so as a global player that has footprints in its home continent and the world in general," he said.
It also offered SA the opportunity to invite the world to partner the country in socio-economic growth and development.
"Our team is still in London and while others may fall and learn from their experience, others will stand tall like Cameron, Chad and the rowing team did. We hope for more medals, and we wish for more," Mbalula said.