Johnson said the yellow card handed to lock Jim Hamilton in the 52nd minute was the turning point as injury-ravaged Scotland went down to 14 men and saw their 10-6 halftime lead overturned.
"The punishment did not fit the crime. It was handbag's stuff, it ruined a great game," the Australian coach told reporters at the Mbombela Stadium.
Hamilton was sent to the sinbin after French referee Romain Poite asked for a television review of an off-the-ball push in the face of Springbok lock Eben Etzebeth.
"I'm bitterly disappointed," said Johnson. "I'd go as far as to say we deserved to win but with 14 men it's not easy.
"We weren't playing tiddlywinks, it's an aggressive sport. They've just asked me if I'd like to cite anybody - probably the fourth official."
Johnson said putting a team together on Saturday proved a trial after several injuries were picked up last week against Samoa in Durban.
Scotland flew out three replacements after a bruising 27-17 defeat by Samoa and patched up other players for the second of their three matches in South Africa.
"We had guys playing today who had no right to even be on the pitch," he said in a reference to forwards Euan Murray and Johnnie Beattie. "To say I'm proud of how we played is an understatement."