The Japan-based forward equalised in the first half and scored the 80th-minute winner in a lively east African derby.
Johanna Omolo was the other scorer for Kenya, who trailed 2-1 at half-time.
Saimon Msuva and Mbwana Samatta netted for Tanzania, who have never
won a Cup of Nations match in two appearances spanning 39 years.
Earlier, Algeria defeated Senegal 1-0 in a foul-riddled first match
of a Group C double-header at the 30 June Stadium and both countries are
set to make the last 16.
Even if Kenya lose their final group match against Senegal, they will
hope to squeeze into the knockout phase as one of the four best
After two rounds, former champions Algeria have six points, Senegal and Kenya three each and Tanzania are pointless.
Coaches Emmanuel Amunike of Tanzania and Sebastien Migne of Kenya
reacted to two-goals losses in their opening group matches by making
three changes each.
France-born Migne spoke before the match of the need "to start well"
but Tanzania were the more committed side in the early exchanges and led
after only six minutes.
Kenya goalkeeper Patrick Matasi could only parry a stinging shot from
Taifa Stars captain Samatta and Msuva tapped in the rebound.
The Harambee Stars were unlucky not to score midway through the half
when headers by Francis Kahata and Olunga struck the woodwork within
seconds of each other.
Msuva nearly scored a second goal when his free-kick from well
outside the box was heading for the corner of the net until Matasi
tipped the ball over.
An intense match produced two more goals inside a minute close to half-time with Kenya levelling and then falling behind again.
When Tanzania goalkeeper Aishi Manula flapped at a free-kick, the
ball struck team-mate Erasto Nyoni and drifted toward Olunga, whose
bicycle kick found the net.
The Kenyan joy was shortlived, however, as Samatta punished clumsy
defending to fire the ball into the net and give Tanzania a 2-1
There was no let-up in the physicality from both teams as the second
half progressed and Manula made a superb reflex save to parry a header
from Kenya captain Victor Wanyama.
But Manula could not prevent Kenya equalising a second time on 62
minutes when a cross after a short corner was powerfully nodded into the
net by Johanna Omolo.
It got worse for Tanzania 10 minutes from time when they fell behind
for the first time as Olunga fired the ball into the net off a post
from the edge of the box.