Downs, who were held 3-3 in the first leg, were undone by efforts from Man of the Match Lerato Chabangu (10, 14) and Bennett Chenene (19), while Teko Modise (22) and Nyasha Mushekwi (91) reduced the deficit at Dobsonville Stadium.
The Dube Birds will contest cup honours with SuperSport United in the final.
Neeskens said: "We made really bad mistakes. I think in a 10-minute period of time they scored three goals.
"Two of these goals were down to individual mistakes again. Lately we have been making too many of these individual mistakes. But that is football so we must keep going."
The Dutchman added: "I can't say they [his men] didn't fight. But it's very difficult to come back against a team like Swallows."