The German visitors had wasted a string of good openings by the time Huntelaar put them in front with a low drive in the 76th minute, his ninth goal of the season for club and country.
Afellay clinched victory for Schalke in the 86th minute, slotting home from close range after Jefferson Farfan had evaded Arsenal's defence again down the right.
The loss ended Arsenal's perfect start to the competition and leaves manager Arsene Wenger with some worrying problems after being outclassed in every department.
Schalke are now on seven points - one ahead of Arsenal.
Champions League results on Wednesday:
Group A
At Oporto, Portugal
Porto (POR) 3 (S. Varela 15, J. Martinez 36, 78) Dynamo Kiev (UKR) 2 (Gusev 21, Ideye 72)
At Zagreb
Dinamo Zagreb (CRO) 0 Paris Saint Germain (FRA) 2 (Ibrahimovic 32, Menez 43)
Group B
At London
Arsenal (ENG) 0 Schalke 04 (GER) 2 (Huntelaar 76, Afellay 86)
At Montpellier, France
Montpellier (FRA) 1 (Charbonnier 49) Olympiakos (GRE) 2 (Torosidis 73, Mitroglou 90+1)
Group C
At Amsterdam, The Netherlands
Ajax (NED) 3 (S. de Jong 45, Moisander 57, Eriksen 68) Manchester City (ENG) 1 (Nasri 22)
At Dortmund, Germany
Borussia Dortmund (GER) 2 (Lewandowski 36, Schmelzer 64) Real Madrid (ESP) 1 (Ronaldo 38)
Group D
At Malaga, Spain
Malaga (ESP) 1 (Joaquin 64) AC Milan (ITA) 0
At St Petersburg, Russia
Zenit St Petersburg (RUS) 1 (Kerzhakov 72-pen) Anderlecht (BEL) 0