The Spaniard was barely tested as he recorded a routine 7-5 6-3 6-3 win over unseeded Croatian Ljubicic but he was far from satisfied after their encounter on court Philippe Chatrier.
"I'm not playing well enough to win this tournament," world No 1 Nadal told a news conference after improving his Roland Garros record to 42-1 to set up a rematch of last year's final with fifth seed Robin Soderling in the last eight.
"Sometimes I still make a few mistakes in a row and this cannot happen again."
Nadal, who handed Ljubicic his fourth straight-sets defeat on clay in as many meetings between the two, struggled in his first two matches at Roland Garros against low-key opponents.
However, he did make light work of Croatian qualifier Antonio Veic in the previous round.
Nadal was threatened only once by Ljubicic when it was 5-4 to the Croat and the score 30-30 on the Spaniard's serve.
The nine-times grand slam champion then reeled off 10 out of 11 points to bag the opening set.
"I played badly, then I was much better and he started to make a lot of mistakes," said Nadal.
The world No 1 clinched victory when Ljubicic, who struggled with his opponent's high-bouncing forehands, returned long.
"It wasn't a bad match from my side," said Ljubicic. "It's just the serve was not there and without a good serve against anybody, it's tough, but against Rafa it's almost impossible."
According to Ljubicic, Nadal's next opponent can start to worry.
"The first set I'm sure there were points where he felt like he could have played better but I think now he has four matches under his belt and if he manages to win the next one he's back in business, confidence wise," said the Croat.
"I think today's match is going to give him quite a lot of confidence. When he relaxed after the first set he played high-level tennis."
Results at Roland Garros on Monday:
Men
Fourth Round
Juan Ignacio Chela, Argentina, def. Alejandro Falla, Colombia, 4-6, 6-2, 1-6, 7-6 (5), 6-2.
Rafael Nadal (1), Spain, def. Ivan Ljubicic, Croatia, 7-5, 6-3, 6-3.
Gael Monfils (9), France, def. David Ferrer (7), Spain, 6-4, 2-6, 7-5, 1-6, 8-6.
Robin Soderling (5), Sweden, def. Gilles Simon (18), France, 6-2, 6-3, 7-6 (5).
Andy Murray (4), Britain, vs. Viktor Troicki (15), Serbia, 4-6, 4-6, 6-3, 6-2, suspended, darkness.
Women
Fourth Round
Li Na (6), China, def. Petra Kvitova (9), Czech Republic, 2-6, 6-1, 6-3.
Victoria Azarenka (4), Belarus, def. Ekaterina Makarova, Russia, 6-2, 6-3.
Maria Sharapova (7), Russia, def. Agnieszka Radwanska (12), Poland, 7-6 (4), 7-5.
Andrea Petkovic (15), Germany, def. Maria Kirilenko (25), Russia, 6-2, 2-6, 6-4.