Kohli hit 115 off 108 balls in the dominant performance of the game after replacing regular captain Mahendra Singh Dhoni, who was rested for the limited-overs series.
Kohli and Rayudu (63) combined for a 159-run partnership for the third wicket to convincingly take the game away from Zimbabwe, which had been held to 228-7 after India won the toss and bowled.
India legspinner Amit Mishra took 3-43 on a turning pitch at Harare Sports Club to lead India's bowling effort.
Pakistan-born batsman Sikandar Raza top-scored for Zimbabwe with 82 for his first ODI half-century.
Scores:
India 230-4 in 44.5 overs (Virat Kohli 115, Ambati Rayudu 63 not out) def. Zimbabwe 228-7 in 50 overs (Sikandar Raza 82, Elton Chigumbura 43 not out; Amit Mishra 3-43).