By Bus: Most of the buses on the Panamerican Highway from Lima to Arequipa, Chincha, Ica etc. won't leave this road, dropping you of at the junction about 4 miles outside town, from where you can pick up a local bus or taxi. Some buses from Lima (3-4hrs) do go in to town and there are local buses to Ica.
Ica is only one to one and half hour. There are also some shared taxis, which should be slightly more expensive then the USD 1 the bus costs.