These are limestone stacks created by erosion due to wave action. The pile of rubble in the foreground was once a 50-meter high stack until July 2005, when it collapsed. This dropped the remaining number of Apostles from nine to eight.
I was standing at 38°39′49.95″S, 143°06′15.48″E (Google Maps) looking northwest. A visitor center was built across the Great Ocean Road from the coastline, so this site is easily accessible.