Supposedly Julius Caesar was worried about this same thing. When he was around 30 he came upon a statue of Alexander, who at his age had conquered the known world. Caesar had very little to show for himself at that point.

He was in his late thirties and early forties before he had any remarkable success or power.