The Roman army had been largely decimated and did not hold well against Hannibal. If he had marched towards Rome, there wasn't much army to resist him. He would likely seize Rome. It is much harder to retake a city then to defend it, especially since Rome was symbolic. To take Rome would be to go straight to the heart and soul of the Roman people.