The main reason why The Anti-Federalists were so concerned about this was that they had just fought a war to remove themselves from tyranny, so they were extra sensitive.
The anti federalist believe that if the central government is given too much power, the country would be no different to the situation when we're colonized by the British Government, so they want to allocate the power to the people.