Not necessarily. If they are the good ones that have the correct centre bore size and proper nuts etc. they should be fine. There are thousands of people who track with spacers on.

Its the dodgy ones that are just a metal plate which start snapping studs.

Cheers