Nah, it's a load of Boogie Woggie Shane. Plenty of guys run wheels without the correct centre hole so what difference is that?
On a race car I'd think twice (then probably still do it)
On a road car, with you driving (ie; someone who maintains their car and knows if a wheel is not right before it comes right off) will be Ok.
Having said that, we all know spacers are far from perfect, and probably slightly illegal.