What he's saying is that white people have the privilege of distancing themselves from the bad. That is all.

And he's only criticizing those who chose to become defensive rather than realize that doing so makes them part of the problem.