You seem to be assuming that people would keep driving as they currently do if we removed speed limits entirely. I’d be willing to bet that this is not the case. Most drivers have a number in mind on how much they’re willing to exceed the speed limit. For me that is 5 - 10kph, so if the limit is 60kph, then you’re not going to catch me going 80. Without speed limits I probably would.
So why do we have such laws? Because they work. Not perfectly but to some extent.
Concluding that someone is a lesser human because of their skin color seems just as illogical to me as calling someone a racist because they drive a lifted truck. These things are completely unrelated.