They're safe to do without, but if they're installed as dependencies of other packages, then you won't be allowed to remove them without force. And if you do force them they'll come right back the next time you upgrade a package depending on them.
In that case, to weed that out, you's have to remove the metapackages containing them. The chain for these ones goes like this:
xorg -> xorg-video-drivers -> xf86-video-(
$chip) -> mesa-(
So if you really don't want them to be there, then you'll want to remove the metapackages in the order above.
This will leave (many) orphan packages. If you want to keep them (and you probably want at least some of them) then you'll have to either never purge orphan packages, or go through and mark them as manual instead of auto.