Okay, collective brain, help me out here.
There almost certainly already is a term for the retort in the form of:
"I don't think that's offensive, because it doesn't bother me, personally"
If not, perhaps we might coin such a term?
[Much of the effort thus far has been expended on trying to attribute the above retort to a mindset, rather than in characterizing the speech act on its own terms.]