The peer, who is no longer a member of the UUP, also said he had no intention of apologising for referring to Mr Varadkar as “the Indian” in a Twitter message last Thursday.
Mr Varadkar is the son of an Indian doctor and an Irish nurse.
“I got about 40 or 50 tweets and most of them were from these foul-mouthed Irish republicans,” Lord Kilclooney told the BBC’s the View on Thursday night.
“Reasonable people sent me messages of support and understood what I was saying.
“Anyone who knows me knows I’m not racist and I mix with Indian people all the time,” Lord Kilclooney added.