David Davis said "quite large exemptions" could be granted to businesses that are not given authorised economic operator status - "trusted traders" - to enable trade to continue between Northern Ireland and the Republic.
Mr Davis also said Parliament is "unlikely" to sign off the withdrawal deal unless it is a "substantive" agreement.
"The withdrawal agreement involves payments of up to £39 billion. It's a lot of money and Parliament is unlikely to sign off such a deal unless we can be pretty substantive what's going to be there in the long run," he said.
In an interview with Gerard Baker, editor in chief of the Wall Street Journal, at WSJ CEO Council Europe in central London, Mr Davis was asked how he would avoid a hard border on the island of Ireland.
He replied: "There's a tax border between the North and South, there's an excise border, there's a currency border. There is, in effect, no personnel border because of the Common Travel Area. The Common Travel Area was created in 1923, it's been there all the time, it's not going to go.
"So what we're talking about is, number one, how we continue to maintain a tax border - bear in mind we're looking for a zero-tariff outcome - a tax border and an excise border and so on, much as we do now."
Mr Davis said one "area of difficulty" was "very small businesses" straddling the border, such as farms and "tiny" companies.
"That is going to have to be addressed, we think, in the first case by quite large exemptions, so in effect we will give them tax-free status."
He told the conference he did not agree with shadow international trade secretary Barry Gardiner that the border issue is a "shibboleth".
Mr Gardiner was heard in a recording suggesting the row over the border issue and the accord had been "played up" and he described the deal as a shibboleth, a Hebrew term used to describe a long-held custom that is outdated.
Mr Davis said he was "old enough, I'm sorry to say, to have lived through it all, so I remember the beginning of the Troubles".
"I do not agree with one Labour politician who s