The dispute is over checks on goods coming into Northern Ireland from the UK mainland.
Lord Frost says he has not given up on talks aimed at resolving a post-Brexit trade dispute with Brussels.
Since it quit the EU, London has refused to implement some border checks on goods coming into Northern Ireland from the British mainland.
The checks were agreed as part of Brexit and aim to stop a hard border between Northern Ireland and the Republic of Ireland.
The EU says tighter controls are necessary to protect its single market. It has relaxed its demands in recent weeks in a bid to find an agreement with London.
But with no sign of a deal, the UK has threatened to trigger Article 16 of the Northern Ireland Protocol, the Brexit deal for Northern Ireland.
"They seem to be claiming that it will be entirely unreasonable for the British government uniquely to use these wholly legitimate safeguard provisions [Article 16] ... they're also suggesting that we can only take that action at the price of massive and disproportionate retaliation," Frost told the UK parliament's upper chamber.
"I gently suggest that our European friends should stay calm and keep things in proportion."
Nevertheless, Frost insists he is not ready to walk away from talks yet.
"If the talks do in the end fail, we will of course publish in full our assessment of the EU's proposals and set out why they fall short of a durable settlement, but we will not do that until we have exhausted all the negotiating possibilities. For now, I wish to preserve the integrity of the negotiations and remain positive," Frost said in a statement.
"Accordingly, we continue to work to see whether the EU position on these issues can yet develop further and whether it is possible to find a way to deal with the other important matters necessary to put the protocol on a sustainable footing. […] That work will continue in the talks underway this week. In my view, this process of negotiations has not reached its end," he went on.
Brussels has yet to announce any measures if this safeguard clause is triggered.
However, Ireland's deputy leader Leo Varadkar warned the bloc would strike back if the UK triggered Article 16.
Varadkar said on Tuesday that Ireland is making "contingency plans" should this happen and that officials were "dusting off" plans that were previously made ahead of fears of a no-deal Brexit.
Further talks between the UK and Europe are expected on Friday.