The Alternative Arrangements Working Group, with Leave and former Remain MPs will meet for the first time on Monday after the Commons voted to find another way of avoiding a hard Irish border.
Home Secretary Sajid Javid has said "existing technology" could be used.
The Irish PM said the UK was reviewing ideas that had "already been rejected".
Leo Varadkar told RTE radio it was "very frustrating" the UK government was "going back to the idea of technology".
The UK is due to leave the EU at 23:00 GMT on Friday 29 March, when the two-year time limit on withdrawal negotiations enforced by the Article 50 process expires.
Brexit: A really simple guide
How ready is UK for a no-deal Brexit?
Who is in the working group?
Number 10 said the working group of Conservative MPs was set up following "significant support" for the so-called "Malthouse Compromise" - named after housing minister Kit Malthouse who encouraged talks between different groups of MPs.
Engineered by both Leavers and Remainers, the proposal includes extending the transition period for a year until the end of 2021 and protecting EU citizens' rights, instead of using the backstop.
Members of the working group include Conservative MPs Steve Baker, Marcus Fysh, Owen Paterson, Damian Green and Nicky Morgan.
The group will hold regular meetings with Brexit Secretary Stephen Barclay, as well as senior government officials from HMRC, Cabinet Office Europe Unit and Number 10.
Mr Baker, deputy chairman of the Eurosceptic group the European Research Group (ERG), warned on Sunday there was "trouble ahead" for the prime minister.
He tweeted: "Leave-backing MPs voted to support alternative arrangements in NI but with grave misgivings about the whole agreement."
The working group will also hold meetings on Tuesday and Wednesday.
Analysis
By political correspondent Chris Mason
Deadlock remains. Brexit is just over seven weeks away. So the government is thrashing around for a solution.
The government's willingness to provide civil service support from Number 10, HMRC and the Cabinet Office to this group of Conservative MPs is an indication of how seriously Downing Street is treating their idea.
But it is also still attempting to salvage the current blueprint, or a variation of it. The Attorney General, Geoffrey Cox, the government's most senior lawyer, is examining the legal changes to the backstop that could be requested.
The bottom line is this: the government knows it has to return to Brussels with something to ask for. And the Prime Minister could be back there by the end of the week. So they need to come up with something, and quick.
What is the backstop?
By Tom Edgington, BBC Reality Check
Media captionConfused by Brexit jargon? Reality Check unpacks the basics
The backstop is an "insurance policy" - designed to avoid a hard border "under all circumstances" between Northern Ireland and the Irish Republic.
Under the current Brexit deal, the 310-mile Irish border would become the only land border between the EU and the UK.
This would probably mean checks on goods crossing it, unless both sides could reach a comprehensive trade deal.
If such an agreement could not be reached, then to avoid those checks and border posts or other infrastructure, the backstop would come into force.
It would keep the UK in a "single customs territory" with the EU and leave Northern Ireland effectively in the EU's single market for goods.
A number of MPs fear the UK could be "trapped" in this arrangement for years, leaving it unable to strike its own trade deals on goods with the rest of the world.
Read more: Could technology solve the Irish border problem?
What "alternative arrangements" are there?
Alternatives to the backstop that the prime minister has said she wants to discuss with EU leaders include:
a "trusted trader" scheme to avoid physical checks on goods flowing through the border
"mutual recognition" of rules with the EU
"technological" solutions
Media captionSajid Javid: Alternative Brexit backstop "can be done"
Mr Javid told the BBC's Andrew Marr Show that an alternative arrangement to the Irish backstop "can be done" using "existing technology".
He said: "I asked Border Force months ago to advise me to look at what alternative arrangements are possible and they've shown me quite clearly you can have no hard border on the island of Ireland and you can use existing technology - that is perfectly possible.
"The only thing that's missing is a bit of goodwill on the EU side."
UK 'still safe in no-deal Brexit' - Javid
He said Attorney General Geoffrey Cox was "leading on another strand of work" - looking at whether a "hard time limit or proper exit mechanism" was possible.
However, Sabine Weyand, the deputy to European Commission chief Brexit negotiator Michel Barnier, questioned Mr Javid's comments.
She tweeted: "That would not be 'goodwill' but a dereliction of duty by public authorities in the EU that have a duty to ensure public health and safety of consumers, protect against unfair competition and enforce public policies and international agreements
