Jacob Rees-Mogg is sticking by plans to make MPs return to Westminster next month, following five weeks of semi-virtual working.
The Commons leader wants MPs to come in after 2 June, following the Whitsun recess, to set an example to the rest of the country.
He argued working via video link during the coronavirus pandemic had made it harder for ministers to pass new laws.
But opposition MPs said returning would be irresponsible and put them at risk.
Responding to an urgent question on Wednesday, Mr Rees-Mogg confirmed plans to bring an end to a “hybrid” working model which has been in place since mid-April.
But he said MPs’ staff would continue to be encouraged to work from home, and that ministers were working with the House authorities to see how MPs with health conditions could “continue to contribute”.
Under the “hybrid” arrangements, a maximum of 50 MPs are allowed in the Commons chamber, with up to 120 taking part via video conferencing technology.
A system has also been developed to allow MPs to vote remotely on their computers rather than voting in person as they would normally.
Mr Rees-Mogg argued this has cut the time available for debating legislation by around two thirds, and prevented “proper scrutiny” of the government.
He said MPs would continue to follow social distancing rules on their return, including by keeping six feet apart when casting their votes in the chamber.
“We will not be returning to the crowded, bustling chamber of old,” he said, adding MPs would be returning to a “safe working environment”.
However Lib Dem MP Alastair Carmichael, who asked the urgent question on post-June arrangements, said returning would put MPs and staff “at risk”.
He added that the last five weeks had shown the job of an MP “can be done from behind a screen”.
The MP, who represents Orkney and Shetland, said travelling to Westminster would force MPs to undertake “non-essential journeys” and “stay in second homes”.
“If ever there were a case of do as I say, not as I do, this is it,” he added.
Labour’s shadow Commons leader Valerie Vaz said any return should be “phased”, so as “not to overpower the NHS, or overpower House staff”.
“This is about Parliament being a good model employer,” she added.
SNP MP Tommy Sheppard said MPs were being forced to “make a choice” between representing their constituents and putting their own health at risk.
Older MPs or those suffering from ill health would be “disproportionately affected” by the plan to return, he said, calling it “cavalier and downright dangerous”.
He added that returning to Westminster would have a greater impact on MPs living furthest away from London.
Speaking on Tuesday, Mr Rees-Mogg had argued opposition MPs “like having a hybrid Parliament” because they were “willing to sacrifice a degree of scrutiny to stymie the government’s programme.”
‘Best option we have’
Speaking on his ConservativeHome podcast, he also said it would be “unreasonable” for pupils to start returning to schools in England – which could happen from 1 June – while MPs stayed away from Parliament.
But Conservative MP, Karen Bradley, who chairs the Commons Procedure Committee, said MPs should be allowed to continue working remotely until it is safe to return.
She told the BBC: “Hybrid proceedings are not ideal.”
“But in my view some form of continued virtual participation is the best option we have, until it is safe to use the Chamber as before and all members are able to return.”
The plan to return to physical working was also criticised by two unions representing parliamentary staff.
Garry Graham, deputy general secretary of Prospect, said the government had chosen to “ignore” concerns raised by those working in Westminster.
Amy Leversidge, assistant general secretary of the FDA union which represents Commons clerks, said the move was “incredibly disappointing”.
“This needlessly puts staff and parliamentarians at risk,” she added.