Baroness Flather said Pakistani and Bangladeshi families had large families to get a bigger house The peer, born in Lahore before the partition of India, said the issue did not apply to families of Indian origin.
The cross-bencher said benefit cuts could help to discourage extra births.
Baroness Flather, speaking during a debate on the government's welfare changes, said: "The minority communities in this country, particularly the Pakistanis and the Bangladeshis have a very large number of children and the attraction is the large number of benefits that follow the child.
"Nobody likes to accept that, nobody likes to talk about it because it is supposed to be very politically incorrect."
Baroness Flather
The 67-year-old said that immigrant families must stop having lots of children "as a means of improving the amount of money they receive or getting a bigger house."
Indians 'different' The former Tory peer also claimed Indian families had a different mentality to Pakistani and Bangladeshi communities in the UK."Indians have fallen into the pattern here," she told peers. "They do not have large families because they are like the Jews of old. They want their children to be educated.
"This is the other problem - there is no emphasis on education in the Pakistani and Bangladeshi families."
Baroness Flather called for a gradual reduction in benefits in order to discourage large families and suggested payments should be reduced after a couple's first two children.
She said: "I really feel that for the first two children there should be a full raft of benefits, for the third child three-quarters and for the fourth child a half."
Baroness Flather's comments were not well-received by Labour work and pensions spokesman Lord McKenzie.
Concluding the argument for the opposition, he told the Lords: "I had not expected the treatise on the family sizes of the Pakistani and Bangladeshi communities and hope I don't again."
Welfare reform minister Lord Freud, replying to the debate, did not refer to Lady Flather's comments.
The Welfare Reform Bill is the biggest shake-up of the benefits system for 60 years.
A universal payment to replace income-related work-based benefits, such as child tax credit, is planned, as are stricter rules for people losing their benefits if they refuse a job.