It was Carson's first address to the staff at HUD. He was confirmed by the U.S. Senate last week.
By way of introduction, Carson shared anecdotes from his past career as a neurosurgeon and praised immigrants who worked long hours to build a better life for their children.
"There were other immigrants who came here in the bottom of slave ships, worked even longer, even harder for less," said Carson, who is African-American.
"But they too had a dream that one day their sons, daughters, grandsons, granddaughters, great-grandsons, great-granddaughters might pursue prosperity and happiness in this land," he said.
Enslaved Africans did not voluntarily come to the United States and were denied freedom for hundreds of years.
"This is as offensive a remark as it gets," said Steven Goldstein, executive director of the Anne Frank Center for Mutual Respect.
The remarks sparked outrage on Twitter, including from the actor Samuel L. Jackson. The National Association for the Advancement of Colored People (NAACP) also criticized Carson.
A HUD spokesman later called the tempest "the most cynical interpretation of the secretary's remarks to an army of welcoming HUD employees. No one honestly believes he equates voluntary immigration with involuntary servitude."
Carson was well received by the hundreds of HUD employees in the room and got a standing ovation at the close of his remarks.
As housing secretary, Carson is in a position to play a leading role in reviving poor neighborhoods, as Republican President Donald Trump has promised.
Carson occasionally stumbled as a Republican presidential candidate, such as when he said he did not believe a Muslim should be president.
After dropping out of the race, Carson threw his support to Trump, who named him housing secretary after winning the November election.