Activists on the ground said two buses arrived in Homs early morning to begin the first of the evacuations.
Injured people who have been trapped in the Old City and surrounding areas for nearly two years under a tight government siege are to be the first to leave.
Under the deal negotiated with assistance of Iran's ambassador to Damascus, the group will be taken to an opposition-held area in the north of Homs province, according to a rebel negotiator.
Fighters will be allowed to withdraw with light weapons, and one rocket launcher will be permitted on every bus used for the evacuation.
The deal will be guaranteed by the United Nations and Iran, according to a copy of the text seen by AFP.
Homs governor Talal al-Barazi told AFP Wednesday that last-minute preparations were still being made but the operation "will take place today, God willing".
The deal was reached as part of an exchange for an unknown number of hostages being held by rebels in the northern city of Aleppo.
And under the agreement, opposition fighters will also allow aid into two Shiite majority towns in the same province, Nubol and Zahraa, which are under rebel siege.
On Tuesday, UN workers cleared the roads to be used in the evacuation of landmines in preparation for the withdrawal.
Once it is complete, the evacuated areas will be turned over to the government, which will then have control of all but one major area of Homs, once dubbed the "capital of the revolution".
The district of Waer will remain under opposition control for now, but negotiations for a similar deal to that to be implemented in the Old City are already underway.
The Old City of Homs and surrounding rebel-held areas have been under a tight government siege for nearly two years.
Earlier this year, around 1,400 people were evacuated from the districts under a UN-Red Crescent operation.
But a group of fighters and civilians, including those with injuries who were unable to reach evacuation points, remained behind.
They faced increasingly tough conditions, with little food or medicine and the start of a government offensive to try to retake the areas.
More than 150,000 people have been killed in Syria since the beginning of the uprising in March 2011.
AFP from BEIRUT