A joint report by the Institute for Democracy and Economic Affairs (IDEAS) and the Centre for Public Policy Studies (CPPS)
cited a range of problems including widespread concerns over the integrity of the electoral roll.
It added that while voting and handling of ballots generally proceeded smoothly, issues including faulty indelible ink, partisan use of government machinery and pro-government media bias left a blot on the election.
"Having conducted an independent and impartial observation of the elections, IDEAS and CPPS conclude that (the election) was only partially free and not fair," its report said.
Opposition leader Anwar Ibrahim has called for a sustained "fierce" campaign for electoral reform, alleging Sunday's elections were "stolen" by Prime Minister Najib Razak's government.
Najib, who was sworn in Monday after his Barisan Nasional (National Front) coalition retained its 56-year hold on power, has dismissed the allegation.
But the watchdogs' report cited "serious flaws when assessing the complete freedom and the fairness" of the election.
They included a lack of transparency in campaign spending, questions over the Election Commission's supposed independence from Barisan, and inequities in the delineation of constituencies.
In this regard Malaysian police have declared illegal a rally planned for Wednesday night by opposition leader Anwar Ibrahim to protest against election results that he says were "stolen" through massive fraud.
"The organiser must comply with the Peaceful Assembly Act 2012. Marching without a permit, aimed at arousing anger, is wrong according to the law," national police chief Ismail Omar said, according to state news agency Bernama.
Ismail added that participants would be arrested, Bernama said.
His comments appeared to set the stage for a confrontation if the opposition went ahead with the rally, set to start around 8:00 pm (1200 GMT).