Abe spoke as the grieving father of self-employed security contractor Haruna Yukawa said he had gone "totally blank" since learning of his son's execution, announced in a video posted online by the militants late Saturday.
Abe, speaking to public broadcaster NHK, said he was "speechless" at the "unbearable pain and sorrow" that Yukawa's family were feeling.
"Such act of terrorism is outrageous and impermissible, it causes me nothing but strong indignation," he said.
"I condemn it strongly and resolutely," he said as he demanded the immediate release of Yukawa's fellow captive, freelance journalist Kenji Goto.
Shoichi Yukawa told of the horror of realising the threats to kill his son had been carried out.
"I thought 'Ah, this finally happened' and was filled with regret," he said in a statement to media.
"I went totally blank, I was only sorry... I had no words to say," he said. "In my mind I wish very much that this wasn't true."
US President Barack Obama led the condemnation of the Islamic State group's "brutal murder" of Yukawa. British Prime Minister David Cameron decried the movement's "murderous barbarity", and French President Francois Hollande labelled it a "barbaric assassination".
Japan would continue analysing the images to fully confirm the authenticity of the video, said Abe, but the "credibility" of the footage was "high".
The recording, which lasts nearly three minutes, shows a still image of Goto holding what appears to be a photograph of Yukawa's slain body.
It was posted with an audio recording in which a man claiming to be Goto blames Abe for his fellow captive's death after he failed to pay a $200 million ransom.
The voice also reveals a new demand from the militants for the release of Sajida al-Rishawi, a female militant now held in Jordan, in exchange for Goto's life.
Abe said Japan was in contact with Jordan but declined to comment on whether Tokyo would ask Amman to release the prisoner, who is being held in connection with multiple murders.
The appearance of the video was met with initial skepticism, partly because it was not posted on an official IS channel and does not bear their black and white flag. The execution of Yukawa is also not shown.
Goto's mother, Junko Ishido, told reporters Sunday that she was left with little hope after seeing the picture of her son, in which he "looked very tense".
Mother's appeal for mercy
The hostage saga began on Tuesday when the militants released a video in which they demanded a $200 million ransom within 72 hours in exchange for the lives of the two hostages.
On Friday Goto's mother Ishido launched an emotional appeal begging for mercy for her son.
"I say to you people of the Islamic State, Kenji is not your enemy. Please release him," she said.
As the deadline passed Friday, there was an agonising silence from the jihadists until this new video was posted late Saturday.
The Islamic State group, which rules large swathes of Iraq and Syria under a strict form of Islamic law, has murdered five Western hostages since August last year, but this is the first time it has threatened Japanese captives.
Japanese officials have repeatedly said they are trying to make contact with the militants through various channels.
Abe advisor Yosuke Isozaki reportedly said there had been some "indirect" communication with the militants, but "nothing direct".
Tokyo has little diplomatic leverage in the Middle East, but local media had previously said Abe may try to use his close relationship with Turkey's President Recep Tayyip Erdogan to rescue the hostages.
The Islamists linked their earlier $200 million ransom demand to the amount Abe said he would earmark to help countries dealing with the influx of refugees fleeing fighting between IS militants and regular forces.
Abe was defiant Sunday, saying Tokyo was not intimidated by the murder.
"We will never give in to terrorism, and actively contribute to the peace and stability of the world together with the international community. We are not wavering at all on this policy," he said on NHK.
The Japanese media has rallied around Goto, a respected and experienced war reporter whose work has appeared on domestic television channels.
In video footage he filmed around the time he entered Syria, he explains that he is aware of the risks.
"Whatever happens, I am the one who is responsible," he says.
"I am asking you, Japanese people, do not place responsibility on the people of Syria. Please. I am sure I will come back alive though."