Mr Abadi was there to announce the city's full "liberation", his office said in a statement.
Iraqi forces, backed by US-led air strikes, have been battling to retake Mosul since 17 October last year.
Islamic State militants seized it in June 2014 before taking much of Iraq's Sunni Arab heartland and proclaiming a "caliphate" straddling Iraq and Syria.
Kurdish Peshmerga fighters, Sunni Arab tribesmen and Shia militiamen have also been involved in the gruelling battle.
The Iraqi prime minister arrived to "congratulate the armed forces and the Iraqi people" on the final defeat of IS in Mosul on Sunday, the statement said.
He met commanders in the city but has not yet given a speech formally declaring triumph.
Iraqi forces have been battling the remaining pockets of jihadists desperately holding out in a tiny area near the Old City.
Airstrikes and exchanges of gunfire could still be heard on Sunday, and plumes of smoke rose into the sky.
Mr Abadi's spokesman, Saad al-Hadithi, said victory would not be formally declared until the few remaining militants were cleared from Mosul, Reuters news agency reports.
Earlier 30 IS fighters were killed as they attempted to escape the advance of the Iraqi forces by throwing themselves in the River Tigris, state media said.
Media captionBBC correspondent Jonathan Beale: 'The is still a lot of misery in Mosul'
The government announced the "liberation" of eastern Mosul in January, but the west of the city, with its narrow, winding streets, has presented a more difficult challenge.
Some 900,000 people have been displaced from the city since 2014 - about half the the pre-war population - aid organisations say.
As "victory" was proclaimed in Mosul, Save the Children warned of the psychological impact on the children "haunted by memories of extreme violence, or of loved ones killed in front of them".
"For children and their families to process these horrors and rebuild their lives, psychological support will be absolutely crucial", said Ana Locsin, Save the Children's Iraq country director.
"But right now the world is providing next to no funding for mental health."
IS has lost large parts of the territory it once controlled in Iraq since the regional offensive began.
But the fall of Mosul does not mean the end of IS in the country. It still has territory elsewhere - such as Tal Afar and three towns in the western province of Anbar - and is able to carry out bombings in government-held areas.
French President Emmanuel Macron was among the first to hail the victory of Iraqi forces in Mosul on Sunday, praising the fighters - including French troops in the coalition - who had made it possible.
By Alan Johnston, BBC World Service Middle East editor
The army has avenged its humiliating defeat by the militants in Mosul three years ago. But victory has come at a terrible cost.
Thousands of civilians have been killed. Much of Mosul has been destroyed.
And the dangers posed by the Islamic State group are not at end.
The militants will go underground and they may well look to make hit-and-run attacks in classic guerrilla style - in Mosul and many other places.
