The president-elect, who will be the nation's first left-wing president since Francois Mitterrand left office in 1995, suggested there was a sense of relief in many European countries because of his win.
With almost all of the votes counted, Hollande was leading with 51.6% to Sarkozy's 48.4%, the nation's Interior Ministry said. Voter turnout was reported at more than 80%.
Exit polls said Hollande won 51.9% of the vote, France 2 television reported.
Sarkozy had fought to keep his job amid a wave of discontent over his inability to rein in unemployment. He defended his economic record despite low growth and unemployment at about 10%, saying the impact of Europe's debt crisis could have been far worse.
France is a key player in plans to lead the eurozone out of its debt crisis, making the election vital to the region.
Sarkozy's defeat marks the latest and most significant of at least half a dozen European leaders swept from office during the eurozone economic crisis, including the Greek and Italian prime ministers.
Sarkozy has been president since 2007.
France's vote came the same day as the Greek parliamentary election. Exit polls in that country indicated Greece's ruling coalition appeared headed for steep losses, while parties on the far left and far right were poised to make significant gains amid anger over austerity measures.
