The vote was triggered by a corruption scandal involving Rajoy's Partido Popular.
There were scenes of jubilation in the Spanish parliament but Spain's political future looks uncertain.
The Socialist Party only holds 84 seats in the Congress of Deputies so Sanchez will have to rely on securing the support of allies such as left-wing Podemos in order to enact legislation.
The King has been informed of the appointment of the new prime minister and Sanchez is expected to name a cabinet early next week. Then it could be a question of just how long the government can last in a divided parliament.