Gaza medics said four Palestinians were killed as tens of thousands of Palestinians rallied to mark the first anniversary of the 'Great March of Return' demonstrations.
Three were 17-year-olds shot dead by Israeli troops while protesting, they said, adding another person was killed at an overnight protest hours before the main rally.
Israeli forces massed on the other side of the frontier fired teargas and said some of the demonstrators threw rocks, grenades and burning tyres towards them.
But Saturday's rally was smaller than expected, despite concerns that the anniversary would see a major escalation.
Hundreds of Palestinian men, some from Hamas, were deployed in bright orange vests to deter people from going near the fence.
The Israeli military said there had been around 40,000 protesters, some hurling grenades and explosives.
Lieutenant-Colonel Jonathan Conricus, an Israeli military spokesman, said the day had "lots of rioters but significantly less violence", adding that it showed Hamas was able to stop violence when it wanted to.
Tensions mounted along the border this week ahead of the commemoration of the protests.
On Monday more violence erupted when a Palestinian rocket attack from Gaza wounded seven Israelis in a village north of Tel Aviv.
In response, Israel launched a wave of air strikes and increased its forces at the border.
This year around 200 Palestinians and one Israeli soldier have been killed.
According to Gaza's health ministry around 7,000 Palestinians have been shot and wounded.
UN investigators said last week that Israeli forces may be guilty of war crimes for using excessive force.
Israel says its troops have no choice because they are trying to stop militants breaching the fence and attacking Israeli communities nearby. Palestinians have also launched incendiary balloons and kites into Israel.