The rights group says in a report that abuses include extrajudicial killings, enforced disappearance and torture.
A spokesman from the Nigerian army rejected the accusations, saying soldiers always operate within the law.
Boko Haram campaigns for an Islamic state and has been blamed for the deaths of hundreds of people this year.
It has carried out frequent attacks in northern and central Nigeria since 2009.
In its report, Amnesty blamed Boko Haram for widespread atrocities, including murder, burning down churches and schools, and attacks against the media.
But it said a heavy-handed response by the security forces had only fuelled further violence.
“The cycle of attack and counter-attack has been marked by unlawful violence on both sides, with devastating consequences for the human rights of those trapped in the middle,” said the group’s secretary general, Salil Shetty.
“People are living in a climate of fear and insecurity, vulnerable to attack from Boko Haram and facing human rights violations at the hands of the very state security forces which should be protecting them.”
Reports of people being shot dead by the army or beaten to death in detention were particularly common in the north-east of the country, Amnesty said.
Amnesty’s report documents what it calls an “increasing climate of fear” in which people are too scared to report crimes and journalists do not dare to cover them.
It calls on Nigeria’s government to clarify the truth about incidents linked to Boko Haram, establish accountability for abuses and bring those responsible to justice.