Fear is instituted in this part of the story because it starts to show where everything starts to go wrong. He is sentenced to death which is his ultimate fear. He was wrongly judged for the government only wanting his land. He gets so overwhelmed by everything he faints from fear.
The pit shows fear by giving him two options. He could either wait on the untimely death of the pendulum or have the rats eat him alive and fall into the pit. With death being his ultimate fear this institutes all of it.
The pendulum represent his fear by almost making him lose his mind. With every sweep of the pendulum shows just how cloudy his judgement of death and hope is. With every time his hope gets taken away puts more fear into each time.