The sides observed a minute’s silence in honour of former Ibrox hero Fernando Ricksen whose death was announced on Wednesday.
The hosts had the better of the chances in the first half as Ryan Mayse hit the post from a Lavery centre, and Ryan Harpur failed to to make contact on the rebound.
And a Josh Kelly cross on 33 minutes almost caught out the Rangers keeper.
Home stopper Jordan Williamson had to be alert on several occasions to keep the visitors at bay. Mayse missed another opportunity before the hour mark,
Declan Carville had a header tipped around the post late on before Atakayi popped up with the winner on 87 minutes after an error by Jonny Addis.
Williamson went up for a corner deep into stoppage time and almost forced an equaliser, but it was Rangers who progressed to the fourth round.