So the handle is nice and tight when you pull it up huh?
The only other thing I can think of is that the rear brakes might just need done. How long has it been since the rear brakes have been serviced? Maybe the rotors are glazed etc. causing the pads to just not have enough good surface to hold to. Since the car is always going forward and the brakes are bedded in that way, it can hold, but going backwards it simply cannot hold.