While You are right about traction the point about disadvantages of 4WD applied in 80' just like now. The Audis suffered from huge understeer (also due to the stupid engine placement) and could not turn into sharp hairpins without reversing. That's why Lancias 037 were able to keep the pace for a long time. For example in 1983 Lancias 037 were completely dominating force on asphalt events even with 1980' tyres which are nowhere close to recent ones.
If I'm not mistaken Saabs were FWD, weren't they? That's IMO the worst possible configuration for traction. Still ten years later FWD were able to fight WRC cars. But that was in time when everything was already about rules. The 1980' were nicer in the point of view that much more was allowed to be tried out.
Actually You brought a good point with turbos. In my opinion one of the reasons why RWD disappeared from rallying was that turbo engines in 1980' were quite unsuitable for RWD. There were no anti-lags, the engines had a raw power but were hard to control. Gearboxes were slow and a smooth delivery of torque to the rear wheels was something to dream about.