To my understanding, thats one of the reasons these stewards exist - to evaluate such situations and assign penalties when unfair advantage is achieved. At least on local level i know that leaving the course is when all 4 wheels are off the road (tarmac or gravel). In most such situations advantage is not achieved, as moving on roads is generally faster than moving on ploughed field or through bushes, but on some occasion it might be faster and then (when sufficient proof is presented) stewards may apply penalty. Also, rally is supposed to happen in environment without deliberately rearranging said environment - so hitting a tree is technically illegal, but when no advantage is gained, no penalty is assigned. But when barriers or cones are pushed away and advantage is achieved, then stewards may assign penalty. As possible variations are endless, quality of proofs varies, then panel of stewards is necessary to weigh each situation. Otherwise clearly defined list of penalties could be sufficient.