It's because the referee is so subjective and contradictory. Look at the turn of events:
After orchestrating the McLaren drivers' immunity from punishment, he is part of the commission that removes their WCC points and fines them $100 million. After doing that, he then says that if the decision was up to him, he would have removed the drivers' points as well, citing that the punishment was not harsh enough.
Spot the contradiction.
If there was any reason that Mosley should be removed from the FIA, it would be because he is a two-faced schmuck to the highest degree.