Former finance minister Joe Oliver is selling his Wychwood house for $4.5 million The ex-MP is giving up his perch in a gated community