Rubber is chemicaly active and deteriorating constantly due to oxidation, which gradually renders it brittle and subject to cracking. A rubber building would fail catastrophically within a decade. But they do use rubber mountings for modern buildings in areas prone to earthquake. The rubber buffers allow the building to sway in the 'quake and remain standing.
There are some houses that have been built using rubber. An example of such a house is known as the 'De Eenvoud' in Almere, and some of the disadvantages of the rubber structural is that a house cannot have a first floor.