I'm a plastics engineer. There are several reasons why you would not want to use Legos. One is creep, which was already mentioned. Force time = distorted plastic. Like if you have an old linoleum kitchen floor, move the refrigerator and you will see the indentations where it has been the last 20 years.

Two, the structural integrity of plastic is not sufficient for building construction. If you had a Lego exterior, where the guts supporting the house were steel or something, that might work.

But another problem is the UV degradation of ABS. Over time, the material will not just become discolored, it will be destroyed by the sun, breaking down into smaller and smaller molecules, slowly losing its mechanical strength and turning into dust.

Vinyl siding may come to mind as a common outdoor plastic. It is full of UV stabilizers which prevents UV radiation from damaging the plastic. Over time these stabilizers are used up, and your PVC chains/molecules start to degrade. If you touch an old vinyl fence, it will leave white dust on your hand. That is the polymer chain breaking into pieces.