EDIT: There's been some confusion. When I say "Liberal" I mean Liberal as in free or open, or classical liberalism. I don't mean Liberal as in "Something that might be supported by members of the Democratic Party of the United States" or "Something that might be supported by progressives", though there is certainly overlap between these.

"Everything is worth what its purchaser will pay for it." Publilius Syrus

Imagine you're a programmer living in India, living in Mumbai. In Mumbai you can earn 1000 USD a month after tax and basic living expenses (excluding rent). In San Francisco, you can earn 5000 USD a month after tax and basic living expenses. The question I ask, how much more are you willing to pay for your rental in San Francisco then you would pay in Mumbai? The answer would be: a number close to but not exceeding 4000 USD. So long as your rent is under 4000 USD, the move will improve your lot in life. Of course it's troublesome to move halfway around the world, so you might not trouble to move unless the gain passes a certain amount, let's say for this example 1000 USD. That means the rent that a particular programmer in India is willing to pay to live in San Francisco is 3000 USD. That means, without any restrictions, programmers will keep leaving Mumbai and moving to San Francisco until the difference in price between the two to rent an equivalent apartment in the two cities is 3000 USD.

Speaking more generally, in a world with stark differences in wealth between different countries and cities, people will move from the poorer regions to wealthier ones, and the people moving from those poorer regions will be willing to pay high housing costs in the destination in order to move. This means existing residents will have to be willing to pay the same rents as those new residents in order to be able to continue to live in their city. This is how a teacher in San Francisco ends up driving for uber or living in their cars, they have to compete for their housing "slot" with a programmer from India who's willing to pay 3000 in rent to access the higher salary he couldn't in his home country.

I'd like to pre-empt a few arguments:
Q: You're only looking at the demand side, what about supply? You can just build more housing to lower prices.
A: This certainly helps, but the cost of migrating is significantly lower(and faster) then the cost of building housing in a wealthy city. The cost to build a single apartment might be 400,000 USD. The cost to immigrate is only 10,000 USD. Immigration will always outpace housing construction.
Q: But if large numbers of people immigrate, won't jobs run out?
A: Immigrants don't steal jobs. In fact, the opposite is likely the case, immigrants tend to start more businesses on average and will likely make their destination city wealthier, making it even more attractive to immigrate to.
Q: Why wasn't there high rents in the mid 20th century when the world was even more unequal?
A: There are a few reasons: 1. immigration restrictions were much higher back then. 2. It was much more expensive to immigrate (EG flying even short distances was the preserve of the super rich), and the poorest regions were so poor that few could afford even the cost of migrating. Ironically as the developing world develops it becomes easier for people to afford to migrate. 3. Building housing was unusually easy due to the invention of the motor car opening up vast amounts of land around wealthy cities for the construction of suburbs. Today, almost all of those suburbs have been built out, and building housing quickly and easily in this way is no longer possible.

Q: You're oversimplifying, there's many more reasons a person might migrate then just money.
A: This is true. But you can add all those factors together to calculate how much more they value living in a distant location then their current one. EG, let's say I work in Lieges, earn 2000 Euro there, and I'm offered a job in Lisbon that pays 3000 Euro. The combination of better weather, nicer architecture and better public transit and other infrastructure might mean that I'm willing to pay 1500 more euro in rent in Lisbon then I am in Liege, even though the job only pays 1000 euro more. If there are enough similar people like me, the rent in Lisbon will end up becoming 1500 euro higher then in Liege.

I'd like to be proven wrong, as otherwise I think immigration is very much a good thing, not only that, I've migrated several times myself, and my parents met via immigration.