There's plenty of supply - I don't doubt even the state of CA has more actual housing available than strictly necessary for all those currently living there, but it's natural for people to want to live in areas that are a) close to family and friends b) provide access to employment and c) have necessary infrastructure/ services/retail options readily available. All those factors contribute to making densely populated areas desirable - unless you can provide transportation infrastructure that enables millions of people to efficiently transit long distances - and just about the only countries that have achieved that are those with very high population densities anyway!
But there are definitely other factors that exacerbate cost-of-housing issues - income disparities that are further magnified by the degree to which banks are prepared to lend money being one that's rarely discussed. If someone earning $200k a year is able to afford to spend up to 5 times more on a house - and by extension the land it's on - than someone earning half that (and quite possibly 10 times more than someone earning 50k), then how can that not have a serious impact on housing affordability, which trickles down to rental affordability, as enough rent needs to be charged to help offset mortgage costs for landlords.