Based on their presentation and design choices (and a tiny bit of experience) it looks to me like the limiting factor is actually bandwidth and latency not flops or electricity or heating or space or even the cost of the transistors themselves. A ton of flops doesn't help if you can't feed them data to process.

The reason they gave for doing things like "knocking out the walls of cabinets" was bandwidth, not space.

It turns out that the highest bandwidth solutions are also quite compact. One giant piece of silicon instead of many smaller pieces of silicon is great because the different parts of the giant piece of silicon can talk to eachother very quickly with very high bandwidth. Short wires are better than long wires, because you have less noise so you can fit in more signal (also less latency).

It's also just not clear to me how you imagine making things bigger would make them cheaper.

Locality has everything to do with performance. It's why your L1 cache is faster than your L2 cache.

I'm not a hardware person at all, but my understanding is that distributed computing systems are hard to scale due to latency, and a huge factor in latency at this scale is physical distance.

This was also mentioned in the AI Day presentation, and listed as a major factor in the design of Dojo.