to the point that I had to pull an extra circuit... but tri phase so good to go even if I would like to go bigger.
I've limited power consumption to what I consider the optimum, each card will draw ~275 Watts (you can very nicely configure this on a per-card basis). The server itself also uses some for the motherboard, the whole rig is powered from 4 1600W supplies, the gpus are divided 5/5/4 and the mother board is connected to its own supply. It's a bit close to the edge for the supplies that have five 3090's on them but so far it held up quite well, even with higher ambient temps.
Interesting tidbit: at 4 lanes/card throughput is barely impacted, 1 or 2 is definitely too low. 8 would be great but the CPUs don't have that many lanes.
I also have a threadripper which should be able to handle that much RAM but at current RAM prices that's not interesting (that server I could populate with RAM that I still had that fit that board, and some more I bought from a refurbisher).
What pcie version are you running? Normally I would not mention one of these, but you have already invested in all the cards, and it could free up some space if any of your lanes being used now are 3.0.
If you can afford the 16 (pcie 3) lanes, you could get a PLX ("PCIe Gen3 PLX Packet switch X16 - x8x8x8x8" on ebay for like $300) and get 4 of your cards up to x8.
All are PCIe 3.0, I wasn't aware of those switches at all, in spite of buying my risers and cables from that source! Unfortunately all of the slots on the board are x8, there are no x16 slots at all.
So that switch would probably work but I wonder how big the benefit would be: you will probably see effectively an x4 -> (x4 / x8) -> (x8 / x8) -> (x8 / x8) -> (x8 / x4) -> x4 pipeline, and then on to the next set of four boards.
It might run faster on account of the three passes that are are double the speed they are right now as long as the CPU does not need to talk to those cards and all transfers are between layers on adjacent cards (very likely), and with even more luck (due to timing and lack of overlap) it might run the two x4 passes at approaching x8 speeds as well. And then of course you need to do this a couple of times because four cards isn't enough, so you'd need four of those switches.
I have not tried having a single card with fewer lanes in the pipeline but that should be an easy test to see what the effect on throughput of such a constriction would be.
But now you have me wondering to what extent I could bundle 2 x8 into an x16 slot and then to use four of these cards inserted into a fifth! That would be an absolutely unholy assembly but it has the advantage that you will need far fewer risers, just one x16 to x8/x8 run in reverse (which I have no idea if that's even possible but I see no reason right away why it would not work unless there are more driver chips in between the slots and the CPUs, which may be the case for some of the farthest slots).
PCIe is quite amazing in terms of the topology tricks that you can pull off with it, and c-payne's stuff is extremely high quality.
If you end up trying it please share your findings!
I've basically been putting this kind of gear in my cart, and then deciding I dont want to manage more than the 2 3090s, 4090 and a5000 I have now, then I take the PLX out of my cart.
Seeing you have the cards already it could be a good fit!
Yes, it could be. Unfortunately I'm a bit distracted by both paid work and some more urgent stuff but eventually I will get back to it. By then this whole rig might be hopelessly outdated but we've done some fun experiments with it and have kept our confidential data in-house which was the thing that mattered to me.
Yes, the privacy is amazing, and there's no rate limiting so you can be as productive as you want. There's also tons of learnings in this exercise. I have just 2x 3090's and I've learnt so much about pcie and hardware that just makes the creative process that more fun.
The next iteration of these tools will likely be more efficient so we should be able to run larger models at a lower cost. For now though, we'll run nvidia-smi and keep an eye on those power figures :)
You can tune that power down to what gives you the best tokencount per joule, which I think is a very important metric by which to optimize these systems and by which you can compare them as well.
I have a hard time understanding all of these companies that toss their NDA's and client confidentiality into the wind and feed newfangled AI companies their corporate secrets with abandon. You'd think there would be a more prudent approach to this.
What is your estimate for when memory prices will decrease?
I agree that we've seen similar fluctuations in the past and the price of compute trends down in the long-term. This could be a bubble, which it likely is, in which case prices should return to baseline eventually. The political climate is extremely challenging at this time though so things could take longer to stabilize. Do you think we're in this ride for months or years?
I can’t be more clear: specificity around predicting the future is close to impossible. There are 9 figure bets on both sides of the RAM issue, and strategic national concerns. I say that prices will go down at some point in the future for reasons highlighted already, but I have no clue when. Keep in mind what I myself have said about human ability to predict the future. You would be a fool to believe anyone’s specific estimates.
Maybe the AI money train stops after Christmas. The entire economy is fucked, but RAM is cheap.
Maybe we unlock AGI and the price sky rockets further before factories can get built.
There are just too many variables.
The real test is if someone had seen this coming, they would have made massive absurd investment returns just by buying up stock and storing it for a few months. Anyone who didn’t take advantage of that opportunity has proved that they had no real confidence in their ability to predict the future price of RAM. RAM inventory might have been one of the highest return investments possible this year. Where are all the RAM whales in Lambos who saw this coming?
As a corollary: we can say that unless you have some skin in the game and have invested a significant amount of your wealth in RAM chips, then you don’t know which way the price is going or when.
Extending that even further: people complaining about RAM prices being so high, and moaning that they bought less RAM because of it are actually signaling through action that they think that prices will go down or have leveled off. Anyone who believes that sticks of DDR5 RAM will continue the trend should be cleaning out Amazon, Best Buy and Newegg since the price will never be lower than today.
The distinct lack of serious people saying “I told ya so” with receipts, combined with the lack of people hoarding RAM to sell later is a good indirect signal that no one knows what is happening in the near term.
I can’t be more clear: specificity around predicting the future is close to impossible.
And I can't be more clear: a single entity bought more than 70% of the wafer production for the next year. That's across all types of memory modules. That will increase prices.
people complaining about RAM prices being so high, and moaning that they bought less RAM
because of it are actually signaling through action that they think that prices will go
down or have leveled off
No, no they're not. They're saying nothing about what they think future prices will be.
I recently learned that it's not just the switch, but also the gasket, so the switch plate material, the foam layers and even the keycap itself. I built two different split keyboards recently with the same simple Kailh box red v2 switch and they sound and feel completely different just because of the thickness of the switch plate and the type of keycaps I use.(check this for example https://www.youtube.com/shorts/HIldaxljpzc )
You can check if you find the switches colors here(it looks like an Akko purple pro, but not quite) https://keeb-finder.com/switches
Whereas rtings has a filtering list that also has sound profiles in the review pages.
You can get really cheap boards on taobao, for sofle, lily58, corne(all 3 are open source/open hardware) keyboards. You can of course also get prebuilt ones with or without switches for cheap if you want to. But in today's world, if you have tools and access to a 3d printer you can get a board for a few bucks some components and finish the whole thing with good switches and keycaps for 20-30 dollars.
It's a fun experience, and a nice reason to play around with SMD soldering techniques. I had my daughter (4 years) solder the hotswap sockets.
Worth checking out the miryoku layout, which is optimized for small keyboards, where I recently added sensor bindings for ec11 encoders[1].
You can get aula f75 for cheap, arround 50$, there are plenty of sound tests on youtube as it's very popular.
I got that version and I am happy, but if I was to buy a new one I would get the full size f108 because it's important for me to have distance between arrow keys and other keys. And tbh I would just get an apple keyboard or something similarly slim because it's more confortable for me. However for thicc (mechanical switch) keyboards, aula f75 has great specs and sound at a very good price.
Disabling offline accounts is one thing but scanning and labeling your files to profile users is a whole other can of worms. This trajectory leads to zero privacy for the user and I feel like switching to Linux/Mac will be the only option sadly.
reply