Awesome details. I did wonder exactly how memcached handled eviction. I once did a game day where we cold-restarted one of 6 sharded memcached servers. We expected slow responses for a while and was prepared for anything worse. It all went fine, and what we discovered was that we were way overprovisioned as even hours (and even next day) later we were only using 1/4 of the memory from before the restart. We restarted all of them one by one. The problem may have been a very long uptime combined with old entries that had long/no TTLs.