I figured if a single AZ has an outage, let alone the entire region, I can rest easy knowing much bigger companies will have bigger problems. It will probably be newsworthy, and when customers email in, my excuse will be defensible, since I can send them links to external status pages, news articles, etc.
Whilst this was mostly true, it was still a very unpleasant experience, and my service was hanging by a thread for much of the time. I recently moved an important part of the stack from EC2 to Fargate, with two services: a single task to post jobs to a queue, and another service running many tasks to process jobs from the queue.
The incident knocked out the job posting service, which would not come back up. Had I left it to AWS to resolve automatically, my service would have been out for maybe 12 hours.
Fortunately the worker tasks were still available and waiting. I tracked down the old "job poster" code that used to run on an ec2. I sshed into an old ec2, and "deployed" the code by copying and pasting onto the server. The service came back up, although I had to edit the code directly on the ec2 to slow things down, since the ec2 had 1vCPU and an upgrade was not possible during the incident. Furthermore, Fargate workers would not scale out if they had too much work.
This was at about 2 or 3 AM my time, and was carried out whilst customers were emailing in, and cloudwatch alarms were going off all over the place. Once the service was back up, even with my unnerving hacky solution, I got a couple hours sleep.
What I've learnt:
- When the incident was first reported, I thought it would last 2 hours max. A 12 - 16 hour disruption to AWS resources is absolutely possible.
- Maybe don't use us-east-1 for future projects, but I'm not convinced there's much logic to this. Despite past issues, it's impossible to predict where an outage might occur and the affected resources, as well as spillover into other regions.
- Think of ways to make my service more portable, to other regions, even other cloud providers, but the motivation to do this will be gone by tomorrow. It's way more valuable for me to focus on customers, new features, etc, rather than bomb-proofing the service. I don't write airline or medical software. An outage of my service isn't going to kill anyone, and most users are understanding. I'll accept the hit.
> I never told them what my business does or what I'd use the money for.
This stuck out to me. I'm in the process of opening a bank account in Singapore for a newly registered company, and boy is it difficult.
Hour-long KYC interview. Details of what the business does. Anything remotely controversial could be a red flag. They want to hear about boring B2B services. They want to see evidence of customer communications or contracts, so they can see you're a legit business and not a shell company. This is very difficult for a company that hasn't started operating yet. Why would I start operating without banking? How can I show business activity before commencing operating?
By comparison, I've opened business accounts in both AU and the US without hassle.
It's likely because of the nature of the country. Singapore and the UAE (another similar country) are notoriously difficult for getting started with business banking because banks often expect you to be either well-funded (from elsewhere) or matured as a business. For a long time, they haven't ever evolved their thought to expect startups to natively grow, and even of late, they expect startups to be these well-capitalized entities with angel/VC funding. That, I presume, is because of the fact that these countries tend to be conduits for money rather than destinations themselves - they'll roll out the red carpet for you if you went with another, more "matured" objective in mind like laundering money or expanding into the country.
On the other hand, Switzerland has been surprisingly very easy to get started with quickly, even though Switzerland falls in the same category. You can easily obtain a (shitty service) Postfinance account to get started, and UBS is a breeze to work with, despite being bulge-bracket - even for new companies. Or you can also opt for a cheaper Kantonalbank, which are surprisingly very open to new businesses.
> The meta-analysis of 33 studies, the first of its kind, looked at the relationship between sensitivity and common mental health problems such as depression and anxiety. Researchers found there was a significant, positive relationship between the two
> In the study, sensitivity was defined as a personality trait that reflects people's capacity to perceive and process environmental stimuli such as bright lights, subtle changes in the environment and other peoples' moods.
> Around 31% of the general population are considered highly sensitive
I recently deployed a few static websites on Cloudflare. I use AWS as a registrar, and I had to use Pages since Workers don't support "Custom domains outside Cloudflare zones" [1]. There's no way I can transfer the domain since I have subdomains tightly integrated with AWS services.
If anyone from Cloudflare sees this, are there plans to have Workers support external custom domains, or is this a fundamental limitation of Workers?
