Early this morning, Imagizer Cloud east coast zone underwent a large scale outage. The majority of image requests to our east coast cluster failed with a timeout.
The outage began at 5:50 am PST and ended at 8:10 am PST, affecting all east-coast regional customers.
What and why it happened:
Although there were many factors contributed to the outage, the trigger cause was due to a large increase in resource-intensive requests. The effect of the query overload was a rapid increase in server creation via autoscaling.
Unfortunately, due to an artificial restriction the cluster did not have enough time to scale to absorb all requests coming from HomeSnap.
We have also failed to react in time to mitigate the outage.
Immediate action taken:
We take all outages seriously and always work hard on improving our methods to provide safest most stable environment for our clients. With actions taken above, we expect a significant improvement in stability across all of our services with minimal if any disruption in the future.