Title: 4.4 - Case Study: Joyent
Course: Introduction to Mission Critical > Section: Introduction to Mission Critical

Click here for a downloadable version of this document: Case Study: Joyent

Case Study

Joyent Data Center

Please read the following case study and related, external links and then answer the follow-up questions.

About Joyent
Joyent is a San Francisco based company that provides cloud hosting with a IaaS (infrastructure as a service) business model. Their clients include many web application and gaming companies including the organization behind the popular "window shopping" smartphone app WANELO. As of June 3, 2014, Joyent advertises a six nines uptime (services available 99.9999% of the time) on their personal website and, as a part of contract agreements, the company offers compensation in the result of downtime. Since uptimes and quality of service is so important to Joyent and the businesses they serve, a mission critical mindset and high level critical thinking and decision making skills are required of anyone working at the Joyent data centers.

Case
On May 27, 2014, Joyent experienced major system downtimes. A technician was remotely installing updates to a set of new servers in a Joyent data center. These servers were only a small portion of the infrastructure on site. However, when the technician went to reboot the newly updated servers, he accidentally sent the reboot command to the entire site; effectively putting the entire US-EAST-1 region (a large group of servers located in the eastern United States) out of commission. As the servers rebooted and propagated, customers experienced downtime that lasted anywhere between twenty minutes and two and a half hours. With Joyent's advertised 6 nines uptime, a customer only expects a maximum of 31.5 seconds of down time annually. In some cases, the amount of downtime experienced was over 170 times what would have been expected. The effects of this simple mistake were wide ranging. Since these clients depend on having their cloud infrastructure operational to provide services for their customers, when the servers went down, they were unable to do business. This potentially resulted in serious financial losses and heavy customer dissatisfaction for Joyent's clients. However, some of those losses will be recuperated since contract agreements with Joyent include penalties for downtime.

EXTERNAL: Register UK: Joyent Cloud Down
EXTERNAL: Joyent Blog: Postmortem