{"id":979,"date":"2010-10-04T17:38:51","date_gmt":"2010-10-05T01:38:51","guid":{"rendered":"http:\/\/www.codebelay.com\/blog\/?p=979"},"modified":"2019-10-29T16:49:08","modified_gmt":"2019-10-29T22:49:08","slug":"why-is-foursquare-down-3-educated-guesses","status":"publish","type":"post","link":"https:\/\/www.codebelay.com\/blog\/2010\/10\/04\/why-is-foursquare-down-3-educated-guesses\/","title":{"rendered":"Why is Foursquare Down? 3 Educated Guesses"},"content":{"rendered":"<p>Why is <a href=\"http:\/\/foursquare.com\/\">Foursquare<\/a> down?<\/p>\n<p><strong><em>Update (5 October 2010 at 5:36 pm PDT) :<\/em> The folks at Foursquare tell us <a href=\"http:\/\/blog.foursquare.com\/2010\/10\/05\/so-that-was-a-bummer\/\">why in a post-mortem<\/a>. There are autosharding issues with <a href=\"http:\/\/www.mongodb.org\/\">MongoDB<\/a>. Yup, my guesses were wrong, unless you consider MongoDB a kind of cache. \ud83d\ude09<\/strong><\/p>\n<p>I used to work for a few sites that required high scalability expertise. Now that we&#8217;re over 5 hours into the outage I&#8217;ll share some of my thoughts.<\/p>\n<p>But before I do, I&#8217;d just like to say, I really hope that it&#8217;s nothing bad and I really like the Foursquare peeps. I&#8217;m not putting out this article to harsh on anybody, but just to share some knowledge I have. Outages happen to everybody!<\/p>\n<p><a href=\"http:\/\/www.codebelay.com\/blog\/wp-content\/uploads\/2010\/10\/foursquaredown.png\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/www.codebelay.com\/blog\/wp-content\/uploads\/2010\/10\/foursquaredown-300x137.png\" alt=\"\" title=\"foursquaredown\" width=\"300\" height=\"137\" class=\"aligncenter size-medium wp-image-980\" srcset=\"https:\/\/www.codebelay.com\/blog\/wp-content\/uploads\/2010\/10\/foursquaredown-300x137.png 300w, https:\/\/www.codebelay.com\/blog\/wp-content\/uploads\/2010\/10\/foursquaredown.png 829w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p><b>Also, I do not feel that this meltdown is in any way indicative of <a href=\"http:\/\/aws.amazon.com\/\">Amazon&#8217;s EC2<\/a><\/b>. I have a <a href=\"http:\/\/www.mychamberapp.com\/\">site<\/a> that shares the same IP space and facility as Foursquare and we have had no outages today.<\/p>\n<ul>\n<li>The worst case scenario is a full scale<a href=\"http:\/\/www.wired.com\/epicenter\/2009\/01\/magnolia-suffer\/\"> Magnolia meltdown<\/a>. This is where because of a backup process that was off, they cannot restore ever from backup. Odds: unlikely.<\/li>\n<li>Someone turned off caching. I&#8217;m not sure how cache dependent the architecture is at Foursquare. If someone turned off the cache and the cache is just plain gone, then the caches have to be re-built. Rebuilding caches, depending on the time and complexity of each query can take up to 100x more time that it takes to retrieve the cache. If there&#8217;s some cached item that takes 100 seconds per user, the site will be down for a long time. They can only put a user back on foursquare at a rate of 100 per second if that&#8217;s the case, unless they can concurrently run the re-building of the cache.<\/li>\n<li>There&#8217;s an issue with a hacker who has broken through security and is wreaking havoc on Foursquare. It&#8217;s happened to the best sites, e.g. Google in the 90s, and it&#8217;s pretty tough to recover from. Sometimes you let the criminals in and do their worst while keeping the site up. Sometimes you have 0 tolerance.<\/li>\n<p>I wish Foursquare the best of luck. I am more than happy to lend a hand to their issues, if they need another pair of eyes.<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Why is Foursquare down? Update (5 October 2010 at 5:36 pm PDT) : The folks at Foursquare tell us why in a post-mortem. There are autosharding issues with MongoDB. Yup, my guesses were wrong, unless you consider MongoDB a kind of cache. \ud83d\ude09 I used to work for a few sites that required high scalability [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[288,289,5,33],"tags":[341,339,340,401],"class_list":["post-979","post","type-post","status-publish","format-standard","hentry","category-scalability-hacking","category-sysadmin","category-techbiz","category-webapps","tag-amazon-ec2","tag-foursquare","tag-scalability","tag-sysadmin"],"_links":{"self":[{"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/posts\/979","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/comments?post=979"}],"version-history":[{"count":0,"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/posts\/979\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/media?parent=979"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/categories?post=979"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.codebelay.com\/blog\/wp-json\/wp\/v2\/tags?post=979"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}