I am here..
Blog
-
Why is Foursquare Down? 3 Educated Guesses
Why is Foursquare down?
Update (5 October 2010 at 5:36 pm PDT) : The folks at Foursquare tell us why in a post-mortem. There are autosharding issues with MongoDB. Yup, my guesses were wrong, unless you consider MongoDB a kind of cache. 😉
I used to work for a few sites that required high scalability expertise. Now that we’re over 5 hours into the outage I’ll share some of my thoughts.
But before I do, I’d just like to say, I really hope that it’s nothing bad and I really like the Foursquare peeps. I’m not putting out this article to harsh on anybody, but just to share some knowledge I have. Outages happen to everybody!
Also, I do not feel that this meltdown is in any way indicative of Amazon’s EC2. I have a site that shares the same IP space and facility as Foursquare and we have had no outages today.
- The worst case scenario is a full scale Magnolia meltdown. This is where because of a backup process that was off, they cannot restore ever from backup. Odds: unlikely.
- Someone turned off caching. I’m not sure how cache dependent the architecture is at Foursquare. If someone turned off the cache and the cache is just plain gone, then the caches have to be re-built. Rebuilding caches, depending on the time and complexity of each query can take up to 100x more time that it takes to retrieve the cache. If there’s some cached item that takes 100 seconds per user, the site will be down for a long time. They can only put a user back on foursquare at a rate of 100 per second if that’s the case, unless they can concurrently run the re-building of the cache.
- There’s an issue with a hacker who has broken through security and is wreaking havoc on Foursquare. It’s happened to the best sites, e.g. Google in the 90s, and it’s pretty tough to recover from. Sometimes you let the criminals in and do their worst while keeping the site up. Sometimes you have 0 tolerance.
I wish Foursquare the best of luck. I am more than happy to lend a hand to their issues, if they need another pair of eyes.
-
Dumping Monaco for 9×15 Font
I’ve been using Monaco as my default font for coding for about 10 years now. I used to be a hardcore Courier fan back in college only because our printers only printed out in Courier. I wanted my emails and printouts to match.
I ran into an article on the 9×15 font being more readable and clean, and decided to use it. Here’s the comparison below with Monaco first,
and then 9×15:
-
Lots of Phones to Test At AppDevAndMarketing.com
We test so many phones at App Dev + Marketing because we are exploring the issues involved with tackling cross phone performance issues.
MyChamberApp is an app we created that works on
- Android
- iPhone
- Blackberry, and
- Mobile Web.
It leads to a lot of late nights and heartache but when you hear from a customer like the Vicksburg Chamber in Mississippi saying how pleased they are it makes it all worth it.
Right now I’m wrestling with Android issues which hopefully me and my team will fix tonight for our partners, Target Marketing aka Chamber Maps.
-
Error 502 and Youtube is Down
If you tried logging into YouTube today, you’ll notice that you got an “Error 502”.
Here’s a quick summary of what that means.
According to RFC 2616, the document which among other things lists the error codes, Error 502 means “Bad Gateway.” That basically means that you can get to the Proxy server or the server that talks to your browser, but not to the servers behind that.
Your Browser -> The Gateway or Proxy Server (Error 502) -> YouTube Back End is Dead! Wikipedia lists all the error codes, too.
-
Want to be Successful Like Joel on Software?
If everything got deleted from the Joel on Software website, I would miss 2 articles:
- The Joel Test for Software Development
- The Joel Plan, which those folks implemented successfully
,
Best Working Conditions -> Best Programmers -> Best Software -> Profit! At the company where I’m the lead software developer, we are committed to the Joel Plan. At first we started out by trying developers on the cheap but have started to focus more towards making budget for a great environment (an office in Westwood close to food and parking), and hiring the best developers. We have one of the guys working on the Kohana framework, Justin Hernandez, helping us with key pieces of software.
Some folks think lean means:
- leveraged open source frameworks (LAMP + Danga) ( @ericries)
- agile
- customer-centric development, where devs interface directly with customers
We definitely agree with what Eric Ries, and other successful entrepreneurs are saying about the lean startup.
However, I personally don’t agree with folks that sacrifice the first two parts of the Joel Plan, the best conditions and the best developers, under the name of lean startup. I won’t name names because these guys in the East Coast C and D’d me before but honestly, if you want your East Coast company to think like a startup, you have to treat your East Coast developers like the valuable resource they are.
-
What You Missed At WordCamp LA
Here’s what you missed:
Installing nginx with php-fpm with varnish on the front end will make your WordPress install fly 50 times faster.
If you’re using apt-get, you can just use:
apt-get install php-fpm
Or try this guide on how-to forge.
Here’s the install process I used using PHP 5.3.3 on OS X:sudo ./configure --prefix=/usr/local --enable-fpm \ --with-fpm-user=daemon --with-fpm-group=daemon \ --with-mcrypt --with-mysql=/usr/local/mysql --with-zlib \ --enable-mbstring --disable-pdo --with-curl --disable-debug \ --disable-rpath --enable-inline-optimization --with-bz2 --with-zlib --enable-sockets --enable-sysvsem \ --enable-sysvshm --enable-pcntl --enable-mbregex \ --with-mhash --enable-zip --with-pcre-regex \ --with-iconv=shared,/usr make && make install cp sapi/fpm/php-fpm.conf /usr/local/etc/php-fpm.conf # edit php-fpm.conf with the right paths cp sapi/fpm/php-fpm /usr/local/sbin/php-fpm cp init.d.php-fpm /etc/init.d/php-fpm /etc/init.d/php-fpm start
If you get an error message it’s probably because you didn’t go through the config to set things up.
The next part is nginx.
./configure --prefix=/usr/local/nginx && make && make install
My conf/nginx.con looks like this. My sites-enabled/default.conf looks like this. My conf/fastcgi_params file is here.
I just start nginx with /usr/local/nginx/sbin/nginx and I’m good to go.
The quote that stuck with me the most was what Josh Highland said about caching:
“You should use WordPress Cache Plugins. It’s like printing money. It’s free!”
For adding your own contact form, I learned about Contact Form 7. You can ditch WuFoo if you have this configured on your WordPress.
There’s also Pods, which is like contact-form-7 except it’s a whole framework for creating your own content types and making them show up where you want.
-
iPhone 4 beats iPhone 3G in 3 ways
I got an iPhone 4 this weekend, and have been so happy with it. Here are the 3 reasons why:
- It is so much faster, 100 times faster on google maps. Apps that I stopped using because of the iOS 4 upgrade slowdown, I can now use again without issue.
- It takes great pictures and records HD video. You get 5 megapixels which is just 1 megapixel shy of a 3 year old Canon Elph.
- I think FaceTime is the bomb.
I am a bit bummed that I had to upgrade. The inconvenience makes Android look more attractive.
But if you’re still on 3G, I totally recommend the iPhone 4.