Blog

  • Why is Foursquare Down? 3 Educated Guesses

    Why is Foursquare down?

    Update (5 October 2010 at 5:36 pm PDT) : The folks at Foursquare tell us why in a post-mortem. There are autosharding issues with MongoDB. Yup, my guesses were wrong, unless you consider MongoDB a kind of cache. 😉

    I used to work for a few sites that required high scalability expertise. Now that we’re over 5 hours into the outage I’ll share some of my thoughts.

    But before I do, I’d just like to say, I really hope that it’s nothing bad and I really like the Foursquare peeps. I’m not putting out this article to harsh on anybody, but just to share some knowledge I have. Outages happen to everybody!

    Also, I do not feel that this meltdown is in any way indicative of Amazon’s EC2. I have a site that shares the same IP space and facility as Foursquare and we have had no outages today.

    • The worst case scenario is a full scale Magnolia meltdown. This is where because of a backup process that was off, they cannot restore ever from backup. Odds: unlikely.
    • Someone turned off caching. I’m not sure how cache dependent the architecture is at Foursquare. If someone turned off the cache and the cache is just plain gone, then the caches have to be re-built. Rebuilding caches, depending on the time and complexity of each query can take up to 100x more time that it takes to retrieve the cache. If there’s some cached item that takes 100 seconds per user, the site will be down for a long time. They can only put a user back on foursquare at a rate of 100 per second if that’s the case, unless they can concurrently run the re-building of the cache.
    • There’s an issue with a hacker who has broken through security and is wreaking havoc on Foursquare. It’s happened to the best sites, e.g. Google in the 90s, and it’s pretty tough to recover from. Sometimes you let the criminals in and do their worst while keeping the site up. Sometimes you have 0 tolerance.
    • I wish Foursquare the best of luck. I am more than happy to lend a hand to their issues, if they need another pair of eyes.

  • Dumping Monaco for 9×15 Font

    I’ve been using Monaco as my default font for coding for about 10 years now. I used to be a hardcore Courier fan back in college only because our printers only printed out in Courier. I wanted my emails and printouts to match.

    I ran into an article on the 9×15 font being more readable and clean, and decided to use it. Here’s the comparison below with Monaco first,

    and then 9×15:

  • 3 Things The Social Network Taught Me About Startups

    Spoiler Alert: You’ve been warned.

    If “The Social Network” has a subliminal message, it’s this:

    Swinging for the fences means forgetting about everything they taught you in business school, and if you went to business school, you are fucked. However, with the right guidance from Acclime Singapore, you might just find your footing in the chaos.

    1.Lunch Meetings and Face to Face is for losers.

    If you are going to New York to talk to ad men, you are just cosplaying “Mad Men.” In fact, any social interaction that isn’t mediated and accelerated through something like Flowtown, or Salesforce, or LinkedIn is just that, cosplay. If you are doing business over lunch, then you might as well be dressing up for Renn Faire.

    The lesson of this is one of the pivotal scenes in “The Social Network.” Not to give away too much, but in the movie, Fincher and Sorkin take great pains to show what happens to someone who doesn’t get it, even if he’s a co-founder. This person who didn’t get it has a business degree from Harvard, and went to New York to make deals for selling ads on Facebook.

    2. Coders can do it faster and better than biz dev or people who cannot code.

    Coders code much faster than any “traditional” business arrangement. If you have an idea, and cannot code it, you can never be relevant if you are swinging for the fences. Case in point: The Vinkelvoss twins did have the “idea” for a social network, but so did everybody and their grandma at the time, but execution is very different. Most business folks focus on the idea and the revenue model. The example with Facemash.com in the movie showed that this idea is flawed. Why? Making waves in society with technology will always be faster than a revenue model.

    3. Our capacity for having the wisdom to understand the technology we create outstrips the rate at which we create technology.

    If this is the case, then the point that the character of Sean Parker pushes throughout the movie, that putting ads on a site is like ending a real cool party at 11 pm, is something all startups swinging for the fences have to take to heart.

    EDUARDO: Settle an argument for us, would you? I say it’s time to start making money from TheFacebook but Mark doesn’t want advertising. Who’s right?
    SEAN: Neither of you. TheFacebook is cool, that’s what it’s got going for it… You don’t want to ruin it with ads because ads aren’t cool. It’s like you’re throwing the coolest party on campus and someone’s telling you it’s gotta be over at eleven. You don’t even know what the thing is yet.

    Great point, Sean. I mean look at what ImDB.com has become because of that ad revenue pop-up model of business. ImDB is so ugly and not cool.

  • Microsoft moves Blogs to WordPress

    At the TechCrunch Disrupt event today, Twitter user, @bigs, broke the news that Microsoft announced that it would be moving over 30 million users of their Live blogging service to WordPress.

    As a WordPress expert that has had to work on WordPress in a Windows environment there is one big gotcha: character encoding.

    Here are what common characters look like when they are moved from a Windows environment to WordPress:

    a return: ^M
    single-quotes: ?~@~Y
    asterisk: ?~@?
    double-quotes: ?~@~\ or ?~@~]

    If the WordPress migration process is robust, then it will translate these characters correctly into UTF-8 or something friendly like that.

    Live users will definitely welcome the link schema and permalinks on WordPress. Instead of

    http://cid-d4909e7f27e254e9.profile.live.com/

    a user will get

    http://barce.wordpress.com/ .

    Also WordPress users can now link their accounts to MSN Messenger so that they can update their MSN friends or co-workers.

    Are you a Windows Live Blogging user? How’d your upgrade process go?

  • Lots of Phones to Test At AppDevAndMarketing.com

    We test so many phones at App Dev + Marketing because we are exploring the issues involved with tackling cross phone performance issues.

    MyChamberApp is an app we created that works on

    • Android
    • iPhone
    • Blackberry, and
    • Mobile Web.

    It leads to a lot of late nights and heartache but when you hear from a customer like the Vicksburg Chamber in Mississippi saying how pleased they are it makes it all worth it.

    Right now I’m wrestling with Android issues which hopefully me and my team will fix tonight for our partners, Target Marketing aka Chamber Maps.

  • Error 502 and Youtube is Down

    If you tried logging into YouTube today, you’ll notice that you got an “Error 502”.

    Here’s a quick summary of what that means.

    According to RFC 2616, the document which among other things lists the error codes, Error 502 means “Bad Gateway.” That basically means that you can get to the Proxy server or the server that talks to your browser, but not to the servers behind that.

    Your Browser -> The Gateway or Proxy Server (Error 502) -> YouTube Back End is Dead!

    Wikipedia lists all the error codes, too.

  • Want to be Successful Like Joel on Software?

    If everything got deleted from the Joel on Software website, I would miss 2 articles:

    Best Working Conditions -> Best Programmers -> Best Software -> Profit!

    At the company where I’m the lead software developer, we are committed to the Joel Plan. At first we started out by trying developers on the cheap but have started to focus more towards making budget for a great environment (an office in Westwood close to food and parking), and hiring the best developers. We have one of the guys working on the Kohana framework, Justin Hernandez, helping us with key pieces of software.

    Some folks think lean means:

    1. leveraged open source frameworks (LAMP + Danga) ( @ericries)
    2. agile
    3. customer-centric development, where devs interface directly with customers

    We definitely agree with what Eric Ries, and other successful entrepreneurs are saying about the lean startup.

    However, I personally don’t agree with folks that sacrifice the first two parts of the Joel Plan, the best conditions and the best developers, under the name of lean startup. I won’t name names because these guys in the East Coast C and D’d me before but honestly, if you want your East Coast company to think like a startup, you have to treat your East Coast developers like the valuable resource they are.

  • What You Missed At WordCamp LA

    Here’s what you missed:

    Installing nginx with php-fpm with varnish on the front end will make your WordPress install fly 50 times faster.

    If you’re using apt-get, you can just use:
    apt-get install php-fpm
    Or try this guide on how-to forge.
    Here’s the install process I used using PHP 5.3.3 on OS X:

    sudo ./configure --prefix=/usr/local --enable-fpm \ 
      --with-fpm-user=daemon --with-fpm-group=daemon \ 
      --with-mcrypt --with-mysql=/usr/local/mysql --with-zlib \
      --enable-mbstring --disable-pdo --with-curl --disable-debug \
      --disable-rpath --enable-inline-optimization --with-bz2 
      --with-zlib --enable-sockets --enable-sysvsem \
      --enable-sysvshm --enable-pcntl --enable-mbregex \
      --with-mhash --enable-zip --with-pcre-regex \ 
      --with-iconv=shared,/usr
    make && make install
    cp sapi/fpm/php-fpm.conf /usr/local/etc/php-fpm.conf
    # edit php-fpm.conf with the right paths
    cp sapi/fpm/php-fpm /usr/local/sbin/php-fpm
    cp init.d.php-fpm /etc/init.d/php-fpm
    /etc/init.d/php-fpm start
    

    If you get an error message it’s probably because you didn’t go through the config to set things up.

    The next part is nginx.

    ./configure --prefix=/usr/local/nginx && make && make install
    

    My conf/nginx.con looks like this. My sites-enabled/default.conf looks like this. My conf/fastcgi_params file is here.

    I just start nginx with /usr/local/nginx/sbin/nginx and I’m good to go.

    The quote that stuck with me the most was what Josh Highland said about caching:

    “You should use WordPress Cache Plugins. It’s like printing money. It’s free!”

    For adding your own contact form, I learned about Contact Form 7. You can ditch WuFoo if you have this configured on your WordPress.

    There’s also Pods, which is like contact-form-7 except it’s a whole framework for creating your own content types and making them show up where you want.

  • iPhone 4 beats iPhone 3G in 3 ways

    I got an iPhone 4 this weekend, and have been so happy with it. Here are the 3 reasons why:

    • It is so much faster, 100 times faster on google maps. Apps that I stopped using because of the iOS 4 upgrade slowdown, I can now use again without issue.
    • It takes great pictures and records HD video. You get 5 megapixels which is just 1 megapixel shy of a 3 year old Canon Elph.
    • I think FaceTime is the bomb.

    I am a bit bummed that I had to upgrade. The inconvenience makes Android look more attractive.

    But if you’re still on 3G, I totally recommend the iPhone 4.