Month: November 2009

  • WP Geo Plugin

    The print_GeoCache_Url function came across my email today from a self-described local designer and geek, but after a little research, I found out it only works up to WordPress 1.2 . Thank goodness for the WordPress WP-Geo Plug-in which I’m using right now.

    More info here: WPGeo.com

    [wp_geo_map]

  • EC2 Backup Script

    This is a quick and dirty EC2 backup script for virtual unix servers that works just fine when crontabbed:

    #!/bin/bash

    DATE=`date +%m%d%Y-%H%m%M`
    BUCKET=”codebelay-$DATE”
    PRIVATE_KEY=’pk-codebelay.pem’
    PRIVATE_CERT=’cert-codebelay.pem’
    USERID=’555555555555′
    AWS_ACCESS_ID=’AKIA0000000000000′
    AWS_SECRET=’asdf+asdf+asdf+asdf’

    s3cmd mb s3://$BUCKET

    cd /mnt
    mkdir img
    ec2-bundle-vol -d /mnt/img -k /mnt/$PRIVATE_KEY -c /mnt/$PRIVATE_CERT -u $USERID -s 9999 –arch i386
    cd /dev
    mkdir loop
    cd loop
    mknod 0 b 7 0

    ec2-upload-bundle -b $BUCKET -m /mnt/img/image.manifest.xml -a $AWS_ACCESS_ID -s $AWS_SECRET

    # rm -rf /mnt/img
    echo “please register $BUCKET/image.manifest.xml” >> /mnt/registerbackups.txt

  • Notes on adding more MySQL databases

    Just notes for myself on adding more MySQL databases without shutting down the master database.

    on existing slave:

    /etc/init.d/mysqld stop

    copy data dir from /var/lib/mysql and data from /var/run/mysqld to new slave database:

    cd /var/lib
    tar cvf Mysql_slave.tar mysql/*
    scp Mysql_slave.tar root@new-db.com:/var/lib/.
    cd /var/run
    tar cvf Mysqld_slave.tar mysqld/*
    scp Mysqld_slave.tar mysqld/*
    scp Mysqld_slave.tar root@new-db.com:/var/run/.

    copy /etc/my.cnf from old slave to new slave
    add entry for new server-id

    start existing slave:

    cd /var/lib
    tar xvf Mysql_slave.tar
    cd /var/run
    tar xvf Mysqld_slave.tar
    /etc/init.d/mysqld start

    start new slave:

    /etc/init.d/mysqld start
    mysql
    start slave;

    on masterdb:
    e.g.:

    grant replication slave on *.* to ‘repl’@’192.168.107.33’ identified by ‘password’;

    test on master:
    create database repl;

    check on slave:
    show databases; /* should show new database */

    test on master:
    drop database repl;

    check on slave:
    show databases; /* new database should be dropped */

    Now it’s time to turn this into an automated shell script with Expect in there.

  • Part II: Getting to 600 Concurrent Users

    I couldn’t sleep last night. I’m worried we’ll lose this client.

    So just to be clear. I wasn’t part of the crew responsible for scaling this site. I had already set up a scalable architecture for the site, that would automatically and horizontally scale at Amazon. That idea got shot down for legal reasons that to my surprise haven’t been in play for awhile. Can we say, “Office politics?”

    I totally recommend Amazon’s Autoscaling to anybody that’s new to this.

    Instead of auto-scaling, the site was architected by a local San Francisco firm who I won’t mention here.

    Let’s just hope enough people read this so that they won’t even have to know the name of the company and will just know the smell of an un-scaleable architecture.

    Scalability requirement: 100,000 concurrent users

    This is how they set it up:

    • two web servers
    • one database
    • four video transcoders that hits the master database
    • one more app server that hits the master database
    • no slave db 😀

    If they had even googled ‘building scalable websites’ they would have come across a book that would have avoided all of this, Cal Henderson’s Building Scalable Websites. It should be mandatory reading for anybody working on a large website, and it just scratches the surface.

    So, how did we get to 600 concurrent users?

    We tweaked mysql by putting this in /etc/m.cnf:

    [mysqld]
    max_connections=10000
    query_cache_size=50000000
    thread_cache_size=16
    thread_concurrency=16 # only works on Solaris and is ignored on other OSes

    We ran siege and were able to get to about 300 concurrent users without breaking a sweat, but now apache was dying.

    So we tweaked apache. We started out with this:

    StartServers 8
    MinSpareServers 5
    MaxSpareServers 20
    ServerLimit 256
    MaxClients 256
    MaxRequestsPerChild 4000

    And ended up with this:

    StartServers 150
    MinSpareServers 50
    MaxSpareServers 200
    ServerLimit 256
    MaxClients 256
    MaxRequestsPerChild 4000

    RAM and CPU were doubled.

  • Scaling from 100 to 100000 concurrent users in a day?

    Well, it looks pretty bad right now. A vendor just ceded control for web application architecture. Initial tests say that the site won’t do no more 100 users concurrently.

    Who the hell makes a web app without a slave database and calls themselves website architects? Apparently these guys did.

    Please start following if you want to see if this web app can make it to launch.