Using new PECL Memcached extension for storing session data

Many of you already know that managing session is a critical task for web applications, specially when you want to avoid I/O hop and also a significant load over your database by writing a custom session handler. Beside that, if your application makes use of multiple web servers behind a proxy, then its more than a critical job to share and manage session data among these servers effectively. This is why a central session manager is very important for your application to scale. In this article I am going to show you how you can use the latest Memcached extension (developed by Andrei Zmievski and his team) to isolate the session storage from web servers. I will show you how to compile the extension and use it.

Step1: Install Memcached Server
If you are using Debian its just plain simple

apt-get install memcached

Step 2: Run memcached instances
Lets run two instances of memcached in same machine (well, this article is just for demonstrating you how you can get things done. In the production environment, you can deploy as many memcached instances as you want in different servers in same network)

memcached -d -l 127.0.0.1 -p 11211 -u <username> -m 16
memcached -d -l 127.0.0.1 -p 11212 -u <username> -m 16

Above commands will run two instances of memcached listening on port number 11211 and 11212, same IP 127.0.0.1. Each of them get an allocation of 16 MB of memory (on RAM).

Step 3: Install the PECL Memcached extension.
Lets install the new PECL memcached extension in your web server. This new extension depends on libmemcached. You can grab the latest distribution of libmemcached from https://launchpad.net/libmemcached and compile it in your own machine. Make sure you have the dependencies met.

wget http://launchpad.net/libmemcached/1.0/0.34/+download/libmemcached-0.34.tar.gz
tar -zxvf libmemcached-0.34.tar.gz
cd libmemcached-0.34
./configure
make && make install

Considering everything went fine, lets install the PECL memcached extension

pecl install memcached

If everything goes fine, you should see the output similar like this

Build process completed successfully
Installing '/usr/lib/php5/20060613/memcached.so'
install ok: channel://pecl.php.net/memcached-1.0.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=memcached.so" to php.ini

Make sure that memcached.so is placed in your PHP extension_dir folder (here /usr/lib/php5/20060613). Add the line “extension=memcached.so” in your php.ini and restart your web server.

To make sure, everything’s done and working – run a phpinfo() and check the output. There should be a “memcached” sesction which will look like the following one.

Memcached PECL Extension

Now we need to make change in our php.ini to register Memcached as a session handler and set the necessary properties there. Open your php.ini and add the following two lines. If you find any similar line un-commented, comment them out first.

session.save_handler=memcached
session.save_path="127.0.0.1:11211, 127.0.0.1:11212"

Restart your web server. And …… you are done! – Now all your session data will be saved and served from these memcached servers. No matter whenever you need to extend your setup by adding extra web servers, all user data and session data will remain valid and served from a central location. No I/O issue, no huge write load on DB servers.

jQuery:hooking form submit and making it ajax request

modern javascript frameworks are big blessings to every front end developer. they made our life so much easier so that we can sleep more and become fat day by day – i am a big fan of jQuery and mootools.

in this post i am going to show you how you can hook a normal form submission process, regardless of it’s method GET or POST, and convert it into an ajax request. the whole process will work dynamically. it will parse form input elements, make a JSON array from them and make an ajax request to the appropriate action url. after that, it will invoke the user supplied callback function.

problem 1: parsing form elements was a small challenge. you can do it in various way (by traversing or serialize or using css selectors). i choose to use serializer routines. jQuery has builtin support for two types of serializing , one is $(form).serialize() and another is $(form).serializeArray(). lets have a look at the output of both of them for the following form

<form id="f1" action='some target' method='POST'>
<input type ="textbox" id='username' name ='username' value='me'/>
<input type ="checkbox" value=1 id='guests' name ='guests'/>
</form>

now lets check the output by both serialize() and serializeArray() method

alert ($('#f1').serialize());
//output is "username=me&guests=1"

alert($('#f1').serializeArray()
//output is [{object},{object}]

are you scared seeing this [object] output of serializeArray()? well dont panic. serializeArray() returns a JSON structure. you can still investigate using toSource() method

alert($('#f1').serializeArray().toSource()
//output is [{"username":"me"},{"guests":"1"}]

but that will not be usable to send in our AJAX request. we need a JSON array key/value pair (or you can use the output of serialize() function too to send as data in ajax request, the serilizeForm part is completely optional )

lets create a new function called serializeForm which will create JSON key/value pair out of serializeArray() and do the rest of the work.

$.fn.serializeForm = function()
{
    data = {};
    url = this.attr("action");
    items = this.serializeArray();
    $.each(items,function(i,item)
    {
        data[item['name']]=item['value'];
    }
    );
    return data;
}

now lets hook the normal submit process of the form using the following hook

function submitHook(form, callback)
{
    $(form).submit(function(e){
        items = {};
        items = $(form).serializeForm();
        url = $(form).attr("action");
        if(""==url)
        {
            alert("Cannot submit form. No action specified");
            return false;
        }
        callback = callback?callback:function(){};
        $.post(url,items,callback);
        return false;
    });
}

now you can just hook the form simply by this

ourCallback = function (data)
{
   alert(data);
}

submitHook($('#f1'),ourCallback);

happy jQuerying

Using Google WorldMap visualization component in your applications

Its really hard to find a good flash based world map component to use in your web applications. They are either too complex to add or they cost quite a lot. And even sometime if you pay that, its tough to customize the chart as you want it to look like. But you know there is someone called “Uncle G” (i.e google) here who has almost all the components in his Pandora’s box. So lets see how can we use the geomap component from their Visualization Library.

First we need to create a datatable which will act as a data source for this world map, as usual like all other google visualization component.

var data = new google.visualization.DataTable();
data.addRows(5);
data.addColumn('string', 'Country');
data.addColumn('number', 'Number of ZCEs');
data.setValue(0, 0, 'Bangladesh');
data.setValue(0, 1, 19);
data.setValue(1, 0, 'India');
data.setValue(1, 1, 150);
data.setValue(2, 0, 'Pakistan');
data.setValue(2, 1, 4);
data.setValue(3, 0, 'Nepal');
data.setValue(3, 1, 5);
data.setValue(4, 0, 'Sri Lanka');
data.setValue(4, 1, 7);

now we will initialize the google visualization framework and draw this component using this data source

var geomap = new google.visualization.GeoMap(document.getElementById('<container_div_id>'));
geomap.draw(data, null);

but wait, we are not done yet – to make sure that everything works properly, we need to wrap all of these code inside a function (for example name this function as drawGeoMap) and we will use that function as a callback to draw this map. and of course, we need to load the google visualization library and geomap component first

so here is the complete code of this

<html>
<head>
  <script type="text/javascript" src="http://www.google.com/jsapi"></script>
</head>
<body>
<div id="map" style="width: 800px; height: 600px;"></div>
  <script type="text/javascript">
    google.load('visualization', '1', {packages: ['geomap']});

    function drawGeoMap() {
      var data = new google.visualization.DataTable();
	  data.addRows(5);
	  data.addColumn('string', 'Country');
	  data.addColumn('number', 'Number of ZCEs');
	  data.setValue(0, 0, 'Bangladesh');
	  data.setValue(0, 1, 19);
	  data.setValue(1, 0, 'India');
	  data.setValue(1, 1, 150);
	  data.setValue(2, 0, 'Pakistan');
	  data.setValue(2, 1, 4);
	  data.setValue(3, 0, 'Nepal');
	  data.setValue(3, 1, 5);
	  data.setValue(4, 0, 'Sri Lanka');
	  data.setValue(4, 1, 7);

      var geomap = new google.visualization.GeoMap(
          document.getElementById('map'));
      geomap.draw(data, null);
    }

    google.setOnLoadCallback(drawGeoMap);
  </script>
</body>
</html>

you can check the demo at http://sandbox.ofhas.in/geomapv1.php

it will display a world map with highlighted countries like below

but wait – lets make ca nifty change and add event listeners to it. we will add event listeners in such a way so that whenever users click on any country in the map, it will take you to zend yellow page corresponding to that country that will make it really an useful component . here is the code

<html>
<head>
  <script type="text/javascript" src="http://www.google.com/jsapi"></script>
</head>
<body>
<div id="map" style="width: 800px; height: 600px;"></div>
  <script type="text/javascript">
  	var data, geomap;
  	var cids = [18,102,166,152,203]
    google.load('visualization', '1', {packages: ['geomap']});

    function drawGeoMap() {
      data = new google.visualization.DataTable();
	  data.addRows(5);
	  data.addColumn('string', 'Country');
	  data.addColumn('number', 'Number of ZCEs');
	  data.setValue(0, 0, 'Bangladesh');
	  data.setValue(0, 1, 19);
	  data.setValue(1, 0, 'India');
	  data.setValue(1, 1, 150);
	  data.setValue(2, 0, 'Pakistan');
	  data.setValue(2, 1, 4);
	  data.setValue(3, 0, 'Nepal');
	  data.setValue(3, 1, 5);
	  data.setValue(4, 0, 'Sri Lanka');
	  data.setValue(4, 1, 7);

      geomap = new google.visualization.GeoMap(
     	document.getElementById('map'));
     	 google.visualization.events.addListener(geomap, 'select', selectHandler);
      geomap.draw(data, null);

    }

    google.setOnLoadCallback(drawGeoMap);

    function selectHandler()
    {
        var selection = geomap.getSelection();
        var item = selection[0];
        var cid = cids[item.row];
        var url = "http://www.zend.com/en/yellow-pages#list-cid="+cid+"&firstname=&lastname=&orderby=name&sid=XX&company=&photo_first=&certtype=&ClientCandidateID="
        window.open(url);

    }
  </script>
</body>
</html>

check the demo at http://sandbox.ofhas.in/geomapv2.php

now you can click on any country and it will open a new tab with that particular country pre selected – and you can see who are the zend certified engineers from that country. i hope you’ve liked this . Thanks goes the theam Visualization team at google for creating these awesome components and to make them free for use

For reference – check out geomap reference at google code at http://code.google.com/apis/visualization/documentation/gallery/geomap.html. you can do many other cool things like displaying only US states map or Canadian States map with this

collecting data from streaming APIs in twitter

twitter’s streaming API is still in beta and is a good source of collecting public tweets. but unfortunately not all those methods are instantly usable by third parties (u need to provide written statements and so on). but for testing, three of these streaming APIs are usable by anyone at this moment which are spritzer, track and follow. spritzer streams a tiny part of public tweets to the collecting processes. in this blog post i’ll show you how to collect data from spritzer API.

as it is a stream data, so twitter keeps the HTTP connection “alive” infinitely (until any hiccup, by using Keep Alive). so when you write code, you must take care of that. and i would also suggest to make separate processes for collecting data+writing them (or sending them in queue to be written) – and for analyzing those data. and of course, to minimize the bandwidth consumption, use the json format. and json data is also easier to parse than XML as every tweet is separated by a new line (“\n”) character from twitter – so you can read these data line by line, dcode them using json_decode() and do whatever you want

here is how you can create the collector process in php

< ?php
//datacollector.php
$fp = fopen("http://username:password@stream.twitter.com/spritzer.json","r");
while($data = fgets($fp))
{
    $time = date("YmdH");
    if ($newTime!=$time)
    {
        @fclose($fp2);
        $fp2 = fopen("{$time}.txt","a");
    }
    fputs($fp2,$data);
    $newTime = $time;
}
?>

this script will write the data collected hourly from the spritzer streaming API in filen (with names like <YmdH>.txt ). so in the directory where you are runnign this script u will see hourly data files. like 2009062020.txt . there is a special advantage to keep collecting in this way – as the file will remain open for writing (hence LOCKED) you will process files only for previous hours. it will make analyzing the data more hassle free

now run this script in background via the following command in your terminal

php datacollector.php &

the reason for appending an “&’ at the end of the command is starting this process in background. so that you dont have to wait for the script to end to get access to your shell back. as it is a streaming data, the script will run infinitely. and it will consume very minimal bandwidth you can check yourself.

so i hope it will help those developers who are looking for a solution to collect data from twitter’s streaming API via PHP. If you want to track any specific keywords, use the “track” API instead . and if you want to follow some particular person use the “follow“. Check out twitter’s documentation of streaming API for more

my business card

just printed some copies of my business card – would love to see your comments on this.

expanding short url to original url using PHP and CURL

there are numbers of url shortening services available these days, including the good old tinyurl and something really short like u.nu. now when you get the short url shortened by using any of these services, you dont know where your browser is taking you! so if you are interested to figure out the original url hiding behind these short url, you need to have a little knowledge on how these services actually work. if you go to any of these short urls, they tell your browser “HTTP 30X: Object has moved” HTTP HEADER (optionally, some does it, some doesn’t) and then asks your browser to move to the original url using “Location” in HTTP HEADER. so all you have to do is just get the HTTP HEADER out first (PHP and Curl is pretty good at doing this, heh heh) and then parse the “Location” parameter from it.

lets see how that works in code

< ?php
$url = "http://tinyurl.com/2dfmty";
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_HEADER,true);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,false);
$data = curl_exec($ch);
$pdata = http_parse_headers($data);
echo "Short URL: {$url}<br/>";
echo "Original URL: {$pdata['Location']}";

function http_parse_headers( $header )
    {
        $retVal = array();
        $fields = explode("\r\n", preg_replace('/\x0D\x0A[\x09\x20]+/', ' ', $header));
        foreach( $fields as $field ) {
            if( preg_match('/([^:]+): (.+)/m', $field, $match) ) {
                $match[1] = preg_replace('/(?< =^|[\x09\x20\x2D])./e', 'strtoupper("")', strtolower(trim($match[1])));
                if( isset($retVal[$match[1]]) ) {
                    $retVal[$match[1]] = array($retVal[$match[1]], $match[2]);
                } else {
                    $retVal[$match[1]] = trim($match[2]);
                }
            }
        }
        return $retVal;
    }
?>

now you see that the output of this code is

Short URL: http://tinyurl.com/2dfmty
Original URL: http://ghill.customer.netspace.net.au/embiggen/

pretty interesting huh? if you analyze the full headers for each of these services you will find that most of them are using PHP in backend with Apache. only http://u.nu is using mod_rails (hence RoR) and bit.ly uses nginx

have fun in expanding!

using oauth pecl extension to talk to twitter

if you are interested in developing twitter applications, you must have read about twitter API and it’s authentication protocol. your application can fetch user’s private data but it has to authenticate itself as the user for that. so there are two ways to do it

1. asking user to provide his twitter username and password to your application (well, i am not interested to give away my PASSWORD to anyone!!!)
2. let twitter handle the authentication on behalf of you and ask user to grant permission to your application (hmm!! interesting)

now you see that #2 is more safe for your user. and i think most security concerned users will choose this way. so your application have to initiate this type of authentication system using twittter’s supported authentication protocol oAuth (it’s a commonly used authentication protocol used among number of popular service providers like yahoo, google and others)

to implement oauth in php, the best way is to use an existing library. there are now numbers of libraries available for this purpose. following are some of them
1. oauth lib by andy smith
2. oauth library by marc worrell
3. oauth pecl extension by rasmus lerdorf and john jawed and felipe pena

now you see, pecl extensions are written in c and runs pretty faster. so i choose it without thinking much abt it. i have assumed that you know how to install a pecl extension in your php hosting and i am not going to blog detail about that right now. all that can help you right now is shell command “pecl install -f oauth” – you know, nothing talks better than command or code

after installing oauth extension in my hosting account, i start developing my twitter application. first i have to register my application with twitter. you can create your one by pointing your browser to http://twitter.com/oauth_clients/new. please remember that you have to provide a callback url which twitter use to redirect user of your application after a success/unsuccessful authentication. i will refer to that url as “callback_url” through out this blog post. my applications callback_url is “http://mydomain.tld/auth.php”

after you have done registering your application with twitter, it will give you the following important data.
1. consumer key
2. consumer secret
3. request token url
4. access token url
5. authorize url

you will be going to use all of these in your application. now lets see how oauth works in brief. it initiate the talk using your consumer key and secret key. and then it request the “request token” from the service provider. if u r successful, you have to forward user of your application to the “authorize url” with the “request token”. now the service provider will ask to grant permission to your application from the user. if user grants (or disagree) the permission, the service provider (here, twitter) will forward your user again to the “callback url” of your application with a “new token”. now with the help of this new token and the token grabbed from previous “request token” your application will ask for “access token”. once you have the access token, you can authorize you application as the user itself with same privilege.

lets see how to do it in php with the help of oauth pecl extension. here we are going to initiate the talk, get the token and forward user to the service provider’s authorizing url.

token.php

< ?php
//token.php
$oauth = new OAuth("consumer key","consumer secret",OAUTH_SIG_METHOD_HMACSHA1,OAUTH_AUTH_TYPE_URI); //initiate
$request_token_info = $oauth->getRequestToken("http://twitter.com/oauth/request_token"); //get request token
file_put_contents("token.txt",$request_token_info['oauth_token_secret']);//store the oauth token secret of request token
header('Location: http://twitter.com/oauth/authorize?oauth_token='.$request_token_info['oauth_token']);//forward user to authorize url
?>

you see that we are storing the oauth_token_secret of the “request_token” because we need it in our next step to fetch access token. in the example above i am storing it in flat file, but you will have to store it in db/file with proper index to the userid so that you can retrieve it later in our next step.

if user visit this page, he will be redirected to twitter authorize url and that may look like the following one with different app name.

now lets see how we handle if the user click “allow” or “deny” in the above page.

this is the callback file you specified in settings of your app [auth.php]

< ?php
//auth.php
$oauth = new OAuth("consumer key","consumer secret",OAUTH_SIG_METHOD_HMACSHA1,OAUTH_AUTH_TYPE_URI); //initiate
$request_token_secret = file_get_contents("token.txt"); //get the oauth_token_secret of request token we stored previously
if(!empty($_GET['oauth_token'])){
$oauth->setToken($_GET['oauth_token'],$request_token_secret);//user allowed the app, so u
$access_token_info = $oauth->getAccessToken('http://twitter.com/oauth/access_token');
}
?>

access token is the most important token for your application. there are two object in this token – one is “oauth_token” and “oauth_token_secret”. if you print_r the access token it will look like the following one (actual value is not shown here)

Array (
    [oauth_token] => abcdefg
    [oauth_token_secret] => uvwxyz
)

you have to store this access token for authorizing later as this user (the user that was visiting). using this token you can anytime authorize yourself as that user and fetch user’s data from twitter. so lets see how we can fetch user’s profile data in rss (or json) format. the REST API url to fetch this data is “http://twitter.com/account/verify_credentials.json”. you can find other important REST urls to fetch user’s timeline, public timeline and friends timeline (also update status) in twitter’s documentation of it’s REST API

fetch user’s profile data

< ?php
//profile.php
$oauth = new OAuth("consumer key","consumer secret",OAUTH_SIG_METHOD_HMACSHA1,OAUTH_AUTH_TYPE_URI); //initiate
$oauth->setToken($accesstoken['oauth_token'],$accesstoken['oauth_token_secret']);
$data = $oauth->fetch('http://twitter.com/account/verify_credentials.json');
if($data){
    $response_info = $oauth->getLastResponse();
    echo "<pre>";
    print_r(json_decode($response_info));
    echo "</pre>";
}

the output of this code is the following one (my twitter username is hasin)

stdClass Object
(
    [time_zone] => Dhaka
    [friends_count] => 97
    [profile_text_color] => 666666
    [description] => Smoking too much PHP
    [following] =>
    [utc_offset] => 21600
    [favourites_count] => 2
    [profile_image_url] => http://s3.amazonaws.com/twitter_production/profile_images/84574185/afif_b_normal.jpg
    [profile_background_image_url] => http://s3.amazonaws.com/twitter_production/profile_background_images/5565492/777481225666153.jpg
    [profile_link_color] => 2FC2EF
    [screen_name] => hasin
    [profile_sidebar_fill_color] => 252429
    [url] => http://hasin.wordpress.com
    [name] => hasin
    [protected] =>
    [status] => stdClass Object
        (
            [text] => ok, understood how twitter auth works via oauth pecl ext. of #php. thanks to @rasmus for his excellent example
            [in_reply_to_user_id] =>
            [favorited] =>
            [in_reply_to_screen_name] =>
            [truncated] =>
            [created_at] => Sat May 02 16:08:28 +0000 2009
            [id] => 1679349376
            [in_reply_to_status_id] =>
            [source] => web
        )

    [profile_sidebar_border_color] => 181A1E
    [profile_background_tile] => 1
    [notifications] =>
    [statuses_count] => 1147
    [created_at] => Fri Nov 09 10:40:14 +0000 2007
    [profile_background_color] => 1A1B1F
    [followers_count] => 265
    [location] => Dhaka, Bangladesh
    [id] => 10094392
)

wrapper for google static map api

google maps is one of the very best mapping services available these days. using their api you can embed interactive maps of different types in your web pages. and there are many plugins to make your life easier to embed such maps in your web pages and applications.

to embed interactive google map, you need to include some external javascript files first. now when you are using some hosted solutions like wordpress, they may not allow you to include external javascript files in your blog/pages for different security pre-cautions. this is where google static map api comes handy.

google static maps api takes latitude and longitude of some places as parameter, along with other optional formatting params and delivers a static image in your expected image format (default is gif, but you can choose among gif, png and jpeg). so you dont need to bother about permission of including any other javascript files in your pages and you can embed such static maps by specifying the url as a “src” attribute of <img> tag.

there is still one problem that might knock you down. for example if you want to display “dhaka”, the capital or bangladesh in your static map, you need to know it’s latitude and longitude. so not everyone of us speaks in a language of ‘latitude’ and ‘longitude’ but onle the name of the places. like we all know about “university of berkeley” but only very few can figure out it’s lat-lon or they have to find it out from existing geocoding services.

yesterday night i was studying about this static map api and decided to build a geocoding wrapper so that you can pass only the name of a place and my wrapper class will convert it to latitude and longitude using geociding services and then generate a static map image for you. the whole wrapper will work around two objects – geocoding service and google static map api. for geocoding i prefer to use google because of it’s amazing precision but i must say that yahoo’s geocoding service is also pretty good.

here is my wrapper class and i am sure you can find it useful. now you can use it to embed google map to anywhere you would like to . this class is released under LGPL.

there is a sample usage file (gmap.php which you will make use of) and a source object “staticgmap.php”. this class became a part of orchid framework and prothom alo blog. if you enjoy/use this class, dont forget to drop a line here in this post. the class and usage file is moderately commented, so you will not face any problem to understand. however the usage file is also pretty fine working.

please download the class from this page
download: http://svnpg.net/gmap/

supported features
1. geocoding service, both yahoo and google (default is yahoo)
2. support of zooming (0-19, default is 12)
3. support of image format (gif, jpeg and png. default is GIF)
4. maptype support (roadmap, satellite, mobile, hybrid and terrain where default is roadmap)
5. support of markers (just as plain location name)
6. variable dimension support (default size is 400×400)

sample maps

locate dhaka and moti jheel
http://svnpg.net/gmap/gmap.php?location=dhaka&markers=dhaka,moti jheel

locate berkeley and emeryvlle
http://svnpg.net/gmap/gmap.php?location=berkeley&markers=berkeley,emeryville&zoom=11

place where i live – uttara 14 with custom zoom level and custom size
http://svnpg.net/gmap/gmap.php?location=uttara 14&zoom=16&size=400×300

Look Ma, everyone’s computing out there for me!

SETI@Home is probably the greatest example of low cost distributed computing which become a big hit. After their tremendous success, many others over there started following the same strategy and used the power of distributed computing for other purposes like cancer research. In this article I will show you how we can use the same power at almost zero cost, and specially for your web applications.

As I am currently working on building an open source version of FriendFeed (not targetted to be an alternative, coz those people at FriendFeed done their job really well) and scaling such a huge load effectively at low cost, so I will mainly talk more about FriendFeed through out this blog post and use it as an example for my proposal.

If you consider FriendFeed as a repository of feed URLs, a lot of peoples and how they are related to each other, you can assume how big it is or it could be in near future. And scaling such a service would cost numbers of sleepless nights of many developers out there. So in basic, lets focus where actually the problem is and how we can introduce distributed computing.

Beside optimizing database to serve huge sets of data, one of the main problems of such a service has to parse millions of feeds in a regular interval. If we want to bear all the loads on your server, fine, if you can afford. But what about some low cost solutions. Lets consider a simple scenario, if your application has one million of users and each of them browse your application for 10 minutes a day, you really have 10 millions of computational power just wasting over there, in lanes and by lanes of internet – heh heh. So lets make use of such an incredible CPU power. All you have to do let the visitors machine do some calculations for you and free your server from gigantic load.

When the users of your application and relation among them are stored in your database, you can easily find out the first degree and second degree friends of a specific user. If you don’t know what does that mean, its simple, If A is a friend of B and C is a friend of A, then A is B’s first degree friend and C is B’s second degree friend. For a huge social network, it may look like the following one when you visualize the relationship

image courtesy: http://prblog.typepad.com

Now what we want to do is when B is visiting our application we want to parse most of his/her second degree friends in client using his browser. So while generating the page for B, we will supply him a bunch of feed URLs, a hash of their last known update time or hash of the latest item of each of these corresponding feeds, and a javascript based parser (for example Google’s ajax Feed API would do fine) script. Now, while B is browsing our application we will parse those feeds of his second degree friends using javascript without bothering him for a single second and post back the parsed contents (after checking against the hash for really updated contents) to a server side script which will then blindly (not totally, after some validation or authentication for sure) insert those result to our database. Now when A comes and visit his page (A is C’s first degree friend), he will get all the latest result from C’s feeds as B has already done the parsing job for us and we have those latest result from C’s feeds stored in our Database.

There are definitely more challenge than it is explained here, like what if a person is second degree friend of a multiple user. In such cases as we supplied last update time of these feeds while generating a page, we can calculate from server side which feeds we really want to parse.

And moreover, we can add more check for our javascript parser than just blindly parse those feeds. We can download a chunk of RSS or Atom feeds (using a proxy script developped using curl range options) and read just up to the latest update time and extract the time using simple reg ex or string functions instead of downloading full feed data. Now if we know that a script is not uptodate just by downloading 1-2 killobyte of data instead of downloading full feed, parsing the xml data, it would save us more computing resources for performing other jobs.

But of course, you cannot live completely on your client’s machine for parsing all your data. You must have several cron’d scripts to parse left overs or other feeds at server side. But what I am saying is that with little help of javascript you can make use of such a tremendous distributed computing power in your application, all at almost no cost.

I will come with example code once I am done developing my open source clone of FriendFeed and then I am sure, you will find it was worth writing a blog post about.

image courtesy: http://www.naccq.ac.nz/bacit/0203/2004Caukill_OffPeakGrid.htm

Have a nice RnDing time.

Building services like FriendFeed using PHP – Part2

Following the first installment in this series, here is the second part. In this part I will focus mainly on Bookmarking and News services supported by FriendFeed . Here we go

Supported bookmarking services by FriendFeed
1. Del.icio.us
2. Furl
3. Google shared stuffs
4. Mag.nolia
5. Stumbleupon

Except google shared stufss, all of the rests require just an username to generate the access point for retrieving user’s bookmarked items. And for the google shared stuffs, it requires the fully functional url of the feed available from your google bookmark service.

Access points

Del.icio.us

AP: http://feeds.delicious.com/rss/<user name>
example: http://feeds.delicious.com/rss/anduh

---

Furl

AP: http://rss.furl.net/member/<user name>.rss;
example: http://rss.furl.net/member/pigge.rss

---

Google Shared Stuffs
you can find your google shared stuff url from http://www.google.com/s2/sharing/stuff

example: http://www.google.com/s2/sharing/stuff?user=110703083499686157981&output=rss

---

Ma.gnolia

AP: http://ma.gnolia.com/rss/lite/people/<user name>.rss;
example: http://ma.gnolia.com/rss/lite/people/gerryquach

---

Stumbleupon
Whoops, Double Whoops, Tripple Whoops. It took quite a time for me to find the feed url. I dont know why it is kept so “SECRET” – LOL

AP: http://www.stumbleupon.com/syndicate.php?stumbler=<user name>.rss;
example: http://www.stumbleupon.com/syndicate.php?stumbler=jd001

---

Supported news services by FriendFeed
1. Digg
2. Google Reader
3. Mixx
4. Reddit

Here are the access points

---

Digg

AP: http://digg.com/users/<user name>/history.rss;
example: http://digg.com/users/msaleem/history.rss

---

Google Reader

You can find your shared item’s feed url here http://www.google.com/reader/view/user/-/state/com.google/broadcast

---

Mixx

Unfortunately during the time of writing this article, Mixx was napping – here is the screenshot. Once they are awake, I will update this section

---

Reddit

AP: http://www.reddit.com/user/<user name>/.rss;
example: http://www.reddit.com/user/jack_alexander/.rss

In next installation I will focus on scaling such a huge load successfully. Hope that will be interesting to many of you. Following installation will focus again on the access points.