Look Ma, everyone’s computing out there for me!

SETI@Home is probably the greatest example of low cost distributed computing which become a big hit. After their tremendous success, many others over there started following the same strategy and used the power of distributed computing for other purposes like cancer research. In this article I will show you how we can use the same power at almost zero cost, and specially for your web applications.

As I am currently working on building an open source version of FriendFeed (not targetted to be an alternative, coz those people at FriendFeed done their job really well) and scaling such a huge load effectively at low cost, so I will mainly talk more about FriendFeed through out this blog post and use it as an example for my proposal.

If you consider FriendFeed as a repository of feed URLs, a lot of peoples and how they are related to each other, you can assume how big it is or it could be in near future. And scaling such a service would cost numbers of sleepless nights of many developers out there. So in basic, lets focus where actually the problem is and how we can introduce distributed computing.

Beside optimizing database to serve huge sets of data, one of the main problems of such a service has to parse millions of feeds in a regular interval. If we want to bear all the loads on your server, fine, if you can afford. But what about some low cost solutions. Lets consider a simple scenario, if your application has one million of users and each of them browse your application for 10 minutes a day, you really have 10 millions of computational power just wasting over there, in lanes and by lanes of internet - heh heh. So lets make use of such an incredible CPU power. All you have to do let the visitors machine do some calculations for you and free your server from gigantic load.

When the users of your application and relation among them are stored in your database, you can easily find out the first degree and second degree friends of a specific user. If you don’t know what does that mean, its simple, If A is a friend of B and C is a friend of A, then A is B’s first degree friend and C is B’s second degree friend. For a huge social network, it may look like the following one when you visualize the relationship

image courtesy: http://prblog.typepad.com

Now what we want to do is when B is visiting our application we want to parse most of his/her second degree friends in client using his browser. So while generating the page for B, we will supply him a bunch of feed URLs, a hash of their last known update time or hash of the latest item of each of these corresponding feeds, and a javascript based parser (for example Google’s ajax Feed API would do fine) script. Now, while B is browsing our application we will parse those feeds of his second degree friends using javascript without bothering him for a single second and post back the parsed contents (after checking against the hash for really updated contents) to a server side script which will then blindly (not totally, after some validation or authentication for sure) insert those result to our database. Now when A comes and visit his page (A is C’s first degree friend), he will get all the latest result from C’s feeds as B has already done the parsing job for us and we have those latest result from C’s feeds stored in our Database.

There are definitely more challenge than it is explained here, like what if a person is second degree friend of a multiple user. In such cases as we supplied last update time of these feeds while generating a page, we can calculate from server side which feeds we really want to parse.

And moreover, we can add more check for our javascript parser than just blindly parse those feeds. We can download a chunk of RSS or Atom feeds (using a proxy script developped using curl range options) and read just up to the latest update time and extract the time using simple reg ex or string functions instead of downloading full feed data. Now if we know that a script is not uptodate just by downloading 1-2 killobyte of data instead of downloading full feed, parsing the xml data, it would save us more computing resources for performing other jobs.

But of course, you cannot live completely on your client’s machine for parsing all your data. You must have several cron’d scripts to parse left overs or other feeds at server side. But what I am saying is that with little help of javascript you can make use of such a tremendous distributed computing power in your application, all at almost no cost.

I will come with example code once I am done developing my open source clone of FriendFeed and then I am sure, you will find it was worth writing a blog post about.

image courtesy: http://www.naccq.ac.nz/bacit/0203/2004Caukill_OffPeakGrid.htm

Have a nice RnDing time.

How to make your own springloops in PHP

Springloops is a nice code management service recently came into focus. It helps you to manage the code base of your application, monitor the commit and deploy the final version easily to another server. So if you are wondering how to build such a system and how it actually works, this article is for you.

Please note that I am neither way affiliated with Springloops nor any of it’s contacts. This article expresses completely my own opinion.

The primary obstacles of making such a service are
1. Managing the subversion repositories and users
2. Interaction with subversion repositories (Not same as option 1)
3. Payment gateways
4. Code browser
5. User friendliness
6. Scaling

Among these options, 4 and 6 are out of the scope of this article. There are thousands of article explaining those topics to you. So I am not going to talk about them. There are excellent libraries available to manage many popular payment gateways. Just google it and you are done. And about user friendliness and design, you can hire a consultant who’s specializing on this subject and get it done.

1. Managing the subversion repositories and users
In springloops or any other subversion hosting service, you need to do some basic shell scripting which is required for adding the subversion repositories dynamically and to add users in it. There are some excellent articles how you can do it with apache2. To avoid any traffic related issue, best practice will be hosting these repositories in another server, and using htaccess - just point your user’s svn URL to the exact url. Check out the following urls to find out how to host subversion repositories with apache. Linux should be the first choice, heh heh.

Reference:
1. Setting up subversion with apache in Ubuntu gutsy gibon
2. Setting up subversion with apache2
3. Setting up subversion with apache2 and DAV in debian
4. Host your open source projects in ubuntu in 3 easy steps

All you have to do is writing shell script to automate these steps and restart the apache2 demon once an user register his/her project with your service. You are done

2. Interaction with subversion repositories (Not same as option 1)
Now this is quite a challenging part and many of you are lost how to achieve this kind of functionality for such services. For PHP developers, you know there are an excellent repository of extensions named PECL and also repository of libraries name PEAR. In PECL there is an excellent extension which is cent percent appropriate for this work. Yup, I am talking about “SVN” extension. You have to install this extension to use it with your PHP code. Once you are done with installing it, now you got the complete power to interact with the subversion repositories hosted by your users. This extension is very rich and provides all the functionalities you need to interact. For details please point your browser to the appropriate section in PHP manual.

Reference
1. Installing PECL SVN extension for PHP in Ubuntu Gutsy Gibon
2. PECL svn package
3. SVN extension documentation

4. Code Browser
Another challenging part of the total setup. If you cannot provide excellent code browser which will clearly demonstrate the changes in code from different revisions, figure out the log and the notes made during each commit and finally, make note on it to show to your fellow team members.

Using svn_diff, svn_ls and svn_log you can clearly find out the difference of a file in two different revisions, the file and directory structure of the repository and the log of each commit. So basically once you got the file and directory structure of your repository for a revision, you can just traverse through it and display as a nice report using your PHP code. And when user will ask you to display the difference, you will find the difference using svn_diff which returns the difference as standard diff format. Now using regular expression (or whatever way) you will mark the lines which was changed and just display them highlighting using different colors to your user, as a nice report. And You can also browse the log of any revision using svn_log and show it to your user.

And last but not the least, you can create tags, branch and whatever addition by maintaining a local shadow working copy of your repository. Only the thing that you cannot do (or still I am thinking how-to) is merging, heh heh.

And when it comes about deploy the latest code base to your server, you can do it using svn_update where the path is the working directory in your server.

Reference
1. Standard Diff Format
2. svn_diff
3. svn_log
4. svn_ls
5. svn_add
6. svn_update
7. svn_checkout

Basically this is how a service like Springloops is built. I hope, you’ve enjoyed this article as much as I did to write it, heh heh.

WorldTimeEngine - How about making your own in PHP?

I recently came by this site WorldTimeEngine where users can search the local time of any place using the name, street address or just latitude and longitude. Since that time I was thinking how easily you can make your own. As long there are some good people over there (For Geocoding API) - its a not a big deal, you know? Lets have a look at the following PHP code.


<?
$place = “Bucharest”;

$geodata = getGeoCode($place);
print_r($geodata);
echo getTime($geodata['lat'],$geodata['lng']);

function getTime($lat, $lng)
{
$url = “http://ws.geonames.org/timezone?lat={$lat}&lng={$lng}”;
$timedata = file_get_contents($url);
$sxml = simplexml_load_string($timedata);
return $sxml->timezone->time;
}
function getGeoCode($address)
{
$_url = ‘http://api.local.yahoo.com/MapsService/V1/geocode’;
$_url .= sprintf(’?appid=%s&location=%s’,”orchid_geocode”,rawurlencode($address));
$_result = false;
if($_result = file_get_contents($_url)) {
preg_match(’!<Latitude>(.*)</Latitude><Longitude>(.*)</Longitude>!U’, $_result, $_match);
$lng = $_match[2];
$lat = $_match[1];
return array(”lat”=>$lat,”lng”=>$lng,”address”=>$address);
}
else
return false;
}
?>


Changing the variable “$place” to “Bucharest”,”Dhaka”, “Toronto” and “Oslo” gives me the following result


Array
(
[lat] => 44.434200
[lng] => 26.102955
[address] => Bucharest
)
2008-03-01 08:41

Array
(
[lat] => 23.709801
[lng] => 90.407112
[address] => Dhaka
)
2008-03-01 12:42

Array
(
[lat] => 43.648565
[lng] => -79.385329
[address] => Toronto
)
2008-03-01 01:42

Array
(
[lat] => 59.912280
[lng] => 10.749980
[address] => Oslo
)
2008-03-01 07:43


Nice, Huh?

List of RSS Feeds I read almost everyday

I am sharing a list of RSS feeds that I read almost everyday. And which one is my favorites RSS reader? Well, I use GoogleReader because of it’s excellent features like star and feed history. also I like it’s feed sharing feature.

1. Ajaxian : http://www.ajaxian.com/index.xml
2. Cow’s Blog : http://cow.neondragon.net/xml.php
3. IBM Developer Works (Web) : http://www.ibm.com/developerworks/views/web/rss/libraryview.jsp
4. IBM Developer Works (Open Source) : http://www.ibm.com/developerworks/views/opensource/rss/libraryview.jsp
5. Designer Folio : http://feeds.feedburner.com/dezinerfolio
6. Digg Technology : http://digg.com/rss/containertechnology.xml
7. DZone Latest Front Page Links : http://www.dzone.com/feed/frontpage/rss.xml
8. Freelance Switch : http://feeds.feedburner.com/FreelanceSwitch
9. HacksZine : http://hackszine.com/index.xml
10. International PHP Magazine News : http://www.php-mag.net/magphpde/psecom,id,26,noeid,26,.html
11. JSLabs High Performance Web Apps : http://feeds.feedburner.com/jaslabs
12. LifeHack : http://www.lifehack.org/feed/
13. Mashable : http://feeds.feedburner.com/Mashable
14. Maxdesign : http://www.maxdesign.com.au/feed/
15. Newsvine (PHP) : http://www.newsvine.com/_feeds/rss2/tag?id=php&d=v
16. PHP Freaks : http://www.phpfreaks.com/feeds/articles.xml
17. PHP Magazine : http://www.phpmagazine.net/syndicate.xml
18. PHP Coding Practise: http://php-coding-practices.com/feed/
19. PHP Developer : http://phpdeveloper.org/phpdev.rdf
20. PHP Geek : http://www.phpgeek.com/wordpress/feed
21. PHP Architct News : http://www.phparch.com/phpa.rss
22. Planet Ajaxian : http://planet.ajaxian.com/index.xml
23. Planet PHP : http://planet-php.org/rdf/
24. Programmable Web : http://feeds.feedburner.com/ProgrammableWeb
25. ROScripts : http://feeds.feedburner.com/ArticlesAndProgrammingTutorials
26. Sitepoint Blogs : http://www.sitepoint.com/blogs/feed/
27. Sitepoint News : http://www.sitepoint.com/recent.rdf
28. Smashing Magazine : http://www.smashingmagazine.com/wp-rss.php
29. Jonathan Snooks Blog : http://snook.ca/jonathan/index.rdf
30. TechCrunch : http://feeds.feedburner.com/Techcrunch
31. Technorati Javascript Links : http://feeds.technorati.com/search/Javascript
32. Technorati PHP Links: http://feeds.technorati.com/search/PHP
33. Veerle’s Blog : http://feeds.feedburner.com/veerlesblog
34. Web2List : http://feeds.feedburner.com/Web2list
35. Zen Habits : http://feeds.feedburner.com/zenhabits
36. Del.icio.us PHP Tags : http://del.icio.us/rss/tag/php
37. PHPExperts Forum : http://rss.groups.yahoo.com/group/phpexperts/rss
38. 456 Berea Street : http://www.456bereastreet.com/feed.xml
39. Particle Tree : http://feeds.feedburner.com/particletree
40. Simple Bits : http://simplebits.com/xml/rss.xml

Prelude to foundation: Its time to go for a better PHP Framework

I remember those old days when I had to write everything by myself. I wrote huge libraries to work with MySQL. Then I learned PostgreSQL and SQLite but didn’t rewrote my old library to work with those, I was running short of time. So I forsake the opportunity to write a db library which works with them. What I did was plain code relevant to database specific portions. Oh ya, that was a long time ago.

Soon after that I came to know adoDB which made my dream come true. I was so much happy getting my all db specific works done in a much more smarter way. I get rid of database portability issues. I was very happy that time.

I learned smarty soon after I realize that my codes are getting ugly with all the inline HTMLs and PHPs. Nothing could be smarter than separating the presentation logic from the business layer. I am a big smarty fan since that time. It saves my sleep for many nights.

But again I am recurrently suffering from maintainability issues. I was not surprised to find that my code is becoming huge unmanageable giant and it takes huge time for refactoring the application. I was very sad those days. Oh what a disaster that was.

When working with my team members located remote places, I fall into a deep shit. How can we manage and track the changes done by us? Even I was getting strange code in my routine which I bet was not written by me!! It was a terrific job

WordPress Bangla Plugin has been released

Hello

Today we have released wordpress plugin for bengali wordpressians. For reference, take a look at here to know what this plugin actually does.

You can download this plugin from here : Download

WordPress Blogrolls Importer - Opensource

Today I developed this tool to import wordpress blog rolls as XML document. You know when you export data from wordpress.com that doesn’t include the blog rolls data. So if you want to keep a backup of your blog rolls, you can use this tool to import your blog rolls data.. This one is developed using PHP and Curl

You can doewnload it and see the code in action here

Opensource WordPress Blogrolls Importer

################

The trick is lying here –
;

    

    $ch = curl_init();

    curl_setopt($ch, CURLOPT_COOKIEJAR, “./login.jar”);

    

    

    curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);

    //curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);  // this line makes it work under https

    curl_setopt($ch, CURLOPT_URL,“{$url}/wp-login.php”);

    curl_setopt($ch, CURLOPT_POST, 1);

    curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);

    curl_setopt($ch, CURLOPT_POSTFIELDS, “log={$username}&pwd={$password}&&redirect_to=wp-admin/link-manager.php&wp_submit=”);

    

    $buffer= curl_exec ($ch); // execute the curl command

    

phpexperts get-together at 16th september, 2006

It was a great party yesterday we had at somewherein. From this month we started the phpexperts monthly meeting which is sponsored by somewherein this month. Yesterday we were scheduled to start at 3:00 pm. The first who knocked the door were tahsin, najnin and kamrunnahar. Tahsin is the member of phpexperts and other two were her interestd friends. Later Prasun and Biplob came in. Shortly after that we had Mizan bhai, Rinku da, Munaz bhai, murshed, Arafat bhai, hasn, Emon, joyeb, rafik bhai, shoeb bhai, raisul and another friend of rafik bhai.

We introduced somewherein and pageflakes in our meeting. We discussed about the current trends and the market situation. Everyone agree that we should increase the awareness abd professionalism in this field. RInku da and Mizan bhai shows us what is the current situation. Later we had an interesting presentation from Hasan regarding design patterns and then we all started talking about DP. Dp is always useful but unnecessary use of DP may decrease the productivity and increase the cost of project.

After hasan’s presentation we discussed about security features of an web application and I show how sql injection and xss could be vulnerable for your applications.

Mizan bhai and ringku da talked about some interesting issues regarding database optimization.

The most interesting part was after hours of the meeting. We 9 people were still there in some close conversation and we discussed about promoting php in Bangladesh as well as promoting the name of local developers and groups internationally. We all agreed that we should bring forward the name of Bangladesh in this sector. We planned about starting some open source projects. But starting it in large scale from the beginning wouldn’t be that much effective as we see many well planned initiative comes to an end because of communication gap and lack of management. So we planned to start it in small scale – our first open source project will be a “Bangla Chat Solution” – I advanced a bit in planning and I will share it with group from today.

Finally we got a fantastic idea to write a very good bangle book on PHP. Everyone was so happy with this idea that we were surprised – this is a real problem that there is no good bangle book in Bangladesh right now. So if we, the leading php community can start writing one, it would be great obviously. We setup a small group who are interested to write the book. Mizan bhai and I will manage the project and co ordinate the writers. Primarily I, Mizan bhai, Arafat bhai, Munaz bhai, Raisul, Hasan, Murshed and Prasun agree to write. But if anyone of you are interested, you are most welcome to join.

The meeting ends at 8:30 and we are very happy with the decisions taken. We hope that you will join in next meeting and participate with us in these interesting sessions.

Thanks to somewherein for sponsoring this great party.

Working for PageFlakes

I join PageFlakes this month as a Flake Developer. PageFlake team recently won the Ajax StartPage race beating Microsoft and Google. As a result, one of the most successful venture capital company BenchMark capital invetsed $5 million in pageflakes.

PageFlakes has around 18 members here in Bangladesh, one of the greatest developer team I have ever seen. Omar Al Zabir, the CTO of PageFlakes is working and coordinating the BD team from here.

I am enjoying my time with PF. Currently I am developing Graph Flake for PageFlakes which will collect remote data in different format and compose them into nicely viewable charts. I hope I can release it within this week.

Thanks to all in PF for their wonderful effort.

Five Great CMS to vote - Packt Open Source Award

Packt arranged this open source CMS award since july. People could cast vote for their favorite CMS and The top five were supposed to be selected in first round. The first round is over. The following CMS are the top scorer and now you can vote among these five. The top three will be awarded by $5000, $3000 and $2000 respectively.

* Drupal
* e107
* Joomla
* Plone
* Xoops

So goto the above banner and vote for your favorite CMS. Voting will commence on September 12 and will run until November.