PyKata - Bulkuploader Patch

Who stole all that bandwidth? or was it a design accident, was it a python ... ?

That's what happened to Web2Hunter yesterday. Downtime due to quota exceed, I never thought 1 Gb a day would ever exceed... but now I actually have to think about saving every possible bit. Here is my explanation to the downtime of Web2Hunter -

1. I post about Web2Hunter on HN. Drives good deal of traffic to Web2Hunter.
2. Many users throw requests simultaneously, thanks to 5 second ajax loop which never ends for sake of Web2Hunter's simplicity/chaos or randomness.
3. Web2Hunter validates domains using Google, this contributes much less to the outgoing bandwidth, no problem so far.
4. Google gives back html, each roughly 3-4 KB in size. Given a surge in traffic, and the 5 second ajax loop, this means a lot of incoming bandwidth.
5. Again, the response to ajax request is minute (a domain name), so nothing much to do with outgoing bandwidth quotas.
6. So, even if things are ajaxified, the poor design used to suck up lot of bandwidth internally

Solutions - things have been working fine now, and incoming bandwidth quota isn't climbing every 5 minute as it used to do yesterday.

• Ajaxwhois API came out as a life saver. It returns a few bytes of JSON instead of chunk of html, and I don't even need to parse the JSON for it is too simple to make out from it's contents.
• ¶It could be possible for someone to misuse/proxy the domain finding url that ajax loop fetches every 5 seconds. For that I've tried to filter the requesting client by referer information. This would reduce the chances of hotlinking/sucking
  

  
  if (self.request.headers.get('Referer') != None):
  if (self.request.headers.get('Referer').find('web2hunter.appspot.com')!=-1):
     # show your content
  .
  .
  .

  
  
• Images to be converted to jpeg, obviously every bit counts, and we really don't need fancy png without any need of translucency.
• Another one yet to be implemented - since web2hunter does magic of random-combination to get you cool names, one good idea to save some more bandwidth would be to store all the domains that are unavailable so as to not to lookup again for them. Hmmm... but that means we lose out on any domain marked today, if it expires tomorrow. Hmm... I guess this idea would also need monthly truncation/refreshment of domains stored in unavailable list. Not good for maintenance...
  

So, till then I hope Web2Hunter runs smoothly on the Engine without any glitches. So far, even a one hour test has not been able to do anything to 12% incoming bandwidth :)

Web2Hunter - grab those domains before they vanish

Last weekend I was trying to make a more useful app over Google App Engine... at least better than the first one. So here's what I got for you - Web2Hunter, yet another one that brings up some randomized yet organized information to you, thanks to urllib2 and Python in the back. Here is the intro article I wrote for it -

How many times you have been stuck for hours thinking of that new name for your startup? How many times have you come up with that awesome name and you found out that all your imagination was on sale by a domain squatter? Don't worry, for Web2Hunter is here for rescue. It not only saves you from spending extra hours over thinking a new name, but also from the dismay of not finding a domain for the name you liked.

Simply put, Web2Hunter shows you a list of random trendy names which are actually available for purchase. It's about assisting your creativity. It throws up randomness, strange names of any kind. It's fun to watch because it opens your mind to new possibilities when trying to name your service, or even have new ideas.

Let Web2Hunter hunt down some cool names for you before you spend your precious time over a new name. Web2Hunter runs over Google App Engine and tried to be minimalistic. Do you have any suggestions to improve Web2Hunter ?

The original article can be found on The Indian Startup - Tools blog.

Feel free to fork Web2Hunter's source code on github. I also happened to give a short talk on it at PyCon India 2009, which was very impromptu and organized, for all I had to show was code, no slides. Have a look (youtube).