thoughts, ideas, code and other things...

Tuesday, April 27, 2010

Introducing BugBase 1.0

Here's something about what has kept me busy since last week - a college miniproject. The quickiest dirtiest php app I ever wrote. Still incomplete with loads of things to fix, add, etc. Nevertheless - the intro :)

Labels: , , ,

Sunday, April 25, 2010

VRML, Direc!Tax, book, shops, kapoorthala...

Looks like they just picked books from the end of an alphabetical list - VRML, Visual J++. Just reminds me of the time in that Universal bookshop in Kapoorthala, Lucknow when me and @ixpu used to drool at a wall of Computer programming books.
If you asked for a book on DirectX the shop owner usually used to get confuse it for Direct Tax. Wooh boy that was like 8-9 years ago down the line. There was a couple which was correcting each other on right pronunciation of C# telling "Its not See-Hash its See Sharp" and Dipanjan sir or someone said 'This is M$'s answer to Java', etc.

Labels: , , ,

Saturday, April 24, 2010

For each ...

Weird ways I learnt a few things down the line -
  • 'for each' construct from the source code of love worm
  • PHP because writing perl cgi scripts was a pain on the free tripod account, and PHP for all wrong reasons ;)
  • VB - 1st to make pacman, then again for all wrong reasons.
more to come as I recall.


Friday, April 09, 2010

My GSOC proposal for SahanaPy - HTML/JS based reusable frontend for S3XRC

1. Personal Details

  • Name- Abhishek Mishra
  • Communication details -
    Email -
    Freenode IRC Nickname - ideamonk
    Google talk - ideamonk
    Skype - ideamonk
    Twitter - ideamonk
    Blog -
  • I am from a city called Patna in Bihar, India (+5:30 GMT). Currently I am pursuing a B.Tech (a 4 year course) in Amrita School of Engineering, Bangalore. I spend most of my day working on my side projects and cooking new ideas in mind when free from college. In free time and holidays I also work as a freelance web developer with my friends -
  • Resume -

Exposure to similar technologies and/or FOSS in general.

I started relying heavily on open source projects to learn from and use them ever since I switched to Linux about 1.5 years ago. I started learning Python about 1 year ago and it has in a way become my primary language to implement anything that comes to my mind. I have worked on SahanaPy in the past (the current steel theme, custom error pages, a bit of ExtJS work) and am quite familiar as to how it works. Interestingly SahanaPy is the first Open Source project I've contributed to.

Some thoughts over Free/Open Source Software -
Subtle - as they may look, ignore - as many people may do, but at the end of the day, there's no denying that "Open Source is the best infrastructure you can rely on" to - solve complex problems of the world, help people around the world (the way SahanaPy does!), or even to start your business.
I believe FOSS has saved humanity a lot of time by showing them that "Re-inventing the wheel" is not always the right approach to get things right.
Some of my side projects -

“Why would you like to help the Sahana project?”

I was initially introduced to SahanaPy on IRC by a few friends. Later in Dec 2009, I got a chance to participate in SahanaPy workout at I believe that I could put in my experience in developing Web Applications very well into S3XRC frontend and help developers deal with compound results in an easier, uniform, organized way.
Besides, to the common question that people ask "How would a software help in disaster?", Sahana stands as the best example you could give them, given that it has actually been used at many incidents. The recent support shown to Haiti crisis by Sahana foundation is something that would inspire us to do good at all times. I believe I could too make significant contribution to such a great endeavour.

2. Personal Availability

  • Yes I have reviewed the important dates and timeline of GSOC 2010
  • Apart from my institute's end-semester exams that fall from April 30th to May 13th, I have no other conflicts with the schedule.
  • Will you need to finish your project prior to the end of the GSOC?
    Yes, I would take utmost care to finish it before GSOC ends so that developers could start utilizing it fruitfully by then. Though S3XRC frontend would always have my support post-gsoc too in case any issues/bugs come up.
  • I would be available for entire duration of GSOC 2010 starting from May 14th when my exams end.

3. Project Abstract

Currently Sahanapy uses two backends for REST - S3XRC (XML+JSON interface of the restful API) and CRUD (Web2Py crud for html interfaces). Web2Py crud is good at a few things e.g. client side validation, ease of use, but it also has limitations, i.e. It can access only single table at a time.

On current version of SahanaPy this is evident when you take a look at [1] and compare it to [2]

While [1] is generated by S3XRC, it contains nested resources, not only do you get hospital details, but also current status of beds.
On the other hand [2] generated by web2py is not able to do anything worthwhile to this information.

Unfortunately S3XRC has no HTML front-end yet. The backend is able to handle XML and JSON but web2py front-end (SQLTABLE and SQLFORM) are not able to handle the complex resources.

A good way out would be to replace them with new handlers - XRCTABLE and XRCFORM

Which would understand and represent or interpret S3XRC's XML/JSON output appropriately. After that jQuery/ExtJS based web interface would be made which would be more responsive and would be able to take advantage of JSON outputs of S3XRC.

4. Project Plan

This section is to provide the detail of your project proposal. Take as much space as is necessary.

  • Project Deliverable -
    • The ability to present the complex resources returned by S3XRC to the user.
    • A more responsive jQuery/ExtJS based UI that would change less, and actively load data through AJAX
    • We would be able to save good about of bandwidth, would be able to work in low bandwidth conditions when data gets passed in JSON format.
    • Since aimed to be written in a generic and reusable way, this would be reused in other parts of code too.
    • Besides this work would also reduce the number of requests, which implies quick data entry at times of disaster management work.
  • Project Justification -
    • The number of useful inter-related data we wish to retrieve and represent would definitely increase in time. This calls for a way to represent then to the user in a RAD fashion.
    • One can never be sure about availability of bandwidth at sites of disaster. At times one might have to work in very low bandwidth conditions. The reduced size and frequency of requests that the S3XRC frontend would bring, will help rescue workers easily operate SahanaPy in such conditions.
  • Implementation Plan -

    A sample compound resource -

    <s3xrc success="True" results="1" domain="<span style=" class="J-JK9eJ-PJVNOc">rocksolid</span>" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">url</span>="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">sahana</span>" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">latmin</span>="-90.0" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">latmax</span>="90.0" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">lonmin</span>="-180.0" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">lonmax</span>="180.0">
    <resource name="<span style=" class="J-JK9eJ-PJVNOc">hms</span>_hospital" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">uuid</span>="eb903352-0a6c-4d3c-bb8c-f18f6b7a3179" created_on="2010-04-02 15:14:07" modified_by="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Abhishek</span> <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Mishra</span>" created_by="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Abhishek</span> <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Mishra</span>" modified_on="2010-04-02 15:14:07" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">url</span>="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">sahana</span>/<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">hms</span>/hospital/1">
    <data field="website"></data>
    [[ snip snip ]]
    <data field="info_source"></data>
    <data field="phone_exchange"></data>
    <resource name="<span style=" class="J-JK9eJ-PJVNOc">hms</span>_<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">hrequest</span>" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">uuid</span>="c9aa43df-11fc-4b63-977f-d56abaefd1cf" modified_by="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Abhishek</span> <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Mishra</span>" created_by="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Abhishek</span> <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">Mishra</span>" created_on="2010-04-02 15:14:58" modified_on="2010-04-02 15:14:58" <span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">url</span>="<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">sahana</span>/<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">hms</span>/hospital/1/<span style="background: none repeat scroll 0% 0% yellow;" class="J-JK9eJ-PJVNOc">hrequest</span>/1">
    <data field="city"></data>
    [[ snip snip ]]

    ^^ Notice the nested
    • Approach 1 -
      Addition of new view generators for complex resources as shown above, namely XRCTABLE and XRCFORM to SahanaPy. Using these to generate ExtJS code that would do the interface. We already use ExtJS a lot, so this new front-end would be more generic and fast.
      So, this would be an ExtJS based front-end with S3XRC as backend.

    • Approach 2 -
      Exploiting S3REST pre and post hooks. When one calls a REST controller, the following happens --

      1. REST parses the original server request into an XRequest object (jr)
      2. XRequest contains hooks to export / import directly in XML or JSON Eg. jr.export_xml()
      3. After parsing a pre-processor is called - which is dynamic, i.e. a controller can set the pre-processor as response.s3.prep
      4. This means we can pre-process the data before REST action is executed, and even change the REST action itself. Eg. s3rest.set_handler ("foo", my_foo_handler)
      5. If the Pre-processor returns False, then the REST action is not taken at all. This means we can manipulate the flow as we wish to.
      6. At last step, a post-processor is called, which receives both the output of the method handler and the XRequest object. This can be set using response.s3.postp
      7. At this stage we can generate the HTML and ExtJS code required by the views.
      8. The view then builds the generated HTML and ExtJS code into the HTML template.

      An optimal approach is to have the ExtJS objects pre-loaded, and retrieve data to them as needed. XRequest supports server-side validation and responds in Ext-compliant JSON. This could be very ideal for a full-fledged Ext-Based interface.

    Another point to be taken care of while developing this would be to make it re-usable, generic and easy to customize. For eg. An update-data popup for entries listed in a table could be used where-ever update kind of operation needs to be done, and so on.
  • Future Options -
    • Adding usability support for browsers that don't have javascript support enabled. This could be either done by making the interface fall back to web2py crud or having an alternate non-js interface rendered by s3xrc
  • Relevant Experience -
    • I have contributed earlier to SahanaPy -
    • Some of the SahanaPy related work I did -
      • creating the current steel theme, - gives me a good understanding of js, css and views used in sahanapy.
      • improving the main and sub menu (created css based crossbrowser compatible menus),
      • javascript based popup for coordinate conversion and, - I've used ExtJS here for coordinate conversion popup.
      • adding custom error pages - involved tweaking of web2py framework
      • ( )
    • I have worked with MVC architecture in the past ( CodeIgniter, ASP.NET MVC) and even while working for SahanaPy. Recently started learning Web2Py -
    • I am quite comfortable with Python, jQuery, writing good HTML & CSS and have basic idea of ExtJS.

Other references -

5. Project Goals and Timeline

  1. Work already undertaken -
    • Reading more about S3XRC, S3REST and drafts of S3XRC_2
    • Familiarize with web2py
  2. First trimester (26 April - 23 May) - Community bonding
    • Low activity till 13th May due to college exams.
    • As S3XRC work is also related to ucore, I would spend good time before I start coding in learning it more indepth.
    • Creating a list of all things that could be made generic with S3XRC frontend. Right now I'm aware of tables and forms, yet inside forms we could provide a generic way to make them interactive as seen in a work by Michael -
    • Researching some more possible generic widgets that could be implemented in S3XRC frontend and reused later.
    • Collecting feedbacks from developors as to what kind of tasks are restricted due to current web2py crud.
  3. Second trimester (24 May - 12 July) - Coding starts
    • Basic XRCTABLE 24th May to June 13th
    • ExtJS interface to XRCTABLE to provide on the spot update/add feature - June 14th to June 27th
    • XRCFORM as a replacement to existing SQLFORM - June 28th to July 12th
  4. Third trimester (13 July - 9 August) -
    • ExtJS interface for XRCFORM elements to reduce number of requests, and to provide inline edits, assistive popups, etc - July 13th to July 20th
    • Weekly implementation of widgets and generic elements decided in First Trimester - July 21st to Aug 7th
    • Rigorous testing and bug fixes - Aug 7th to Aug 9th

Labels: , ,

Sunday, April 04, 2010

PyKata - Bulkuploader Patch

Labels: , , ,

Thursday, April 01, 2010

GSoC 2010 - PyKata - Proposal

Taking PyKata to another Level

developing a useful supplement to Python in education

Contact Information

  • Name - Abhishek Mishra
  • Email Address -
  • Phone Number - +91-9535009187
  • IRC nickname - ideamonk (
  • Twitter -
  • Skype username - ideamonk
  • Webpage/Blog -
  • College/University - Amrita School of Engineering, Bangalore
  • Major - B.Tech in Computer Science

Project Details

  • Project Description - Help develop PyKata - the website for Python practice ( We want this to be the prime site serving students, teachers, and professionals wanting to learn Python via self-study.
  • Mentor - Andy Harrington and David MacQuigg
  • Key Technologies - Python, AppEngine, Web2Py/Django

What is PyKata?

PyKata is a community-authored website with contributions from teachers and students in math and science, as well as technical professionals. It provides you a way to quickly write programs on the web, get immediate feedback on errors, and work through a set of exercises designed by the best math and science teachers all over the world. For teachers, who may not be programmers themselves, PyKata? offers a way to integrate simple programminmg into their lesson plans, without distracting from the subject being taught.
Website -
Vision -
Code - , currently based on webapp+django
PyKata uses similar doctest environment as seen in Crunchy.

What features are provided currently?

1. It lets you write, execute and automatically evaluate your code online.
2. Saves your work-in-progress.
3. Shows your overall progress as to problems solved.
4. Lets community contribute problems.
5. Static help

Why should PyKata be developed further under GSoC '10?

Python has been a liberating force for both programmers and for scientists and engineers who don't have time for traditional programming. It has a also added some fun and playfullness to programming.
I believe Python is the best when it comes to teaching programming at school and college levels. and personally while discussing python with some friends, I've noticed a fair amount of excitement they've shown towards Python.
PyKata has great potential of making these ideas popular and assisting in quick learning for say, college students who wish to do quick simulations, or say, new engineers on the team who do not wish to spend excessive amounts of time on learning to write programs.

What features am I planning to implement in PyKata?

I am very excited to see a Python+AppEngine task on PSF's Summer of Code Page. Here are some of the ideas that I've come up with after discussions on the PyWhip mailing list.

  1. Porting to Web2Py

    After some experiments and discussions over choice of framework, it seems that web2py is a suitable framework for PyKata development. Django happens to have steep learning curve for new comers who would contribute to PyKata in future. To keep the friction in entry of new developers low, Web2Py seems like an appropriate framework for quick contribution and RAD development.
    Though it is said that web2py follows some anti-patterns, but then "Practicality beats Purity" at times. Also deploying a web2py app on AppEngine is almost effortless.
  2. Porting Python 3

    Right now, Google App Engine uses Python 2.5.2. But there are two interesting things related here -

    • opposed to the propaganda on #python about not using Python 3 yet (, the Python community is more interested in porting existing packages to Python 3
    • This discussion on web2py and Python 3 compatibility is a relief as the core developer believes that it won't take too much to port web2py to Python3.
    Instead of waiting for App Engine to shift to Python 3, I would create a Python 3 compatible branch of PyKata in parallel. Once App Engine shifts to Python 3, PyKata too would get updated in no time. This would also help us give web2py developers useful feedbacks on porting to Python 3.
  3. UI/UX improvements

    PyKata could benefit from an improved design (more web2.0-ish). This would not only make it attractive but also help us present the progress data, user profiles, live statistics, tutorials, news, etc in a better way as it would encourage more social interaction upon growing. A rough mock-up -

    I aim to create an addictive and interactive user experience. Using game style interaction to drive participation, possibly include points, badges, rewards a great example for that would be

    Creating a flow - once a student gets a problem right, PyKata would show her which way to proceed from there.

    If a student goes wrong repetitevely, PyKata could also show him/her possible tutorials related to it.

    Some examples -

  4. User Profiles

    PyKata can provide each user his/her own personal profile to track progress. I also plan to add two levels of views to these profiles - a personal view, a public view. Someone whome the student designates as a teacher will have full access to the student's data. Currently the only way to track progress is to either look at ticks on 'All problems page' or to do the same on category pages. A mockup for this feature -

    Another important detail that a user profile should expose, when a teacher views a student's profile would be the areas where one is weak at. This could be found out by monitoring number of times a student fails in a problem or a class of problems. While the public view would not show any such details.
  5. Live execution results

    Currently PyKata takes you to another page to show results of current code execution. Then the user hits back button to modify the code, or copy-pastes the traceback for reference and hits back to debug the code. This user experience could be simplified by placing a live result box on right side of the editor. As shown here -
  6. Content Management

    PyKata relies heavily on user submitted content. As for content submissions, all new submissions would get reviewed by PyKata editors and then moved to appropriate categories. If that doesn't happen, then a problem remains in 'new' category for the time it accumulates favourable votes.
    As per current plan, only a PyKata contributor would be able to modify the problem/content they submitted. Another group of users - PyKata editors would have the rights to modify problems and content for the purpose of moderation.
    When a user proposes improvements to a problem, the author of the problem would have the right to decide whether to accept it or not.
    As per modifications to problems, the system would preserve previous version and show the history of the problem. This would be useful for Task 9 - Course planning feature.

  7. Batch upload problems/ solution-uploads

    A contributor of problems/tutorials might have many contents already written in say, a python script itself. Copy pasting from them and uploading one by one might not be a great user experience. I would simplify this task by allowing flash/javascript based bulk uploads (to select multiple files). A server-side validation and processing would be provided to such uploads which would process them according to rules specified on PyKata Project Wiki -
    Good news is that I've already written an experimental bulk uploader for PyKata ( ), extending that to above mentioned feature should be easy :)
  8. Export/Import of Data

    This involves creating a way for people to run a local copy of PyKata on their infrastructure. As such it would be nice for them to be able to export and import problems/content from main PyKata website to their local copies.
    Here the batch upload code can be easily reused as PyKata already specifies an upload format. This task would involve generating uploadable file in PyKata specified format. The user then can take it to any other instance of PyKata and merge the updates.
  9. * Course planning feature

    It is expected that PyKata once launched and contributed to, would become a large repository of Python learning problems. It could also include support for classes in Maths, Science and Engineering.
    We could do something interesting here by allowing teachers to create lessons by handpicking existing problems and tutorials and making a definite learning track or a roadmap out of them. This would also reduce the efforts that go on a teachers behalf in creating tutorials.
    A problem editor would mark an edit with "major change" for the purpose of maintaining a history of a problem. This is useful in maintaining consistency in already created learning traks prior to modification.
  10. * PyLint / Pyflakes integration

    Though not sure if it would be possible to run them over AppEngine, I would experiment with it and try to integrate it as well to PyKata. If this feature gets added, PyKata would be able to pinpoint indentation and other minor errors immediately. Crunchy's main developer Andre Roberge suggests that this could be done in a similar way as seen in Crunchy.
  11. * Community Ratings

    This involves creating a way for the community to rate contents - including users, problems, tutorials. This would help us gather good statistical data for further analysis. (this is a non priority task)

  12. * Making PyKata more Social

    Though not at all a priority task, and not even a tough thing to do. At a later finishing stage, it would be nice to have integration to social networks like Facebook, twitter, etc to gain more popularity. Though not at all a concern right now.
    A good way to encourage participation, as suggested by Andrew, would be to expose different users. Different user types - learners, one-by-one contributors, experts, content contributors, admins, developers.
    I believe a 'karma ranking' (as seen on Launchpad, Hacker News, Reddit, etc) or community rating approach to classify users would be great. This would encourage already contributing members to contribute more.
    Problems can also be rated in this manner - "XX out of YYY people found this tutorial useful"
  13. Documentation

    Though not a separate task at all, documentation will be written in parallel with code development. This will be sufficient that a student not familiar with web development will be able to take over as maintainer of PyKata. One of our mentors, Dr. MacQuigg, is experienced in documentation of commercial software, and he will assist with this effort, so that our "lead developer" will not be slowed by documentation problems in future.

Future ideas - a canvas based whiteboard tool for teachers to create better tutorials,

Having mentioned everything that came to my mind for PyKata's improvement, I would like to point out that not every feature might be possible to be implemented within the time frame. Hence I've kept feature 9, 10, 11, 12 marked with a (*) as tentative or post-gsoc plans for PyKata if they don't fit in.
** Feature 9
would possible to implement once all other feaures get completed as it would require all other features to be functional.

Personal Availability

  • I have reviewed the major dates of GSoC
  • Apart from my Institutes end-semester exams that fall from April 30th to May 13th, I have no other conflicts with the schedule.
  • I would be available for the entire duration of GSoC 2010

Major Milestones (weekly plans)

  1. Community Bonding Period ( April 26 - May 23 )

    April 30th - May 13th - Low action due to exams

    May 13th - May 23rd - Start studying existing codebase, old discussions for getting as familiar as possible with the project. Getting familiar with different aspects of web2py and AppEngine with possible experiments.

  2. First Semester ( May 24th - July 11th )

    May 24th - May 30th - Task1 - Porting existing codebase to web2py.
    May 31st - June 6 - Task4 - User Profiles
    June 7 - June 13th - Task5 - Live execution results (AJAX based)
    June 14th - June 27th - Task6 - Content Management
    June 28st - July 4th - Task 7 - Batch Uploads
    July 5th - July 11th - Task3 - UI/UX Improvements (partial) and testing+review for mid term submission

  3. Second Semester ( July 16th - August 16th )

    July 16th - July25th - Task 8 - Export Import of data
    July 26th - Aug 8th - Task9 - Course Planning feature
    August 9 - August 16th - Task3 (completion), Code Review, documentation wherever necessary, testing.

* Each week is a 40 hour work week.
* Since my GSoC work would bring in a lot of changes to PyKata, I will continue as a committer and will support and evangelize PyKata.

Source Code Repository and Communication Plan

I would be working on SVN repository of PyKata on Google Code - and I already have a committor access there. Most probably I would push my changes to test branch which could be merged to trunk after a simple review/test.

I plant to communicate quickly as for any doubts and discussions coming in between the week. In the end of every week, I plan to submit a report to my mentor. As per my schedule, I'm mostly implementing 1 feature per week, with an exception of two weeks to Content Management feature and Course planning feature. So a weekly report would be well suited.

Why me?

As you can clearly observe I've been doing odd small things here and there, Google Summer of Code and PyKata really brings me an opportunity where I can create, give support to, be responsible for something that a lot of people would use and be proud about it.

Python is bound to play a big role in education, rapid prototyping and what not. At PyCon India 09, I was really happy at the announcement of FOSSEE (Free Open source Software for Science and Engineering Education) group about introducing Python to engineering colleges in India. And now I'm delighted at the fact that I too might get a chance to contribute to Python Education in a big way!



Labels: , , ,