General Purpose AMT webapp based on web2py

I wrote in my previous post about some scripts I developed to make interacting with Amazon Mechanical Turk a bit easier from the command line. I didn’t talk much about the web2py + Google AppEngine piece that actually serves the ExternalQuestion HITS. I realized after talking to another PhD student that people might be interested in a way to host general purpose HITS for free. My web2py application does that.

Getting it running is pretty simple. Sign up for an AppEngine account, install their SDK, install web2py into the SDK, and copy my application (really a directory) into your web2py installation. Use the development webserver included with the SDK to test that your installation is sane. Push it to google to make sure your account is working right. Once you have it running on appspot, install the Amazon Command line tools, install my wrapper scripts to make them behave better, and run the test experiment on the sandbox to verify that AMT + GAE + web2py are talking nicely in the clouds.

Once that’s done, defining your own rubrics for HITS is pretty easy:

1. Define a new controller to handle your custom rubrics. Example: grade6.py to hold grade6_test2_problem35_version1
2. Copy the provided method template and modify it to handle a given rubricCode. The template builds a SQLFORM.factory to present the questions to Turkers and then validate the form input. Once the form is accepted, the method processes the result (scores it) and forwards it to a generic method to write it to the GAE datastore and sends it back to AMT. Example: def grade6_test2_problem35_version1() ....
3. Copy the provided view template and modify it to ask the question you want for a given rubric code. The template extends a view class that knows how to display an informed consent in preview mode and track whenever a Turker clicks on a form element. It uses standard web2py tricks to protect you against injection and give you form validation for free. Example: grade6_test2_problem35_version1.html
4. Prep the HITS by using my scripts to cross the rubricCode with the image (or whatever) you want to display. Run it on the sandbox and test it out. Promote the experiment and run it on production when you are ready.

Okay. Well maybe the setup doesn’t look easy, but a complete definition of a HIT is just over 125 lines of code including comments. That’s not really that bad. It’s a heck-of-a-lot easier than trying to put an ExternalQuestion together from scratch. If the internet is interested, I’ll clean up the application code (read: remove the parts pertinent to my research and IRB restrictions) and post it on GitHub. Leave a comment or send me mail if you are interested.