Amazon’s Android App Store has the concept of a Free App a day. The idea being, if you check their marketplace every day, they’ll reward you with the opportunity to get an app that’s normally paid for free. However, this of course requires you to remember to check it each day. Something I’m rather bad at. What I’m not bad at however is visiting my RSS Reader of choice at least once a day. So the only question then is how to get the free app into an RSS format. (Spoiler Alert: You can find the RSS page at http://rss.dougwarren.org/AmazonFreeAppFeed/)

Django to the Rescue

This gave me the opportunity to try out and showcase some technologies I’ve been wanting to use for awhile. In a previous project I parsed HTML with StoneSoup, but it’s not very well supported and lxml.html has been benchmarked[¹] as being superior in every way. I’ve also talked a lot to co-workers about how easy virtualenv keeps your Python code separate and gets rid of dependency conflicts. And I’ve been meaning to look into celery for asynchronous task resolution. (Particularly, I liked the idea of using celery as a cron[²] as I didn’t like django-command-extension’s runjobs system. However, for the sake of finishing this in a single evening, I cut that last dependency. I left the code in place to come back to it in the future, but for now this is powered by cron.) Finally of course, it would be my first public facing Django project. Albeit very trivial and minimal.

Installing Software

[dwarren@thebigwave ~]$ cd /home/dougwarren/
[dwarren@thebigwave dougwarren]$ mkdir rss
[dwarren@thebigwave dougwarren]$ cd rss
[dwarren@thebigwave rss]$ virtualenv -p /usr/local/bin/python2.7 --no-site-packages .
[dwarren@thebigwave rss]$ . bin/activate
(rss)[dwarren@thebigwave rss]$ vi requirements.txt

The (rss) on the path is a reminder as to what virtualenv environment is currently running. In my bashprofile I have several aliases that will call deactivate before activating another virtualenv so I can quickly switch from environment to environment.

Add into the requirements file the following projects:

django
django-extensions
requests
lxml
ipython

Now I’ll install the software and set up our version control system. I’ve been using git lately, but I may spend some time with Mercurial soon to get a better understanding of the pros and cons of each.

(rss)[dwarren@thebigwave rss]$ pip install -r requirements.txt
(rss)[dwarren@thebigwave rss]$ git init .
(rss)(master) [dwarren@thebigwave rss]$ vi .gitignore

Into the .gitignore file I’ll add the list of files and directories that should not be added to source control:

bin/
include/
lib/
db/
migrations/
share/
*.pyc

Starting out with Django

Next, now that there’s a base of the project, I’ll check in what’s there and start a new project called ‘apps’. This is actually a point that I wish could be different. From above, I have a directory structure that looks like:

/home/dougwarren/rss
                /bin
                /lib
                /src
                /share

Now Django-admin won’t start a project in an existing directory, so I have to have another subdirectory off of rss for the project. I’d rather install the project in the rss directory and have the apps off of it. If anyone knows an easy way to accomplish this let me know!

(rss)(master) [dwarren@thebigwave rss]$ git add .gitignore requirements.txt
(rss)(master) [dwarren@thebigwave rss]$ git commit -m 'initial commit'
(rss)(master) [dwarren@thebigwave rss]$ django-admin.py startproject apps
(rss)(master) [dwarren@thebigwave rss]$ cd apps
(rss)(master) [dwarren@thebigwave apps]$ django-admin startapp amazonfeed
(rss)(master) [dwarren@thebigwave apps]$ chmod a+x manage.py
(rss)(master) [dwarren@thebigwave apps]$ mkdir db
(rss)(master) [dwarren@thebigwave apps]$ git add * amazonfeed/*
(rss)(master) [dwarren@thebigwave apps]$ git commit -m 'initial django baseline'
(rss)(master) [dwarren@thebigwave amazonfeed]$ vi settings.py

Exploring Django projects and apps

The previous commands have made a new directory off of rss ‘apps’ inside of this is all of the Django files for the site that I’m creating. Of particular note is manage.py and settings.py. Manage.py is a python script that will be used to interact with the Django internals. One of the first steps I take is making it directly executed because it’s a lot easier to be typing ./manage.py than python manage.py all the time. Settings.py is used to specify django-specific settings for this installation. In particular I’m going to be making the following changes:

# Django settings for apps project.
from os.path import abspath, dirname, basename, join

DEBUG = False
TEMPLATE_DEBUG = DEBUG

ROOT_PATH = abspath(dirname(__file__))
PROJECT_NAME = basename(ROOT_PATH)

ADMINS = (
    ('Doug Warren', 'rss@dougwarren.org'),
)

MANAGERS = ADMINS

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': join(ROOT_PATH, 'db', 'rss.db'),# Or path to database file if using sqlite3.
        'USER': '',                      # Not used with sqlite3.
        'PASSWORD': '',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
    }
}
...
INSTALLED_APPS = (
    'django.contrib.sites',
    'amazonfeed',
)

Two words of note here. First, I always try to make my projects as relocatable as possible. As such, I never hardcode a path if I can avoid it at all. The use of basename() and join() will enable me to make a development copy of the same project on the same machine just by checking it out of git. For one of my other projects I run 3 different versions on the same VPS the only difference is the Apache config (See below for an example.)

Now, for the second point I’m going to contradict the first. The django sites app has a default app called ‘example.com’ and the Django syndication code uses the sites app to get the atom link. So it will need to be set to the proper URL. I’ll handle that after the database has been created.

Adding Model Data

The previous commands have created the app amazonfeed, I’ve added it to the INSTALLED_APPS list so Django knows it exists, and I’ve even defined a database that Django can use. Now to add the code that will describe the tables in the database.

(rss)(master) [dwarren@thebigwave apps]$ cd amazonfeed
(rss)(master) [dwarren@thebigwave amazonfeed]$ vi models.py

from django.db import models
from django.db.models import ForeignKey, DateField, CharField, URLField, TextField

class Vendor(models.Model):
    name = CharField(max_length=128)

    def __unicode__(self):
        return self.name

class App(models.Model):
    name = CharField(max_length=128)
    url = URLField(verify_exists=False)
    image_location = URLField(verify_exists=False)
    vendor = ForeignKey(Vendor)
    description = TextField()
    date = DateField(auto_now=True)

    def __unicode__(self):
        return self.name

I’ve kept things fairly simple here but still somewhat normalized. The main entry of the Free App will be the App, and I’ll keep track of it’s name, the URL where it can be found, the image shown for the app, who makes it, what does Amazon have to say about it and when was it last seen. The vendor data will live in another table so if one vendor is blessed enough to have multiple free apps of the day I won’t duplicate that data. I probably should have taken the App ID out of the URL field, but I’m not certain at this point how portable that will be in the future. I also originally thought of writing my own markup (and hence taking the image_location) but at this point I just display the description from a separate page. The date field is the date that the App was last seen by the scraper, so anytime that the model saves the date will be updated. Finally for both the vendor and the app itself, the __unicode__ special method which is used to display a representation of the object will simply return the name.

Now that the database models have been described, I can create the actual database. this is done through the manage.py script discussed previously. ‘syncdb’ will scan through all of the models in the INSTALLED_APPS and create tables for them. If tables already exist it will not alter them. There is a standard app called South that will handle migrations for you however. (This is a slight failing of Django in my opinion, in that we’re in the 2nd decade of the 21st century and database migrations aren’t considered part of the core project still.)

(rss)(master) [dwarren@thebigwave amazonfeed]$ cd ..
(rss)(master) [dwarren@thebigwave apps]$ ./manage.py syncdb

I mentioned previously that the sites database will contain the example.com entry. I’ll need to change that to be the site where the app is running from or some of the fields won’t come out right. In developing other Django apps I’ve frequently had to delete and restart the database. As such I prize repeatably. For this next task I could have specified a fixture to add a 2nd site ID, and updated the settings.py entry for SITE_ID to be 2. Instead though I’ll have a small script that will change the existing SITE_ID 1:

(rss)(master) [dwarren@thebigwave apps]$ vi sites.py

from django.contrib.sites.models import Site
my_site = Site.objects.get(pk=1)
my_site.domain = 'rss.dougwarren.org'
my_site.name = "Doug's RSS Feeds"
my_site.save()

And now to execute it I turn back to manage.py which using it’s shell subcommand let’s me execute python code with all of the paths correctly set. (See below for doing so outside of manage.py)

(rss)(master) [dwarren@thebigwave apps]$ ./manage.py shell < sites.py
(rss)(master) [dwarren@thebigwave apps]$ cd amazonfeed
(rss)(master) [dwarren@thebigwave amazonfeed]$ vi tasks.py

Scrape the Page

As mentioned in the introduction, I originally intended to use celery as a crontab, and hence the name of the scraper being tasks. I may still do so in a followup. However for now it will just contain a single function that will be run when the script is run:

from lxml.html import fromstring, tostring
import requests
import re
from amazonfeed.models import Vendor, App

free_app_location = 'http://www.amazon.com/mobile-apps/b/ref=topnav_storetab_mas?node=2350149011'

def getfreeappdata():
    """ Get the data on the current free app and insert it into the database """
    r = requests.get(free_app_location)

    amazon_url = 'http://www.amazon.com{0}'
    html = fromstring(r.content)

    app_html = html.cssselect('span.fad-widget-footer-title a')[0]
    app_name = app_html.text
    app_url = amazon_url.format(app_html.get('href'))

    vendor_html = html.cssselect('span.fad-widget-footer-vendor')[0]
    vendor_name = re.sub('by ', '', vendor_html.text)

    image_html = html.cssselect('div.fad-widget-large-artwork img')[0]
    image_location = image_html.get('src')

    description_request = requests.get(app_url)

    description_html = fromstring(description_request.content)
    description = description_html.cssselect('div.aplus')[0]

    # Create Django objects
    vendor = Vendor.objects.get_or_create(name=vendor_name)[0]

    # Check to see if this app already exists
    app_query = App.objects.filter(name=app_name)

    # Update the time on the current app
    if app_query.count() != 0:
        app = app_query[0]
    else:
        # Or create a new one
        app = App(name=app_name,
                url = app_url,
                image_location=image_location,
                vendor=vendor,
                description=tostring(description),
                )

    app.save()

if __name__ == '__main__':
    getfreeappdata()

The code is fairly self-explanatory. The first third is getting the page from amazon, the middle third concerns isolating the variables that I care about, and the final portion creates the database objects. A few things to note though. First, there’s absolutely no error detection or evasion going on. Scrapers are nasty dirty things. If the site changes too much it will fail. And when it does fail, I want to be notified in an E-Mail right away. I don’t want to try to recover from the unexpected here, the unexpected is that a 3rd party changed the feed from under me and it’s not going to be recoverable. Second, I’m looking forward a bit to the time when Amazon lists the same app a second time. When that happens the date field will get updated and things will progress.

The next step is getting this task to be run at a scheduled time, so I’ll add it to the crontab.

(rss)(master) [dwarren@thebigwave amazonfeed]$ crontab -e

VIRTUAL_ENV=/home/dougwarren/rss
PATH=/home/dougwarren/rss/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/dwarren/bin
PYTHONPATH=/home/dougwarren/rss:/home/wyntersoft/dougwarren/apps
DJANGO_SETTINGS_MODULE=apps.settings

0 1 * * * python /home/wyntersoft/rss/apps/amazonfeed/tasks.py

The only thing to note here is that the PYTHONPATH specifies both the apps directory and the root of the virtualenv. It seems odd to me that both are required, but they are. (I would think just the apps directory should be sufficient.) I specified the cron job to fire at 1AM PST as it seems that the free apps rotate at midnight PST.

Django Feed

Now that the model has created the database, and the script has populated the database, it’s time to read the database and output an RSS feed:

(rss)(master) [dwarren@thebigwave amazonfeed]$ vi feeds.py

from django.contrib.syndication.views import Feed
from amazonfeed.models import Vendor, App
from amazonfeed.tasks import free_app_location

class AmazonFeed(Feed):
    title = 'Latest Free App of the Day'
    description = 'Latest Amazon Free App of the Day'
    link = free_app_location

    def items(self):
        return App.objects.order_by('-date')[:5]

    def link(self, obj):
        return free_app_location

    def item_link(self, item):
        return item.url

    def item_title(self, item):
        return "{0} by {1}".format(item, item.vendor)

    def item_description(self, item):
        return item.description

This will return the 5 latest Apps sorted by date descending along with where they can be gotten from, and the description of the App. You’ll note in the title the __unicode__() representation is being used to print both the App and the Vendor. The next step is to map it into the url scheme:

(rss)(master) [dwarren@thebigwave amazonfeed]$ cd ..
(rss)(master) [dwarren@thebigwave apps]$ vi urls.py

from django.conf.urls.defaults import patterns, include, url
from apps.amazonfeed.feeds import AmazonFeed

urlpatterns = patterns('',
    (r'', AmazonFeed()),
    (r'AmazonFreeAppFeed/$', AmazonFeed()),
)

I’ve published /AmazonFreeAppFeed/ as the canonical URL for the project, but for now I’m also allowing / to access it as well. Mostly because I wanted to do this entire project without having to define any templates or write any HTML.

At this point everything is done from the Django end. The only thing left is to get the webserver to serve the pages. I use apache, and I have a small WSGI template that I replicate over and over:

(rss)(master) [dwarren@thebigwave rss]$ sudo vi /etc/httpd/conf/httpd.conf

<VirtualHost *:80>
        DocumentRoot "/home/dougwarren/rss/apps"
        ServerName rss.dougwarren.org
        Alias /static/ /home/dougwarren/rss/apps/static-final/
        WSGIScriptAlias / /home/dougwarren/rss/apps/rss.wsgi
        WSGIProcessGroup wynter
</VirtualHost>

I set a wildcard DNS entry on the dougwarren.org domain to the address of my VPS. As such, whenever I wish, I can add new hostnames to Apache and start serving content from it. The last thing to do is to set up the wsgi that was specified above:

(rss)(master) [dwarren@thebigwave apps]$ vi rss.wsgi

import sys
import site
import os

cur_path = os.path.dirname(__file__)
sys.path.append(cur_path)
base_path = os.path.abspath(os.path.join(cur_path,".."))

sys.path.append(base_path)
prev_sys_path = list(sys.path)

# add the site-packages of our virtualenv as a site dir
site.addsitedir(os.path.join(base_path,'lib','python2.7','site-packages'))
site.addsitedir(os.path.join(base_path,'src'))

# reorder sys.path so new directories from the addsitedir show up first
new_sys_path = [p for p in sys.path if p not in prev_sys_path]
for item in new_sys_path:
    sys.path.remove(item)
sys.path[:0] = new_sys_path

# import from down here to pull in possible virtualenv django install
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = 'apps.settings'
application = WSGIHandler()

This wsgi file is similar to the changes made to settings.py or the crontab, it’s based off of a snippet I found on-line somewhere, but I lost the attribution at some point. If you know where it’s from please let me know so I can update it. Again no actual paths are specified everything is relative to where the wsgi file is specified.

The only thing left to do is restart Apache, and commit the changes!

(rss)(master) [dwarren@thebigwave apps]$ sudo /etc/rc.d/init.d/httpd restart
(rss)(master) [dwarren@thebigwave apps]$ git add settings.py rss.wsgi sites.py amazonfeed/models.py amazonfeed/feeds.py amazonfeed/tasks.py
(rss)(master) [dwarren@thebigwave apps]$ git commit -m 'Final commit'

Well, that’s it! The Amazon Free Apps of the day are now parsed and I hopefully won’t miss any in the future, and if you point your RSS reader at http://rss.dougwarren.org/AmazonFreeAppFeed/ maybe you won’t either.

1.  Python HTML Parser Performance
2.  Django/Celery Quickstart (or, how I learned to stop using cron and love celery)

His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot – albeit a perfect one – to get an “A”.

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.”
- Art & Fear: Observations On the Perils (and Rewards)

Central Nervous System Adaptation

Since I started resistance training in October of last year, I haven’t missed a single day. Three times a week initially moving up to four recently with added depletion workouts when I switched to a Cyclic Ketogenic Diet that’s the same sets of back squats, same sets of dead-lifts, and same sets of bench presses week after week. After a period of time your central nervous system will no longer adapt to the demands placed on it and you must change to move forward. In my case, I knew that I had a week vacation coming up at Disney World. Unlike previous vacations I made no plans to find a local gym, I didn’t pack protein-powder in my luggage, instead I planned on eating three meals a day rather than my traditional 6, and the only exercise I got was hours of low impact cardio from walking through the parks again and again. When I returned I weighed myself the day after resuming the low carb portion of the CKD, and I weighed the same I had the Monday before I left.

Bad Form

The second major problem that I suffered from is that after the loss of my work-out partner, I had no one to ‘keep me honest’ I was close to a few goals in the different compound lifts and I grew sloppy in desiring to reach them. For bench presses I increased the weights if I was able to reach all of the reps in my set regardless of how shakey or close to failure I was. For Squats, I had the squat turn into a good morning coming out of the hole on the first rep, but I was so close to a goal I increased the weight anyway. Finally for Dead-lifts I’d increase the weights even on a case when I missed the last rep of a set. In each case I made my goal, but ran into a plateau almost immediately afterwords. For this issue and likewise for Over Training there’s only one real solution.

De-Load

(Also known as the backoff, or reset) If you reach a plateau in an exercise particularly in one of the major compound exercises where week after week you reach the same weight for the same reps or even fall back a rep or two, the number one way to break this is to reduce the weight by 20% and start increasing the amount again on a weekly basis.

In my case it was only the compound lifts that were stalling (However it was all three of them.) I’m still making gains to every accessory and that feels good, but more importantly after resetting the weight I was able to focus on the form again. I’m convinced now that in the absence of a partner watching the form the most important thing to pay attention to while lifting is the barbell speed. That in addition to the number of reps made in the set will determine if I increase a lift for the next session.

With-in weeks I will be at the same place I was before my de-load, and the week after that I expect to surpass it. (3 weeks for bench, 2 weeks for squats, 3 weeks for dead-lifts.) Because, sometimes to move progress you must move backwards.

OAuth uses a three part handshake to verify the credentials of a connection. If there is an application that wishes to make a transactional request for an user with a 3rd party, the application and the 3rd party (Google, twitter, etc…) first communicate together setting up a public key encryption involving a key and a secret. (Called the client credentials) Using these credentials the application requests a temporary set of credentials called the Request Token that identifies a transaction to take place in the future. The application then redirects the end user to the server giving the request token as an argument to the URL. The end user identifies themselves to the service and on authentication the end user is redirected back to the application website to either a predetermined callback page or to a page specified in the redirect. At this point the client presents it’s oauth token and secret to the application which the application then use to connect to the remote server and validate it. This last step returns an access token. Until the access token expires, it may be cached to link an user with the permitted resources.

By design neither the application developer personally nor the application itself knows anything about the credentials that the user submitted to the remote system, the application does not know any passwords, nor does the application knows any e-mail addresses or other mechanisms to identify the user, it only knows that given this access token the remote server will respond with requested information about a specific user; to the remote system he has been authenticated, and the application has been authorized to make requests.

Acquiring OAuth:

You can get a Python OAuth library straight from its source and install it as a site-package from GitHub It also relies on httplib2 which can be acquired from Httlipb2‘s website. You can simply copy the python2/httplib2 and oauth2 into the web2py/site-packages directory.

For example application I will be creating a twitter client, it’s not the most original thing ever, but it’s a functional test. First register with Twitter’s OAuth Clients On this form, select a browser application, Read & Write access, that your app does want to use twitter for authentication, and specify a callback URL as site/app/controller/function. In the example I am using http://127.0.0.1:8000/twitter/default/callback Then a page will be displayed that shows the consumer key and secret, as well as the three URLs to connect to the server, the Request token URL, Access token URL, and Authorize URL. These are common with any OAuth service that I’ve seen.

Creating The Application

Create a new app through the administrative control panel and name it twitter. Begin editing models/db.py by appending to the end of the file:

consumer_key = "PASTED FROM TWITTER"
consumer_secret = "PASTED FROM TWITTER"

request_token_url = 'https://twitter.com/oauth/request_token'
access_token_url = 'https://twitter.com/oauth/access_token'
authorize_url = 'https://twitter.com/oauth/authorize'

Now that the global read-only data is in place, edit the controllers/default.py file to load the needed libraries:

from oauth2 import Client, Consumer, Token

try:
    from urlparse import parse_qs, parse_qsl
except ImportError:
    from cgi import parse_qs, parse_qsl

import time
import gluon.contrib.simplejson

The parse_sql trick is straight from the OAuth API. urlparse was added to the standard lib with Python 2.5 (And in Python 3.0 it has been moved to httplib.Parse) in Python 2.4 cgi is a base module so the above import will maintain backwards compatibility as is the way of web2py.

OAuth Tickets

There are three states that an user can be in when it comes to an OAuth connection, they can have no tokens at all in which case a Request Token will need to be generated and handed to them, they could be giving the application an oauth token for it to validate, or they could be fully authenticated and authorized and have a valid auth token associated with them. To tackle the states one at a time:

in the default.py controller file, edit the index method as:

def index():
    consumer = Consumer(key=consumer_key, secret=consumer_secret)

    client = Client(consumer)

    resp, content = client.request(request_token_url, "GET")
    if resp['status'] != '200':
        redirect(URL(r=request, f='maintenance'))

    # Turn response into dict
    request_token = dict(parse_qsl(content))

    # Store it in a session
    session.request_token = request_token

    redirect_location = "%s?oauth_token=%s" % (authorize_url, request_token['oauth_token'])
    redirect(redirect_location)

A Note on Redirects

Web2py redirect() method acts as a Python exception, all code stops being executed at that time and a 302 HTTP redirect header is sent to the client. From web2py’s point of view, when the user is redirected a new request and response object is created as it is an entirely new connection. This enables the application to control the work-flow in a dynamic fashion.

What I’ve done here is to create a consumer object using the consumer key, then requested a request token for this transaction. If anything went wrong with this request I’ve redirected the user to a maintenance page. Without a twitter token and with no way to get one the application doesn’t do anything. However if one was retrieved properly (As it probably would be in almost every case) it is then converted to a dict and store it in the current session for later retrieval. In the future it will be stored in the auth database instead. What is being stored in the session? Just two pieces of data:
oauth_token : tPmOLbed75wgvqVKZXmStShhNIBSmaGBjIOBABGHHo
oauth_token_secret : THE SECRET REMOVED

Now access the page with a web browser, You will be asked to log in via a twitter log in page and then you will be redirected to a page that doesn’t yet exist, in my case: http://127.0.0.1:8000/twitter/default/callback?oauth_token=UkGtfUfvQ4rJlNIn35WhHi8UBNXo9EPVMuzvOvcavM where the later part was the oauth token returned from the log in. This oauth_token now needs to be validated with the remote server and turned into an access token to complete the third part of the handshake. Doing this requires the callback function to be implemented:

def callback():
    if session.request_token:
        consumer = Consumer(key=consumer_key, secret=consumer_secret)

        token = Token(session.request_token['oauth_token'],
            session.request_token['oauth_token_secret'])

        client = Client(consumer, token)

        resp, content = client.request(access_token_url, "GET")
        if resp['status'] != '200':
            redirect(URL(r=request, f='maintenance'))

        access_token = dict(parse_qsl(content))
        session.access_token = access_token

    redirect(URL(r=request, f='index'))

At this point there is now a successful access_token in the session and calls can be made using it. Each OAuth server can return additional data about the session with the access_token. However, this is not required and is completely specific to the server being connected. For example, Twitter will return two pieces of data: session.access_token['screen_name'] is the screen name of the logged in person, and session.access_token['user_id'] is the user id. Now that the access_token has been created, new requests can be made to gather more data. Add a new block to the start of index() as so:

    if session.access_token:
        consumer = Consumer(key=consumer_key, secret=consumer_secret)
        token = Token( session.access_token['oauth_token'], session.access_token['oauth_token_secret'])
        client =  Client(consumer, token )

        resp, content = client.request('http://api.twitter.com/1/statuses/home_timeline.json','GET')

        if resp['status'] != '200':
            if resp['status'] == '401':
                session.clear()
                redirect(URL(r=request, f='index'))
            redirect(URL(r=request, f='maintenance'))

        return dict(statuses=gluon.contrib.simplejson.loads(content))

This block of code is something that will be seen repeatedly in dealing with OAuth requests. Create a consumer, Create a token, Create a client, make a request of the client, verify the result of the request, decompress the request into a dict. Additional checks to ensure that the token is still valid. If it’s not then all data associated with the session will be cleared and the login workflow will begin again by redirecting the user back to the index page. Here a request was made for JSON encoded data, one could also ask for XML encoded data from Twitter and parse it with an XML reader. To my knowledge there is no difference among the two methods of encoding. I prefer to request JSON just in case I need to display the raw data on the client and manipulate it via JavaScript.

By the end of the controller’s index() execution, any status updates that the user would see from the front page of Twitter is loaded into the statuses key and can be displayed in the view. To do so, edit views/default/index.html as:

{{if statuses:}}
{{tab = TABLE(TR(TH('Tweeter'),TH('Tweet')))}}
{{for status in statuses: }}
	{{tab.append(TR(TD(status['user']['screen_name']), TD(status['text'])))}}
	{{pass}}
{{=tab}}
{{pass}}

This ends the initial post on how to use OAuth with Web2Py, in the next post I will discuss integrating the web2py auth by associating an existing user with an oauth session, and finally with using oauth completely in place of the auth logins. I will also discuss decorators slightly and global functions to make your life easier.

So one of my goals is to add to this problem. As I have read the past several months of archives of the mailing list I’ve kept notes of what’s interesting in the problems that come up and the solutions offered. Over time I hope to document these bits of knowledge and provide links to running examples of how they can be used.

The first of these however is one that I’ve come up with myself the other day. I have an app in mind that requires the assumption that the user is logged in before taking any action in that controller. Other controllers allow some actions while not logged in, but this controller was deep in the work-flow and each function with-in it was assumed to have a valid user. What I had done initially was to decorate each function in the controller with an auth check such as:

@auth.requires_login()
def index():
    return dict(message="You are logged in")

The problem of course is that as the number of functions in the controller grew it became very error prone to have to decorate each function. Also it turned out that there was a small class of functions where the logged in requirement wasn’t actually a requirement and that the end-user should be able to to access the page while not logged in. So after a few false starts, I settled on:

import urllib

# List of functions to be called without being logged in
whitelist = ('chat')

if not auth.basic() and not auth.is_logged_in() \
   and not auth.environment.request.function in whitelist:
    request = auth.environment.request
    next = URL(r=request,args=request.args, vars=request.get_vars)
    redirect(auth.settings.login_url + '?_next='+urllib.quote(next))

Placing the above snippet in your controller file will allow the chat function to be executed without being logged in, but any other function request will redirect the viewer to the login url and then back to their original destination.