The merge of Django's queryset-refactor branch

by Sean on Apr. 27, 2008

The queryset-refactor branch

This branch was used to develop a major refactoring of the django.db.models.query.QuerySet class to fix a group of SQL problems and make SQL generation easier for database backends requiring customization.

Status

The branch was created on 13 September, 2007, and merged into trunk on 26 April, 2008 (in [7477]).

New features

Along with, and as part of, all the bug fixes mentioned above there are a number of new features in the branch. A number of these features are purely internal details, but there are a few that add extra public functionality.

Ordering querysets across related models has a new syntax that is the same as the way you specify relations in a filter: field1__field2__field3, etc. The new syntax is more natural and consistent, as well as helping solve a few bugs. See the order_by() documentation for more information and some examples.
Model inheritance is now possible. Both abstract base classes and multi-table inheritance are possible. See the model-api documentation for details.
The __iter__() method on querysets does not pull all the results into memory immediately. This reduces memory usage for large querysets where you don't end up accessing all the results. Queryset caching still occurs, though, so a single queryset object will only hit the database once. This change means that testing the boolean value of a queryset will only pull in the first few rows of the result, not all of them.
Slicing a queryset from a particular value to the end of a queryset is possible.
Querysets have a reverse() method that reverses whatever the current ordering is.
The queryset values() method can retrieve fields that are related via a ForeignKey or OneToOneField relation.
A new values_list() method has been added to querysets. This is similar to values() except that it returns a list of tuples, rather than a list of dictionaries.
You can specify a list of related fields to traverse in a select_related() call. This provides a way to select only the related data you are interested in. Only single-valued relations can be selected in this way, however (not ManyToManyFields).
Filtering a queryset by checking if a field attribute is None is equivalent to testing if the corresponding database column is NULL. So qs.filter(foo=None) is now identical to qs.filter(foo__isnull=True).
An update() method has been added to querysets to allow multiple objects to have an attribute updated in a single SQL query.
Q classes now fully support &, | and ~ to combine them in pairs as conjunctions or disjunctions or to negate the sense of a filter, respectively (& and | were previously supported, but returned a different type of class). Thus the QAnd, QOr and QNot classes are no longer required and have been deprecated. Using them raises a warning (although they still work as before).

Backwards incompatible changes

A few backwards incompatible changes are created by the changes in this branch. Most people won't be affected by many of these, and porting code is reasonably straightforward.

Most visible

The OneToOneField class has finally been updated, as the documentation has indicated would be happening for a long while. There are few externally visible changes, with one exception: a OneToOneField is no longer automatically the primary key for a model that includes it. It still accepts the primary_key attribute, however, so you should add primary_key=True to the declaration of any existing OneToOneField instances in your code to preserve backwards compatibility.
If you pass a bad field name into a filter or order_by(), Django now raises FieldError (from django.core.exceptions), rather than Python's built in TypeError. Also, the list of legal field names is now sorted alphabetically for easier searching. This should have no effect on most production code, however some test suites may need to be updated to accommodate the changed traceback output.
There is a slight difference between these two filter statements
```
qs.objects.filter(f1).filter(f2)
qs.objects.filter(f1, f2)
```
This difference only applies when f1 and f2 are referencing the same multi-valued relationship (a ManyToManyField or reverse ForeignKey). The first version allows filtering against different items from the relationship (things that match f1 on one object in the related table as well as f2 on another object in the related table), whereas the second version's filters will be applied to the same item. See the database API documentation section called "Lookups that span relationships" for details.
It is possible to use extra select fields -- those included via extra(select=...) -- for ordering the results. Previously, those fields could be specified to the order_by() method. Due to increased error checking, that is no longer practical. Instead, pass those extra columns to the order_by argument of the extra() method:
```
qs.extra(select={'a': ...}).order_by('a')     # Old style

qs.extra(select={'a': ...}, order_by=('a',))   # New style
```
This only applies to ordering by items in an extra(select={...}) dictionary. Normal ordering is still done with the order_by() method to querysets, there have been no changes there. So this change will affect relatively few people.

Other

The Options.get_order_sql() method is now gone in django.db.models.options. There appears to be no use for this method any longer.
Q objects have changed internally. This is only relevant if you have created custom Q-like objects. You would have created a get_sql() method that returned a data structure that was inserted into the query. In the new code, you create a add_to_query() method that accepts two arguments -- the django.db.models.sql.query.Query instance for the current query and a set of aliases used in the current filter() call. Your Q-like object can then add to the various attributes of this class (select, where, etc) to have whatever effect it likes on the result. Note that the add_to_query() method is called when the object is added to the Query object and more changes may be made before it is turned into SQL and executed against the database.
Still on extra(select=...)... if you want to substitute parameters into these extra selection columns, use the select_params argument to extra(). The params argument is only applied to the extra where conditions.
select_related(False) is no longer possible. Don't worry. You didn't know this existed, so you won't miss it. It was never part of the official API.

Things to Note

Model inheritance has not been integrated into the existing admin. It will eventually be implemented in the newforms-admin branch.
OneToOneField in the admin interface has a similar status.

Do you need BigTable as an webservice?

by Sean on Apr. 15, 2008

Well, the evergreat Google do offer us an abstraction on top of her BigTable database layer using her own API and we could only use these api if we have an GAE account. Obviously it's a little bit disappointing if we only want the BIG TOY in our pocket. But no more than one precious weekend, there is someone jumping only to give anybody free access to the database itself.

This application is called BigTable As A Web Service, it offers an RESTful API of just using GET/PUT/DELETE methods to access the row and column of your database. Just like what we should do with URIs. I must say it's quite a great idea for saving small data if needed. And Google should offer this kind of API as a Webservice, Does she?

The second app that i'm gonna talk about is SPRUNGE. This small application allows you to redirect the standard output scream directly to an webservice. Then yo could check the output with your web browser. What i'm thinking is using it to test my bash scripts, cause the syntax highlignting helps a lot for me to check out the errors. Just like this one.

An interesting Sketch APP on GAE

by Sean on Apr. 13, 2008

With developers being more familiar with Google App Engine's usage, there is some interesting apps coming up.

Just like this one, Simple Sketch App.

simple sketch app

In the box above, you could move your mouse to draw picture in it as well as saving it and share the painting with others. It's quite a cool application to play with.

And here's the author's link:

http://d.hatena.ne.jp/technohippy/

The other one that caught my attention is Latrz

latrz

This is a application for users to submit articles when they are surfing on the internet, and come back to read them if they got time later, to me it's quite a useful app, cause i always can't find the articles when i really want to read them. So thanks a lot!

Trackback Class in Python/Django

by Sean on Apr. 13, 2008

Trackback is a simple way of linking blog entries together, and it might be the most misused for SPAM.

When i'm trying to build Gaelog, I google a lot about trackback lib writing in Python, but only get one called tblib. It's a very simple trackback class using the default httplib. The thing about httplib is that it can not set a timeout attribute which i think is very important in doing the trackback. Because if a remote server can not handle the post in time, the function will always be blocked.

So i end up writing my own trackback class based on the wonderful pycurl.

Here is the code ( Most of them based on the tblib, it's ok? ):

class TrackBack:
    
    def __init__(self, tbUrl=None, title=None, excerpt=None, url=None, blog_name=None):
        self.tbUrl = tbUrl
        self.title = title
        self.excerpt = excerpt
        self.url = url
        self.blog_name = blog_name
        self.tbErrorCode = None
        self.tbErrorMessage = None

    def ping(self):
        if self.tbUrl:
            params = urllib.urlencode({'title': self.title, 'url': self.url, 'excerpt': self.excerpt, 'blog_name': self.blog_name})            
            agent = "Appoil.com/0.0.1 Python"
            c = pycurl.Curl()
            c.setopt(pycurl.URL, smart_str(self.tbUrl))
            c.setopt(c.POSTFIELDS, params)
            b = StringIO.StringIO()
            c.setopt(pycurl.WRITEFUNCTION, b.write)
            c.setopt(pycurl.FOLLOWLOCATION, 0)
            c.setopt(pycurl.USERAGENT, agent)
            c.setopt(pycurl.SSL_VERIFYPEER, 0)            
            c.setopt(pycurl.TIMEOUT, 30)            
            c.perform()                    
            self.tbResponse = b.getvalue()
            errorpattern = r'(.*?)'
            reg = re.search(errorpattern, self.tbResponse)
            if reg:
                self.tbErrorCode = reg.group(1)
                if int(self.tbErrorCode) == 1:
                    errorpattern2 = r'(.*?)'
                    reg2 = re.search(errorpattern2, self.tbResponse)
                    if reg2:
                        self.tbErrorMessage = reg2.group(1)
        else:
            return 1

If you came here from the trackback links that means this class works just fine. :-)

There is one more thing i should say about the pycurl. If you are trying to compile pycurl lib from source code, the LD_LIBRARY_PATH environment variable is a bit tricky. I spent quite some time to figure out why it don't accept the PATH i set. After compiling the lib for several times, it's ok within Bash. But Django still can't find the libcurl.so file. My last attempt was to copy the .so file into my own lib directory. Luckily everything seems all right this time.

Inspired by:

http://weblog.philringnalda.com/2003/02/01/tblib-a-python-trackback-library

http://coderseye.com/2007/trackbacks-on-banjo.html

Rich Text Editor in Django Admin

by Sean on Apr. 12, 2008

The built in Django admin application is quite useful in typing simple data into your system. But when dealing with complex html format, it just can't work.

Then integrating with rich text editor is a good choice, but it need some extra work either in the admin itself or using the newform admin branch. After looking around into this branch, i'm not sure if it's the right time to use it right now. On the other hand, i choose the other way round to solve this problem, yup, i'm not adding the rich editor into django, but to setup a FCKeditor outside.

Now i can insert all the things in this editor, like images, urls, flash, etc. After finishing, just click the source button to get the xhtml1.1 code and paste them into the TextField in Django admin interface. Honestly, it saves me a lot time.

The one thing more i want to say is that, the rerason why there is no popular blog system built in Django out there yet is that everybody could start their own app in quite a short time, and find sort of UGLY ways to overcome the shortages like this. And more ugly the method is, the less people wanting to show their code to the others. So i really hope that the newform admin branch could fix these kind of problem and bring us a much powerful admin interface in the future.

Inspired by:

http://yuiblog.com/blog/2007/08/13/rte-notes/

The Why

by Sean on Apr. 12, 2008

On Apr. 7, 2008, the ever great google reveals world's biggest secret that she is ready to host any kind of web application in the unique contain which is called Google App Engine.

To a web developer, this kind of movement is no doubt revolutionary. We used to do a whole lot of worthless things such as to buy a server, to install an OS, to choose a language and a framework. Now we don't need to compare the pros and cons among all the choices. A simple but tremendous solution is right there waiting.

Once upon a time, i watched a video about Guido van Rossum's talk about his job at Google involving a code review system using Django's template framework. This really starts an debate over which one is the right python web framework. And before long, with the launch of App Engine including a naturely built in Django support. To a guy make the right choise, nothing can be better than this.

From what i can imagine right now, there will be millions of new application taking shape on the this platform. Tracking all these new apps down should be quite exciting. And that's the reason why i took some time to set up this website. I hope that in one year from now, i could collect enough applications to cover all the aspects about online activities and all of them based on this simple but powerful engine.

Google has offered us his engine, so let's burn it with oil!