May 13, 2008

Django, Sphinx, and full-text search

Filed under: cool, django, orm, search, tool — michaelangela @ 6:27 pm

This makes some interesting things possible. And it makes sense just looking at it. That’s even better. 🙂

Announcing django-sphinx 2.0.0 Full-text Search | David Cramer’s Blog

One of my tasks lately has been updating the django-sphinx
library to work with Sphinx 0.98, as it includes GIS components which
we have been wanting to possibly utilize on our new search engine. So
today, the conclusion of.. very few hours of work, I’d like to announce
2.0.0 of the Django integration.

The SphinxSearch manager has been completely
rewritten to rely strictly on the Python Sphinx API (a request from the
author of Sphinx), which means it should have much more compatibility
with newer versions in the future. I have also updated the querysets it
returns to work like normal Django QuerySet instances, to where they
clone themselves so you can fork your queryset into several different


May 7, 2008

Snippet to update Django tables to utf8

Filed under: cool, django, orm, python, snippet, tip, tool — michaelangela @ 11:54 pm

So you get the dreaded illegal mix of collations:

Pythoneer » Convert MySQL table to utf8

Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)

What do you do? Not too bad fortunately. And if you’re working with Django, it’s a little bit easier. Launch the shell with python manage.py shell. Then in the shell do something like this:

from django.db import connection
cursor = connection.cursor()
cursor.execute('SHOW TABLES')
for row in cursor.fetchall(): results.append(row)
for row in results: cursor.execute('ALTER TABLE %s CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;' % (row[0]))

I’m sure this can be optimized a bit but it got the job done simply enough.

Full-text search for Django

Filed under: api, django, orm, tip — michaelangela @ 11:34 pm

Just making a note…

Extending Django’s database API to include full-text search (Mercurytide)

April 24, 2008

Concat Django Querysets

Filed under: dev, django, orm, tip — michaelangela @ 9:46 pm

Here is a good discussion on concatenating Django Querysets. Very helpful! It also introduced me to itertools. This leads to some other talk about itertools. Very interesting!

The jist though is like this:

How do you concatenate two querysets? – Django users | Google Groups

import itertools
q1 = Model1.objects.all()
q2 = Model2.objects.all()
for thing in itertools.chain(q1,q2):

Very handy.

March 29, 2008

Exporting a web2py table in XML

Filed under: cool, dev, education, one-liner, orm, web2py — michaelangela @ 4:02 am

I recently wrote about exporting a Django database to xml. As a courtesy, Massimo, creator of web2py, put together a great little example for web2py

exporting in XML – web2py Web Framework | Google Groups

def export_xml(rows):
    colnames=[item.replace('.','_') for item in rows.colnames]
    for row in rows.response: records.append(TAG['record'](*[TAG[colnames[i]](row[i]) for i in idx]))
    return str(TAG['records'](*records))>/pre>

Here is how you would use it:

for i in range(100): db.mytable.aaa.insert(myfield=i)

def index():
    return export_xml(db().select(db.mytable.ALL))

This is using the version currently in trunk. The next update (1.28) should be out in about 1 week though so it should be possible with the release version very soon. One thing I really like about web2py’s tag handling is the TAG item. You can make any tag needed with that and it’s quite sweet.

This particular example just defines the export function, defines a database, a connection, a table, populates the table, and shows how to use it. Sweet. If I get the change, I’ll port a current Django app to web2py… but future apps will be on web2py I think.

And the server is using the wsgi server of cherrypy so it’s rockin’.

March 20, 2008

Python Time Object Diffs

Filed under: dev, orm, python, storm — michaelangela @ 5:06 am

(Python) PyProgram #7 – Time Object Diffs « the minds of a tuxian

I had two time objects and needed to find the difference of time between them. Upon Googling, querying in #python and checking the documentation, found that there is not built-in function available for this. Though there were some cookbook recipes, I wanted to write my own solution to find the difference between two datetime.time objects. Though datetime.timedelta was for time difference requirements it only has a days, seconds and microseconds property while I need hour, minute and seconds. Thus, I wrote my timediff method which accepts two time objects btime and stime where btime is a bigger time than stime (b for big and s for small 😉 ), and it returns a datetime.time object for the difference time. (Why I need to have time objects? I use storm ORM 😉 )

Storm and OpenID

Filed under: dev, orm, python — michaelangela @ 5:05 am

OpenID as a WSGI filter, part 2 – Damian’s archive – Alleged Literature

Programmers learn through programming, and my research into OpenID has got as far as a web application that lets you do nothing at all except log in with OpenID. Along the way I have played a bit with Canonical’s own object-relational mapper (ORM) called Storm.

March 18, 2008

Formencode example

Filed under: dev, orm, python, web2py — michaelangela @ 9:20 am

A full example, albeit with SQLObject, but the concept is pretty clear. With web2py, the form.validate would be followed with an explicit data insert. From the looks of the **form_result insert, form.validate returns a list of fields that can be directly inserted. The same format should be able to be used with Storm.

Handling Form data with Formencode + SQLObject

Two of my favorite and most often-used Python packges are formencode and sqlobject.
Using these packages together is done fairly often, but I’ve rarely
seen any documentation describing a full form display, validation, and
insert into a database of new data from said form.

In this example, I’ve tried to wean down the code as much as possible
to get a very concise snippet (only 12 lines of web page Controller
code) that will:

  1. Display a new form to the user
  2. Handle form submission
  3. Validate the form
  4. Coerce form values into the proper types for the database
  5. Insert the form data to the database, or
  6. Display the form with error messages, and maintain their values

nice thing about using formencode, is that 3 of the 6 steps I listed
above are handled by it in a fairly automatic way. Formencode will
check the data given a form schema, coerce the values into the Python
types you ask for, fill in errors should they occur, and maintain the
existing values.

March 13, 2008

Simple web2py and Elixir

Filed under: dev, elixir, orm, sqlalchemy, web2py — michaelangela @ 6:56 am

So I have a sample data set in a database created by web2py. After some experimentation I can access, modify and create data with Elixir/SQLAlchemy. Just some notes, like remember to encode web2py blobs with utf8 if you’re not going through web2py’s functions.

r[0].instructions=u’some text to submit’.encode(‘utf8’)

And then something happend before this got pulished and it broke. Shame. Oh well…

March 10, 2008

Elixir or straight SQLAlchemy

Filed under: dev, elixir, orm, python, sqlalchemy — michaelangela @ 1:40 pm

Or “When not to use something that looks like it’s obviously going to make your life so much easier” 🙂

SQLAlchemy, Declarative layers, and the ORM ‘Problem’

On first glance, its pretty obvious that everyone should love Elixir vs the obviously more tedious SA approach of layout out your tables, then mapping them to the class objects. However, look at these two examples, and try to quickly answer the following questions:

  • How do you add a column to the many to many table to store an additional bit of info for the join?
  • Do they both enforce referential integrity?
  • How do you control whether SA is eager loading the relation? Can you restrict it to just one column of the relation?
  • What are the table names used?
  • How many tables are in your database?
  • Where do you change the id column name?
  • Which one is closer to the Zen of Python?

Blog at WordPress.com.