Simple XML creation with Python

I was so happy when I first found Amara. It was the easiest thing I had seen for creating XML with Python. It’s quite serious stuff because it can probably handle just about anything you throw at it. Then I heard about the wonderful web2py project. One of it’s beautiful aspects is XML generation, or any tag type of thing actually. Some of that discussion is here. Thing of beauty. I wanted to use web2py but because of time restraints, my current project will continue to use Django, which definitely isn’t a bad thing. I just understand web2py more intuitively. I thought about using the TAG but I didn’t want the extra database hit when importing TAG from the gluon.html library. 😦

So I decided to search for python simple xml thinking of PHP’s SimpleXML. Results with DOM, SAX or elementtree… or anything like that… automatically disqualified. Amara was better for me than those. Then I saw pyfo…

pyfo – Easy XML Generation from Python

This package was developed by Luke Arno for Central Piedmont Community College in response to dissatisfaction with available alternatives for quickly generating XML. Concatenating strings is ugly and error-prone, using OO APIs for XML is heavy and overkill for generating simple XML output.

No kidding!

In [16]: from pyfo import pyfo

In [17]: a=(‘test’,(‘test’,’more stuff’,{‘cool’:’beans’}))

In [18]: pyfo(a)
Out[18]: u'<test><test cool=”beans”>more stuff</test></test>’

In [19]: print pyfo(a, pretty=True, prolog=True)
<?xml version=”1.0″ encoding=”utf-8″?>
<test>
<test cool=”beans”>more stuff</test>
</test>

Does it get any easier than that?!

PyAMF flying high

More python/Flex goodness in the works in the future perhaps? 🙂 I really like Python… learning all the time with my web2py friends. I’d love to be able to combine this with Flex. Looks like it’s getting easier and easier. Now to combine this with Orbited…

Ted On Flex: The AMF Revolution

The growth of AMF since the release of Blaze DS and the AMF spec
has been nothing short of amazing. I continue to see lots of great new
server software projects leveraging AMF data formats. This morning I
was reading the FlexCoders mailing list and an announcement of a new AMF library “PyAMF – AMF for Python” appeared leveraging Python targeting Twisted (I love twisted), Django, Pylons, TurboGears and CherryPy. Seeing a single project add this much functionality into Python web development is really great for the Flash/Flex developer community.

Python
PyAMF Project
PyAMF Examples

Running postgresql createdb from batch without password prompt

As noted in this post, you need to change a line in pg_hba.conf to “trust” instead of “md5” (no quotes) to be able to batch create tables. Note this is only for testing purposes and not recommended for production. The only reason is because I am running some benchmarks for web2py. I want to be able to create the database itself from within the benchmark script and not have to create it each time.

Error while running createdb from a batch file in windows xp – PostgreSQL Forum

Unless you set up your database access with the “trust” option this prompt is probably asking for your password, not the password for the username you passing to the createdb command.

Exporting a web2py table in XML

I recently wrote about exporting a Django database to xml. As a courtesy, Massimo, creator of web2py, put together a great little example for web2py

exporting in XML – web2py Web Framework | Google Groups

def export_xml(rows):
    idx=range(len(rows.colnames))
    colnames=[item.replace('.','_') for item in rows.colnames]
    records=[]
    for row in rows.response: records.append(TAG['record'](*[TAG[colnames[i]](row[i]) for i in idx]))
    return str(TAG['records'](*records))>/pre>

Here is how you would use it:

assuming:
db=SQLDB('sqlite://test.db')
db.define_table('mytable',SQLField('myfield'))
for i in range(100): db.mytable.aaa.insert(myfield=i)




def index():
    response.headers['Content-Type']='application/xml'
    return export_xml(db().select(db.mytable.ALL))

This is using the version currently in trunk. The next update (1.28) should be out in about 1 week though so it should be possible with the release version very soon. One thing I really like about web2py’s tag handling is the TAG item. You can make any tag needed with that and it’s quite sweet.

This particular example just defines the export function, defines a database, a connection, a table, populates the table, and shows how to use it. Sweet. If I get the change, I’ll port a current Django app to web2py… but future apps will be on web2py I think.

And the server is using the wsgi server of cherrypy so it’s rockin’.

web2py new website

Web2py is humming along and just got a redesigned front page. Nice, clean, and accessible. Gotta love the “complex” tutorial. Fits in a page! 🙂

web2py Enterprise Web Framework

A taste of web2py

Consider the following complete application which consists a model (which describes the data representation): db.py

1. 2.

db=SQLDB('sqlite://images.db') db.define_table('image',SQLField('file','upload'))

a controller (which describes the application logic and workflow): images_examples.py

1. 2. 3. 4.

def index(): form=SQLFORM(db.image) if form.accepts(request.vars,session): response.flash='image uploaded' return dict(form=form)

and a view (which describes the data presentation): images_examples/index.html:

1. 2. 3.

{{extend 'layout.html'}} <h1>Upload page</h1> {{=form}}

What does it do?

• Creates the database db in file ‘images.db’
• Creates the table ‘image’ which contains a field called ‘file’. If
  the table exists but does not match the definition it is altered
  accordingly.
• Creates a web-based database administrative interface for db.image
• Creates a web page called index with upload form for db.image. Try it here
• On upload the file is renamed in a secure way, saved, and the name of the file is stored in a new field db.image record.

Export Django database to xml

Looking at this helps me understand more of what is going on internally with Django‘s models. Using web2py is a bit more intuitive for getting some of this info so it’s probably not too hard to port the concept over… for a future exercise.

/home/siddhi: Export django database to an xml file

Export django database to an xml file
Here is a simple python script to export your django database to an XML file. I haven’t tested it out very thoroughly. It seems to work for the fields that I have in my model – CharField, TextField, DateField, IntegerField, PositiveIntegerField and ForeignKey. If you find any bugs, add a comment to this post.

Formencode example

A full example, albeit with SQLObject, but the concept is pretty clear. With web2py, the form.validate would be followed with an explicit data insert. From the looks of the **form_result insert, form.validate returns a list of fields that can be directly inserted. The same format should be able to be used with Storm.

Handling Form data with Formencode + SQLObject

Two of my favorite and most often-used Python packges are formencode and sqlobject.
Using these packages together is done fairly often, but I’ve rarely
seen any documentation describing a full form display, validation, and
insert into a database of new data from said form.

In this example, I’ve tried to wean down the code as much as possible
to get a very concise snippet (only 12 lines of web page Controller
code) that will:

1. Display a new form to the user
2. Handle form submission
3. Validate the form
4. Coerce form values into the proper types for the database
5. Insert the form data to the database, or
6. Display the form with error messages, and maintain their values

The
nice thing about using formencode, is that 3 of the 6 steps I listed
above are handled by it in a fairly automatic way. Formencode will
check the data given a form schema, coerce the values into the Python
types you ask for, fill in errors should they occur, and maintain the
existing values.

Predefined variables with lambda variables

This little line right here solved a problem. How do you dynamically assign a function to a variable to be invoked later but you need it to have a preset value? Define it when you assign it. I just didn’t know you could do that with lambda functions. Cool! This was used to update my version of the web2py shell. More on that in a sec.

p-nand-q.com : Computer : python : Stupid Lambda Tricks

f = lambda x: (lambda result=range(20):do_something(result,x))()

Web2py’s interactive shell in iPython

This shell for web2py is quite cool. (You can look here for a Google tranlation of the original.) For those new to this stuff (like me), and if you’re working on something specific you can specify that in the environment. If you want to play with your model straight away, and it’s named ‘db’, you can add

environment[‘db’]=__import__(‘applications.%s.models.db’ % app, fromlist=[‘*’])

Just a convenience thing for testing.

As I worked with it, I wanted to load it into iPython. My very poor way is I have a small script to just load the environment.

# file:web2pyshell.py
from shell import *
app=raw_input(‘app to load environment for:’)
for k,v in env(app).items(): locals()[k]=v

# in ipythonrc add
execute sys.path.append(‘C:\\path\\to\\web2py’)
execute os.chdir(‘C:\\path\\to\\web2py’)
execfile web2pyshell.py

so when I load up iPython, it’s all loaded up and waiting to go. Very very nice.

Simple web2py and Elixir

So I have a sample data set in a database created by web2py. After some experimentation I can access, modify and create data with Elixir/SQLAlchemy. Just some notes, like remember to encode web2py blobs with utf8 if you’re not going through web2py’s functions.

r=Recipe.query.all()
r[0].instructions=u’some text to submit’.encode(‘utf8’)
cb.session.flush()

And then something happend before this got pulished and it broke. Shame. Oh well…

web2py interactive database sessions

After some experimentation, at least with the way I am working with web2py’s orm, the migrate command causes problems from an interactive shell session, i.e. within iPython.

So instead of

migrate=’some_file.table’

I have to use

migrate=’some_file.table’ if request.env else False

That’s no so bad. Previously I thought I would have to use

migrate=’some_file.table’ if __name__==’__builtin__’ else False

I didn’t like that. There may be another way but this is pretty clean and consistent for now. It also means I can combine ORMs with web2py’s orm now.

Elixir: Simple sweet access with SQLAlchemy behind the scenes

I really like using web2py. I only have one major problem with developing with it. I like code completion and other good things that come with developing in an IDE.

First off, SqlSoup for SQLAlchemy. Very cool. It’s got runtime introspection which, in plain English, means that you can experiment with the database, seeing what is available, etc., without always having to keep the whole thing in your head. The only thing I don’t see it doing, so far, is being able to tell you which tables are available at runtime. Somebody tell me if I am wrong! But I digress.

With web2py’s orm, there are a lot of neat functions. It’s cool. I have used it a bit differently though. That’s for another post. I wrap specific calls as methods in the model so that I can always update things in one place. I am trying to keep it DRY. I need to change how I get whatever specific value? I change the method at the model level. That’s something I got used to with Django. I like it, it works for me, and it’s helpful.

The side effect of doing that, especially since I use classes in my models for web2py, is that I get code completion for that in my IDE. I use PyDev in Eclipse. I think I really started to like that. So I thought, “It would be neat to get to the fields of my database objects while in the IDE.”

The first step was SqlSoup with it’s runtime introspection. The next step is Elixir with it’s simple (drop dead simple) database class setup.

Elixir – Trac

How would I use it with web2py? Keep the standard web2py define_table calls. This allows me to keep the use of the admin and other functions that run off of web2py’s orm, like SQLFORM, SQLTABLE, and other things. Then, in an ideal world, I could setup models with Elixir and be able to use SQLAlchemy with web2py. I just have to remember to not use the autoload or autosetup options, and to use “setup_all(create_tables=False)” so that it doesn’t try to re-build the tables.

I like web2py’s orm but I am still debating some things about it. It’s not that it is bad or anything, just balancing things as I learn and develop.

Single place for settings for database

While using web2py, this is how I am working with databases now.

Make a new application called ‘init’
In init.models create an empty __init__.py file so you can import the settings.
In init.models add create a settings.py file.
in init.models.settings add:

DB_DEV=’dev connection string’
DB_STAGING=’staging connection string’
DB_PRODUCTION=’production connection string’

def getDBConnection(request):
connect_string=None
try:
# we will only have http_x_farwarded_host when using mod_proxy on apach
# so it can only be the staging or production server
if request.env.http_x_forwarded_host:
connect_string=DB_STAGING if ‘staging’ in request.env.http_x_forwarded_host else DB_PRODUCTION
# so if there is no http_x_forwarded_host then it is the local dev box
elif not request.env.http_x_forwarded_host:
connect_string=DB_DEV
except:
connect_string=DB_DEV
return connect_string

In applications.yourapp.models.your_db_file

from applications.init.models import settings
db=SQLDB(settings.getDBConnection(request))

The global request variable is used to determine the host so it must be passed in. This can be expanded of course to include other things like email settings or whatever is needed.

Interactive web2py testing

web2py is a great web development app. It’s under heavy development and the creator answers tons of questions making many updates to the core code in very short order.

In the quest for TDD, now BDD, I wanted to be able to get an interactive session going in iPython. There are some things I had to do to the iPython ipythonrc file, like adding the web2py to the sys.path variable, but it seems to be working.

There are some gotchas. When you import something with iPython, it runs it, or so it seems. I’m still a newbie at this though so I am most likely wrong!

But, this works for my particular setup. In an iPython session:

In [62]: %doctest_mode
*** Pasting of code with “>>>” or “…” has been enabled.
Exception reporting mode: Plain
Doctest mode is: ON

>>> from gluon.globals import Request
>>> from applications.cookbook.models import db
>>> r=Request()
>>> cb=db.Cookbook(r)
>>> cb.ge #auto completion in the shell
cb.getcategory cb.getrecipe
>>> cb.getcategory(1)[‘name’]
‘dinner’
>>>

So I now have interactive access to the database which is what I want for testing. And I can get doctests straight away as well. That’s pretty cool. This gives shell experimentation similar to Django’s shell app admin. As you can see there is some auto completion going on there as well. Neat!

Oh yes. In my ipythonrc.ini I have the standard

import_mod sys os

and added

execute sys.path.append(‘C:\\path\\to\\web2py’)

and in that session I also did a

os.chdir(‘C:\\path\\to\\web2py’)

Perhaps I can/will make different profiles for different tests but this is the main thing I’ll be testing. I’ll just put those things into the ipythonrc file. I thought I wouldn’t need the sys.path.append AND the os.chdir but both are needed. sys.path.append allows the import, and os.chdir allows autocompletion in the shell.

ScribeFire seems to be causing trouble with the code prompts though. 😦 Trying an edit in Zoundry’s Raven now… and it worked! 🙂