Ramblings

May 31, 2010

Django OSX MySQL MAMP

Filed under: django, macports, mamp, mysql, mysqldb, osx — michaelangela @ 7:29 am

For my overly customized MAMP install, it wasn’t pretty. I forget why it’s actually set up this way now but the different parts didn’t like each other and it took quite a while to get it working. Not fun. But fortunately there were those who had gone before me and it wasn’t hard in the end. It just took a long time.

http://birdhouse.org/blog/2009/02/21/python-mysql-connections-on-mac-os/

http://www.marteinn.se/blog/?p=467

Correction. It just took a really REALLY long time. The reason? You need to have libs and headers and such in places that all make sense for everything to compile. (This is the SHORT version.) So MacPorts to the rescue. Install py25-mysql, copy some files around, use a slightly different “HOST” setting for the database connection and you’re good to go. But installing py25-mysql installs ALL the dependencies needed in order to get that running. Which is good when that’s what you need. Which is what took a long time. But at least it was a one command thing and not hopping all over the ‘net to find and install things as I was very close to doing.

And that bit about the unique “HOST” setting for mine turns out to be

DATABASE_HOST = ‘/Applications/MAMP_1.7.2/tmp/mysql/mysql.sock’

December 17, 2009

Some notes on python and com using comtypes

Filed under: com, python — michaelangela @ 2:25 pm

CodeProject: Working with custom COM interfaces from Python. Free source code and programming help

There are a lot of tutorials on Python and COM over the Internet, but in real practice, you might quickly be confused just going beyond standard IDispatch things. The same occurred to me when I decided to write unit tests for our set of COM components. The components are rather simple, they implement one custom interface (derived from IUnknown) and one outgoing IDispatch interface for events.

First, I tried to use the standard pythoncom module, but it turned out that it didn’t support custom COM interfaces. Then, I downloaded the comtypes package and started playing with it. Due to a lack of documentation, it took me about one night to write a simple example. So, here is a step-by-step guide on how to begin using comtypes.

up next…

Python Order2Go Example – DailyFX Forum

For all Python programmers out there that like to play a little with Order2Go:
(On Windows Platforms only)

These are of interest specifically because they deal with getting events from the COM objects and handling them.

December 7, 2009

Plotting libraries

Filed under: python, tool — michaelangela @ 6:59 am

I wanted to look up some plotting libraries just to get an idea what’s out there. Plus I like Python for throwing numbers around although R has just become a new favorite for quickly working through some numbers… just need to learn statistics now. 🙂

But I turned to stack overflow for an answer.

matplotlib is powerful.

CairoPlot is pretty.

December 2, 2009

R and statistics

Filed under: automation, data processing, python, r, tool — michaelangela @ 11:41 am

If I have a database and I need to do some random queries where I need to do experimental queries interactively, I turn to python’s SqlSoup. But if it’s CSV, I used to just do some file handling in Python, load it up, and use lists (which are very cool) or some other straightforward data processing on it. Python’s good at that. Then I heard about R. I don’t have a lot of time to dig into the details at the moment but the link below really opened my eyes to what it can do. And how easily at that.

Basic statistics using R

The following page is a quick guide for using R to do most statistics necessary in an introductory statistics class.

Like loading in a table as a dataset:

my.table=read.csv(file.choose())

I mean seriously! While it didn’t pop up a dialog on iTerm (using latest R for OSX) I just pasted in the path to the file and my data was there as

my.table

ready for manipulation. Some searches on filtering led to this post showing:

>tt <- matrix(1:20, ncol = 4)
tt
     [,1] [,2] [,3] [,4]
[1,]    1    6   11   16
[2,]    2    7   12   17
[3,]    3    8   13   18
[4,]    4    9   14   19
[5,]    5   10   15   20
tt[tt[,1] < 3, ]
     [,1] [,2] [,3] [,4]
[1,]    1    6   11   16
[2,]    2    7   12   17

and I tried

 tt[tt[,1]>3 & tt[,3]<15]

based on a conditional seen here just to see if I understood it and got

[1]  4  9 14 19

which is what I expected but didn’t realize how easy that would be. Nice! The subset command is more explicit though with its named columns. Either way, wow that’s really nice to work with. Of course, there is also RPy:

RPy is a very simple, yet robust, Python interface to the R Programming Language. It can manage all kinds of R objects and can execute arbitrary R functions (including the graphic functions). All errors from the R language are converted to Python exceptions. Any module installed for the R system can be used from within Python.

Two flavours are available: rpy and rpy2.

Best of both worlds? 🙂

October 15, 2008

Python and PHP serialization

Filed under: dev, php, python, tool — michaelangela @ 5:57 pm

I have some data that I am pulling via Amara. I need to put that in a database. The app that uses that data is in PHP and would prefer the data to be serialized. Fortunately, there is help! 🙂

Scott Hurring: Code: Python: PHP Serialize implemented in Python: v0.4b

This is a python implementation of PHP’s native serialize()
and unserialize() functions.

Taken along with my perl serialize implementation,
this code will enable you to transfer simple data structures
between PHP, Python, and Perl using PHP’s data serialization format.

This code aims to be a very lightweight data transport
layer between PHP, perl and python using PHP’s native
serialization format.

September 11, 2008

Pythonic grep sort of

Filed under: cool, python, tip, tool — michaelangela @ 4:00 pm

Looking for a way to crank through files and search for specific text within matching files. Found this and it’s very neat.

Python: find files using Unix shell-style wildcards « Muharem Hrnjadovic

The download though was a little hard to find so I just got the source tar.gz file from here:

http://ppa.launchpad.net/al-maisan/ubuntu/pool/main/s/scriptutil/

[update]In the end I tried grep which was significantly faster, just reading the output of the command in while looping through a list of files:

def fgrep(the_string,the_files):
    all_lines = []
    for the_file in the_files:
        command = 'grep "%s" "%s"' % (the_string,the_file)
        response = os.popen(command,'r')
        lines = response.readlines()
        for line in lines:
            all_lines.append(line)
    return all_lines

July 6, 2008

Getting rid of the dreaded “can’t encode” errors with Python and MySQL and SqlSoup

Filed under: automation, cool, dev, mysql, python, sqlalchemy, sqlsoup, tip, xls — michaelangela @ 10:47 pm

So I was getting problems inserting data that has the ‘™’ symbol in it. Brutal stuff. Couldn’t find a way around it. Did it by hand in the end. Nasty. The problem was that I was getting

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2122' in position 12: ordinal not in range(256)

where the '\u2122' is the ‘™’ symbol. There were some others as well. Finally happened upon this post which, fortunately, cleared it up. To use this tip with SqlSoup, the connection string needs a couple of extra parameters.

Instead of

db = SqlSoup('mysql://user:password@host:port/database')

use

db = SqlSoup('mysql://user:password@host:port/database?charset=utf8&use_unicode=0')

The requirement of course is that the database tables are set up with the UTF-8 charset.

But now I can thrash data at will! Mwahahahaha! Sorry… but now I can pull in unicode data from an Excel file and throw it into a database with rediculous ease. OK not rediculous ease but wow this simplifies it.

[edit]I like to use iPython to do my data thrashing with SqlSoup and SqlAlchemy. If you use the second connection string with the charset and use_unicode options, don’t reset the db connection to the previous one without those. Once doing that I couldn’t set it back and had to exit and restart the session. Bummer.[/edit]

mysql unicode issues – sqlalchemy | Google Groups

a. have use_unicode=0, set charset=’utf8′ on the connection, AND

b. ensure you have
table options = {‘mysql_charset’: ‘utf8’} when creating tables, and
all should be well.

Python Excel file reader

Filed under: cool, python, tip, xls — michaelangela @ 1:36 am

This thing is just brilliant! I am working on a project that uses PHP. But I still do my data mangling with Python because of tools like SqlSoup running under SqlAlchemy. Using those under iPython has saved incredible amounts of time. I actually like working with data in this way. It just makes it more fun. Well, I have some data in an Excel file that has multiline text in it. That can be a pain with CSV files. A little digging located a Python Excel writer which had stopped development, with the suggest to use a Python port of a Perl writer, which while cool for writing, has trouble reading… but a helpful post pointed to xlrd. A quick “sudo easy_install-2.5 xlrd” later and I was able to do:

import xlrd
book = xlrd.open_workbook("myfile.xls")
sh = book.sheet_by_index(0)
for r in range(sh.nrows)[1:]:
print sh.row(r)[:4]

Sweet. Indeed. All of the Unicode elements and multilines are intact and good to go. Good stuff.

Python Package Index : xlrd 0.6.1

Library for developers to extract data from Microsoft Excel ™ spreadsheet files

Extract data from new and old Excel spreadsheets on any platform. Pure Python (2.1 or later). Strong support for Excel dates. Unicode-aware.

June 30, 2008

How to add duration info to a FLV file with the flvtool2 command

Filed under: cool, flash, flex, flv, linux, one-liner, python, shell, tip, tool — michaelangela @ 1:37 am

Very useful. Sometimes some flv conversion tools don’t include info that is vital for playback. In this case, the seekbar wouldn’t show up in the player until after playback finished. I’ll be testing to see if this resolves it but either way, this is good to know. The first hint was this post stating that the duration metadata probably wasn’t present in the file. A quick test with flvtool2 showed that indeed there was NO metadata. Using the command shown here, the data was added. I might either wrap this up in a python wrapper to batch a directory with it, or just bite the bullet and learn more shell programming to loop through all matching files in a directory.

[edit] I guess looping in shell programming isn’t so bad:

ls -1 *.flv | while read file; do cat "$file" | flvtool2 -U stdin "$file" ; done

Add duration metadata into flv movie – Shell Script (Bash) – flv, movie, flash, duration, metadata, flvtool2

By default an flv movie doesn’t contain the duration metadata. Using the flvtool2 program it is injected like this into the movie file.

cat mymovie.flv | flvtool2 -U stdin mymovie.flv

May 13, 2008

Django, Sphinx, and full-text search

Filed under: cool, django, orm, search, tool — michaelangela @ 6:27 pm

This makes some interesting things possible. And it makes sense just looking at it. That’s even better. 🙂

Announcing django-sphinx 2.0.0 Full-text Search | David Cramer’s Blog

One of my tasks lately has been updating the django-sphinx
library to work with Sphinx 0.98, as it includes GIS components which
we have been wanting to possibly utilize on our new search engine. So
today, the conclusion of.. very few hours of work, I’d like to announce
2.0.0 of the Django integration.

The SphinxSearch manager has been completely
rewritten to rely strictly on the Python Sphinx API (a request from the
author of Sphinx), which means it should have much more compatibility
with newer versions in the future. I have also updated the querysets it
returns to work like normal Django QuerySet instances, to where they
clone themselves so you can fork your queryset into several different
actions.

May 8, 2008

Django blogging apps…

Filed under: blog, django, tip — michaelangela @ 4:05 am

that are more or less ready to use. The list is at the following link.

Empty Thoughts: Django Blogging Apps

People ask about blogging apps for the Django platform all the time. I
have not investigated each and every one of these, but so far I still
feel pretty confident that not one of these is fully baked (although
some are getting close) in any sort of manner comparable to something
like WordPress or Typo. That said, there’s a lot of great code here, and if you’re not interested in creating your own, use one of the following.

Bash and Django – working together

Filed under: bash, cool, django, shell, shortcuts, tool — michaelangela @ 4:02 am

A note… definitely something to come back to.

Talk Funnel » Blog Archive » Django bash shell shortcuts

SmileyChris had a post up recently on setting up bash aliases for Django. He uses the classic alias command which works for one-line shortcuts. I got inspired to put mine up too, but if you want to do something more elaborate, bash gives you this handy scripting language along with ’shell commands’ so you can do something a bit more involved.

What I tried to do with these shortcuts was to create a set of
mini-commands that quickly do what you need to do when developing
Django. It might help someone out and squeeze an extra 2-3.5
microseconds each time you run a Django command. Hey, it all adds up.

May 7, 2008

Snippet to update Django tables to utf8

Filed under: cool, django, orm, python, snippet, tip, tool — michaelangela @ 11:54 pm

So you get the dreaded illegal mix of collations:

Pythoneer » Convert MySQL table to utf8

Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)

What do you do? Not too bad fortunately. And if you’re working with Django, it’s a little bit easier. Launch the shell with python manage.py shell. Then in the shell do something like this:

from django.db import connection
cursor = connection.cursor()
cursor.execute('SHOW TABLES')
results=[]
for row in cursor.fetchall(): results.append(row)
for row in results: cursor.execute('ALTER TABLE %s CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;' % (row[0]))

I’m sure this can be optimized a bit but it got the job done simply enough.

Full-text search for Django

Filed under: api, django, orm, tip — michaelangela @ 11:34 pm

Just making a note…

Extending Django’s database API to include full-text search (Mercurytide)

April 25, 2008

A Django HttpRequest QueryDict is not your normal Python dict – i.e. getting ids from a friend request-form

Filed under: dev, django, facebook, python, snippet, tip — michaelangela @ 5:34 pm

Doesn’t work:

ids=request.GET['ids[]']

Does work:

ids=request.GET.getlist('ids[]')

A typical Facebook friend invite sends the ids to be invited to your site as an array in 'ids[]'. Normally, you can pull values out with just request.GET['key']. But since the value of this particular key is an array and not just a string, you can’t (at least I wasn’t able to) just do ids=request.GET['ids[]']. That returns the first value of the array as a string. Not very helpful.

But, this is a QueryDict object, not a standard {} dict. As such, Django provides the getlist(key) method.

Django | Request and response objects | Django Documentation

getlist(key) — Returns the data with the requested key, as a Python
list. Returns an empty list if the key doesn’t exist. It’s guaranteed to
return a list of some sort.

April 24, 2008

Python array copy

Filed under: cool, dev, one-liner, python, tip, tool — michaelangela @ 9:52 pm

I had heard about passing by reference vs. passing by value. Here is a case of passing by reference.

In [76]: a=[1,2]
In [77]: b=a
In [78]: b.append(3)
In [79]: a
Out[79]: [1, 2, 3]

That’s not what I was looking for. I was hoping to get 'b' to be [1, 2, 3] leaving 'a' unchanged. Oops. It passed 'a' to 'b' by reference so any changes to 'b' affected 'a'. So… need a copy. Simplest way I’ve seen of copying an array in python so far is to use [:].

Learn Python in 10 minutes | Poromenos’ Stuff

A useful use of the array range colon is to copy an array:

newary = oldary[:]

Concat Django Querysets

Filed under: dev, django, orm, tip — michaelangela @ 9:46 pm

Here is a good discussion on concatenating Django Querysets. Very helpful! It also introduced me to itertools. This leads to some other talk about itertools. Very interesting!

The jist though is like this:

How do you concatenate two querysets? – Django users | Google Groups

import itertools
q1 = Model1.objects.all()
q2 = Model2.objects.all()
for thing in itertools.chain(q1,q2):
do_something(thing)

Very handy.

April 22, 2008

Django snippets to check http_referer

Filed under: deep-linking, dev, django, snippet, tip — michaelangela @ 6:31 pm

Django snippets: Referer-checking view decorators

Here are a couple of Django decorators for limiting access to a view based on the request’s HTTP_REFERER. Both raise a Django PermissionDenied exception if the referer test fails (or a referer simply isn’t provided).

The first, referer_matches_hostname, takes a hostname
(and port, if specified) and matches it against the referer’s. If
multiple arguments are supplied a match against any of the hostnames
will be considered valid.

The second, referer_matches_re, takes a regex pattern (like Django’s urlpattern) and tests if it matches the referer. This is obviously more flexible than referer_matches_hostname providing the ability to match not just the hostname, but any part of the referer url.

Finally there’s an simple example decorator, local_referer_only, that limits a view to the current site by using Django’s django.contrib.sites to look up the current hostname.

Working with field level permissions with Django

Filed under: dev, django, snippet — michaelangela @ 5:46 pm

This might be useful but, buyer beware.

Django snippets: FieldLevelPermissionsAdmin

Have you ever needed to customize permissions, for example, allow only some fields for editing by some group of users, display some fields as read-only, and some to hide completely? FieldLevelPermissionsAdmin class does this for newforms-admin branch. Not tested well yet (&amp;gt;100 LOC!).

April 18, 2008

Neat little Facebook snippet for Django

Filed under: dev, django, facebook, python, snippet, tip — michaelangela @ 7:51 pm

You can use this to set up a Facebook User within you Django app. This of course uses the middleware which is optional as described here.

Django snippets: Little middleware that create a facebook user

  1. Install **PyFacebook** package.
  2. After make all steps in tutorial above, put this code in your app’s
    models.py module (you maybe prefer split it and put the middleware
    class in some other file).
  3. Put the FacebookUserMiddleware python-path in the
    MIDDLEWARE_CLASSES in your settings.py (after
    facebook.djangofb.FacebookMiddleware).

April 17, 2008

Django, generic views, and setting the media url via a template context processor

Filed under: dev, django, tip, tool — michaelangela @ 11:12 pm

So far I haven’t really used Django for “web sites”. It’s really been used as a data backend. Well the time has come for some web stuff. It’s set up to use a dev, staging, and production server. All is well and good. But each has a different media server. In Django, how do you tell it to use different media servers in the template?

A couple points. First, Django templates can extend other templates. So even if you have 1,000 templates (?!) but they all extend one basic template (often named "base.html") then changing the base.html updates them all. Keeps things DRY. Python is big on that and it’s good.

And to keep this process DRY, there is a context processor which can automatically provide data for your templates, i.e. a "media_url" variable. Better still, if you use Django’s generic views, any template context processor defined in your site’s settings will automatically be passed in.

James has a great writeup about this process explaining it. In fact, he even has template util library for just this very thing among others.

The B-List: Django tips: Template context processors

A recent example that came up on the django-users mailing list was a template tag to retrieve the base URL
for your media (typically you want to store “media” like images,
stylesheets and JavaScript in a particular location on your server, or
possibly even have a separate server for them if you’re using Apache
and mod_python — incurring the overhead of mod_python on a request
which will just serve up a file from disk wastes resources). Django
lets you specify where your media files come from via the MEDIA_URL setting.

…snip…

As it turns out, Django provides an extremely easy way to do this.
Every time you render a template, you give it a “context”; this is a
dictionary-like object whose keys are variable names and whose values
are the values of the variables. When you render a template with a
given context, every key in the context dictionary becomes a variable
in the template that you can access and use.

Reduce Django facebook server load

Filed under: dev, django, facebook, python, tip — michaelangela @ 2:09 am

So say you have a Facebook app and it runs on Django. Standard practice is to load the middleware. Middleware by default adds a bit of overhead. It’s needed for some bits but completely unnecessary for anything that doesn’t need it. This is one way to take it out if you feel the need for the extra speed. I wonder how much performance suffers for the bits that don’t need the middleware?

Using PyFacebook without the Facebook middleware

Recently I have been experimenting with using PyFacebook to create a Facebook app. In the early stages of development I ran into a bit of a quandry: how to you use PyFacebook with Django, without using the PyFacebook middleware?

This is not an inrrelevant question, because the PyFacebook
middleware is just bloat for any views being loaded that do not
directly communicate with Facebook, and is even inappropriate for some
views that do communicate directly with Facebook (requests made with
the FBJSM Ajax object).

April 10, 2008

Simple XML creation with Python

Filed under: cool, dev, education, python, tip, tool, web2py — michaelangela @ 10:39 pm

I was so happy when I first found Amara. It was the easiest thing I had seen for creating XML with Python. It’s quite serious stuff because it can probably handle just about anything you throw at it. Then I heard about the wonderful web2py project. One of it’s beautiful aspects is XML generation, or any tag type of thing actually. Some of that discussion is here. Thing of beauty. I wanted to use web2py but because of time restraints, my current project will continue to use Django, which definitely isn’t a bad thing. I just understand web2py more intuitively. I thought about using the TAG but I didn’t want the extra database hit when importing TAG from the gluon.html library. 😦

So I decided to search for python simple xml thinking of PHP’s SimpleXML. Results with DOM, SAX or elementtree… or anything like that… automatically disqualified. Amara was better for me than those. Then I saw pyfo…

pyfo – Easy XML Generation from Python

This package was developed by Luke Arno for Central Piedmont Community College in response to dissatisfaction with available alternatives for quickly generating XML. Concatenating strings is ugly and error-prone, using OO APIs for XML is heavy and overkill for generating simple XML output.

No kidding!

In [16]: from pyfo import pyfo

In [17]: a=(‘test’,(‘test’,’more stuff’,{‘cool’:’beans’}))

In [18]: pyfo(a)
Out[18]: u'<test><test cool=”beans”>more stuff</test></test>’

In [19]: print pyfo(a, pretty=True, prolog=True)
<?xml version=”1.0″ encoding=”utf-8″?>
<test>
<test cool=”beans”>more stuff</test>
</test>

Does it get any easier than that?!

April 7, 2008

Another python audio library that can handle FLAC

Filed under: audio, automation, cool, dev, python, tip, tool — michaelangela @ 5:18 am

Just a note about another one.

TagPy | Andreas Klöckner’s web page

TagPy is a set of Python bindings for Scott Wheeler’s TagLib. It builds upon Boost.Python, a wrapper generation library which is part of the Boost set of C++ libraries [3].

Just like TagLib, TagPy can:

  • read and write ID3 tags of version 1 and 2, with many supported frame types
    for version 2 (in MPEG Layer 2 and MPEG Layer 3, FLAC and MPC),
  • access Xiph Comments in Ogg Vorbis Files and Ogg Flac Files,
  • access APE tags in Musepack and MP3 files.

Python and FLAC audio files

Filed under: audio, automation, cool, dev, python, tip, tool — michaelangela @ 5:16 am

Kinda neat this library is. Even works on multilingual song files.

Development/Mutagen/Tutorial – Ex Falso / Quod Libet – Trac

      from mutagen.flac import FLAC
audio = FLAC("example.flac")
audio["title"] = "An example"
audio.pprint()
audio.save()
Older Posts »

Blog at WordPress.com.