grep instead of python?

I have an xml file with a listing of images that I need for a site. Typically I fire up IPython with Amara and do my xml wrangling there. But I also needed the files. I thought about using python to also grab and save the files but a little search led to this post. Ironically, while the tip to use Python is good, there was a tip to use curl to get the feed, grep to parse the images, and finally xargs to feed it to wget for downloading.

Thinking Serious » Using Python to Grab Images From a Web Site

curl -s http://99designs.com/contests/6999/feed | grep -Po “src=\”.*(png|jpg)” | grep -o “http.*” | xargs wget -q

My situation is a bit different though. There are no extensions on the files, they are in a tag, and I need to rename them with extensions. After a little Googling I used this which worked very well.

curl -s http://domain.tld/feed | egrep -o "<tag>.*</tag>" | egrep -o "<tag>(http.*)</tag>" | sed -e 's/<[^>]*>//g'
for f in *; do mv ./"$f" "${f}.jpg"; done

I still need to do some xml wrangling with Amara but the files are now just need to be moved to the right directories. Nice.

How to add duration info to a FLV file with the flvtool2 command

Very useful. Sometimes some flv conversion tools don’t include info that is vital for playback. In this case, the seekbar wouldn’t show up in the player until after playback finished. I’ll be testing to see if this resolves it but either way, this is good to know. The first hint was this post stating that the duration metadata probably wasn’t present in the file. A quick test with flvtool2 showed that indeed there was NO metadata. Using the command shown here, the data was added. I might either wrap this up in a python wrapper to batch a directory with it, or just bite the bullet and learn more shell programming to loop through all matching files in a directory.

[edit] I guess looping in shell programming isn’t so bad:

ls -1 *.flv | while read file; do cat "$file" | flvtool2 -U stdin "$file" ; done

Add duration metadata into flv movie – Shell Script (Bash) – flv, movie, flash, duration, metadata, flvtool2

By default an flv movie doesn’t contain the duration metadata. Using the flvtool2 program it is injected like this into the movie file.

cat mymovie.flv | flvtool2 -U stdin mymovie.flv

Python array copy

I had heard about passing by reference vs. passing by value. Here is a case of passing by reference.

In [76]: a=[1,2]
In [77]: b=a
In [78]: b.append(3)
In [79]: a
Out[79]: [1, 2, 3]

That’s not what I was looking for. I was hoping to get 'b' to be [1, 2, 3] leaving 'a' unchanged. Oops. It passed 'a' to 'b' by reference so any changes to 'b' affected 'a'. So… need a copy. Simplest way I’ve seen of copying an array in python so far is to use [:].

Learn Python in 10 minutes | Poromenos’ Stuff

A useful use of the array range colon is to copy an array:

newary = oldary[:]

Extract .bz2 on Linux

I am updating some python libs on my webfaction host. The python easy_setup isn’t working for this particular bit though which is unusual. It’s the first time it didn’t work. The reason? It seems it chokes on .bz2 archives. The one in particular is python-dateutil. Fortunately a quick wget later and it’s downloaded. On OS X, .bz2 archives are handled natively. I didn’t know how to do that on Linux though but this post had the answer. In this case I needed the tar xjvf variant. A later poster noted that years later it was still a useful post. This is now even later and still useful. 🙂

how do i extract .bz2 – Linux Forums

If it’s .tar.bz2, then you can use

Code:

tar xjvf cornbread.tar.bz2

to extract it all at once. If it’s just .bz2, then use

Code:

bunzip2 spankythefish.bz2

Hope this helps.

Update: I was installing Amara which I have been using for some XML handling. Installing it for python 2.5 on my host prompted the install of python-dateutil as a dependency. That didn’t happen on the Mac as I did it just now. Interesting. Also installing python-dateutil on the Mac went without a hitch. It recognized the archive type and just brought it in no problem. It probably has to do with bz2 support on the system then? But the tar xjvf can handle it on the host so something somewhere isn’t checking the file type. Interesting.

Exporting a web2py table in XML

I recently wrote about exporting a Django database to xml. As a courtesy, Massimo, creator of web2py, put together a great little example for web2py

exporting in XML – web2py Web Framework | Google Groups

def export_xml(rows):
    idx=range(len(rows.colnames))
    colnames=[item.replace('.','_') for item in rows.colnames]
    records=[]
    for row in rows.response: records.append(TAG['record'](*[TAG[colnames[i]](row[i]) for i in idx]))
    return str(TAG['records'](*records))>/pre>

Here is how you would use it:

assuming:
db=SQLDB('sqlite://test.db')
db.define_table('mytable',SQLField('myfield'))
for i in range(100): db.mytable.aaa.insert(myfield=i)




def index():
    response.headers['Content-Type']='application/xml'
    return export_xml(db().select(db.mytable.ALL))

This is using the version currently in trunk. The next update (1.28) should be out in about 1 week though so it should be possible with the release version very soon. One thing I really like about web2py’s tag handling is the TAG item. You can make any tag needed with that and it’s quite sweet.

This particular example just defines the export function, defines a database, a connection, a table, populates the table, and shows how to use it. Sweet. If I get the change, I’ll port a current Django app to web2py… but future apps will be on web2py I think.

And the server is using the wsgi server of cherrypy so it’s rockin’.

Predefined variables with lambda variables

This little line right here solved a problem. How do you dynamically assign a function to a variable to be invoked later but you need it to have a preset value? Define it when you assign it. I just didn’t know you could do that with lambda functions. Cool! This was used to update my version of the web2py shell. More on that in a sec.

p-nand-q.com : Computer : python : Stupid Lambda Tricks

f = lambda x: (lambda result=range(20):do_something(result,x))()

More list comprehensions

This may get mangled with ScribeFire. It was, and fixed in Zoundry’s Raven.

It’s a great example though of taking a list and changing it with a list comprehension. That last one qualifies as a one liner.

3.6. Mapping Lists (from Dive into Python)

>>> params = {“server”:”mpilgrim”, “database”:”master”, “uid”:”sa”, “pwd”:”secret”}
>>> params.items() [(‘server’, ‘mpilgrim’), (‘uid’, ‘sa’), (‘database’, ‘master’), (‘pwd’, ‘secret’)]
>>> [k for k, v in params.items()]
[‘server’, ‘uid’, ‘database’, ‘pwd’]
>>> [v for k, v in params.items()]
[‘mpilgrim’, ‘sa’, ‘master’, ‘secret’]
>>> [“%s=%s” % (k, v) for k, v in params.items()]
[‘server=mpilgrim’, ‘uid=sa’, ‘database=master’, ‘pwd=secret’]

Python Decorate Sort Undecorate Pattern

How do you srt a dict of items in one line?

Pythoneer » A functional task, I am sure

d = {4: 6, 12: 4, 14: 14, 18: 16, 32: 6,
    33: 24, 40: 27, 41: 6, 44: 25, 106: 2}
sorted(d.items(), key=itemgetter(1))

You can also do something similar for strings of text:

Python DSU sorting idiom » STRAY neuron

words = “This is a test string from Andrew.”.split()
sorted(words, key=str.lower) [’a', ‘Andrew.’, ‘from’, ‘is’, ’string’, ‘test’, ‘This’]