Home
Nick [entries|archive|friends|userinfo]
Nick

[ website | gagravarr.org ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Compiling for your N900 on debian [Jan. 12th, 2010|07:34 pm]
I've just got my Nokia N900, and while the phone-phone synchronisation pulled my calendar entries and contacts over from my N95 just fine, it didn't do the sms's.

Since it runs linux, the fact that the official software doesn't support it isn't the end of the world. Handily, someone's written a small sms importer program that reads in the sms's from the S60 pc-suite csv export, and loads them onto the phone.

Only snag is the pre-built version didn't support long sms's, so I needed to patch and re-compile. That means needing an arm cross-compiler, the build environment etc.


Handily, that's fairly easy to setup. The best guide I found was http://maemo-sdk.garage.maemo.org/install-debian.html, which got me almost all the way there.

One thing to note is that you need to install the "Nokia Binaries" if you want to do much development, which includes various key system components and dev libraries that aren't open source. http://wiki.forum.nokia.com/index.php/Maemo_5_SDK_installation_for_beginners has what I found to be a slightly easier guide to follow on that.


Now I'm about ready to try my newly compiled smsimporter. A couple of commands that I've found to be helpful are:
  • sb2 -eR apt-get install [pkg] - installs the given arm native package in the build root

  • maemo-sdk enter devel - enter a shell suitable for compiling for native arm


Hopefully I'll be knocking up a few more little bits of code on the weekend :)
linkpost comment

Nearly everything runs linux [Jan. 12th, 2010|07:23 pm]
My new phone runs Linux, which is great. So, I thought I'd do a quick roundup of all my computing devices, and see how close I am to all Linux:

  • Desktop - Debian Stable

  • New work laptop - Ubuntu Karmic

  • Dell Mini laptop - Ubuntu Karmic (netbook edition)

  • Media PC - Ubuntu Karmic (mythbuntu edition)

  • Nokia N900 - Maemo Linux

  • 2nd wireless access point - tomato (linux)

  • Apple TV - stripped down OSX, so Unix but not Linux

  • ADSL + wireless router - not sure, but not apparently linux

  • Nintendo Wii - not linux


I think that's about it for computing devices, so not bad all in all. When 802.11n adsl routers that run Linux come down in price a bit more, I'll ditch the current cheap'n'chearful (but with a few annoying bugs) no-name router I have for one of them, and I'll almost be there!
linkpost comment

New phone! [Jan. 12th, 2010|07:19 pm]
At long last, vodafone finally had a N900 in stock, so I have a new phone!

In general, it's proving really good. It runs linux, which is making everything so much easier (no need to hack it like you do with S60 - just apt-get install whatever you want!)

So far, I've enabled SSH, installed several extra bits and pieces, and imported almost everything over from my N95. Will shortly attempt to import the SMS's too....
linkpost comment

Now I remember why I try to avoid coding in c.... [Sep. 5th, 2009|07:12 pm]
At work, in common with most IT firms, we have a need to securely store a lot of passwords. Quite a while ago we settled on PWMan as the tool to do this with. It has a nice ncurses front end, and uses gpg encrypted xml files under the hood, so there's no worries about getting your data out if you ever want to go to anything else. After posting a few patches, several of us were given commit access, and all was good.

Roll forward to last week, and I'm now the only active maintainer of the project. We haven't had any problems with pwman for a while, but there are one or two features that people have asked for, and I figured it might be good to try to implement a few of them. The biggest thing was search (filtering results on the screen has been in there for an age, but not finding all entries anywhere in the tree). As pwman is written in c (using ncurses and libxml2 directly), there isn't anyone else at work who was comfortable making the changes, so it was either I do them or no-one would. I had a couple of hours spare, and I figured that might be enough. My hunch was 20-30 minutes were it in python, double it as I hadn't worked on the codebase lately, apply my usual "it's java not python, double it" multiplier, and hope that c and java coding times are similar. That gave me 2 hours, which seemed fine.

How wrong was I...

We're now at about 6 hours of coding, and search is almost but not quite fully implemented. Firstly I wrote the search algorithm and structures as I would in python, perl or java, only to remember at the last moment that auto-extending arrays and vanilla c don't mix, and I didn't want to have to pull in something like APR as a dependency at this stage. So, back to the drawing board, and I had to re-write the whole thing (algorithm, data structures etc) using linked lists.

Then, when I compiled and ran the program normally, and did a search it segfaulted. Compiled it without the optimisations, and with the -g flag, and ran it under gdb. No segfault, seemed to be fine. I was very bemused. Then I ran it a bit longer under gdb, and eventually managed to trigger the segfault.

At this point, I was once again reminded of why I don't like c. The bug was a simple, schoolboy error - I'd added a couple of new fields to a structure, but I'd forgotten to null one of them. Not something you need to worry about with java, python or perl, but as any C hacker will tell you, very very important there. I'd remembered for almost all of my new fields, forgotten one, and thus as soon as the memory was re-used for another similar structure (more common at the higher optimisation levels), it all went to pot :(

So, I did a full review of all the structure creating code, spotted a few other places that might need it too and added it explicitly, then all seemed fine. I do now know a lot more about using gdb than I used to, but it's all taken longer than I'd hoped for, and so I still haven't managed to find the time to write the rest of the UI. Hopefully I'll get a chance soon, and then pwman will have nice and shiny search. In the mean time, I'm reminded why lots of people end up deciding that porting non-performance critical code from c to python is worth doing when they want to add moderately complicated features....
linkpost comment

Fringe 2009 [Sep. 3rd, 2009|11:23 pm]
It's that time of year again, and so I find myself up in Edinburgh, binging on culture! As we're up for the last weekend, my reviews won't be all that helpful for other fringe goers (what with being posted just as the festival ends...), but they're rather handy for us in future years, and also when chatting to people later in the year about what we saw. So, reviews follow!

Thursday



My first show of the year was Austen's Women at Assembly George Street. This delighful mid morning show was a one woman series of monologues. The script took various parts of Jane Austen's work, as both character and narrater, exploring different ideas of what it is to be a women, and what she should do. Each character was presented very differently, and I think really enhanced the work. The result was an enthralling mix of different scenes, from scatty to serious, dry to passionate, and all engaging. A pleasant 4*

Next up was another Assembly George Street show, Party. This followed an evening as 5 passionate but clueless students endevoured to found their own political party. It was filled with great comedic moments as the students displayed their ineptitude, ignorance and misguided passion, whilst also holding up a mirror to all political parties, both new and old. Funny, cringeworthy, cutting, and certainly a 4*.

Finally for Thursday was Words with A.L. Kennedy. This started rather slowly, and made me initially wonder about the good reviews I'd seen, and whether I should leave.... However, after 10 minutes she'd got into her stride, and all was good! Her show was part autobiography, part explanation for her passion for words, and part exposition in how we think and talk. It did loose its way once or twice, but when working was excellent, both fun and insightful. Overall, 4*.

Friday



Shakespear for Breakfast is now a regular for us, since it's always good fun, is early when not that much else is on, and comes with free croissants! This year didn't dissapoint, with a slicker story than last year, based around Midsummer Nights Dream, and the usual mix of real shakespeare and fun new stuff woven together. What's more, we got to witness Alex being propositioned by a fairy, which is always good for a laugh! Solid 4*

Our only free show was Carl Sagan is my god, and Richard Feyman too. This lunchtime comedy was hosted by Robin Ince, and was in the basement of a pub on the Royal Mile (most of the cheap/free venues seem to be pubs/clubs). Robin himself gave us about 30 minutes of material, and was generally both funny and interesting. His choice of guests wasn't quite so good, but that does vary between days, and it looks like we caught a less good set. Still, it was fun, and proved that free needn't be rubbish. 3*

Next up was some dance / physical theatre in the form of Crime of the Century. This was dealing with the issue of knife crime in London, how teenages are drawn into it, and how it affects them. It was powerful and moving, but I couldn't help feeling it would've been even better if they'd concentrated a little more on either the dance, or the story. As it was, I felt the two parts were both a little underdeveloped, which was a shame. Overall, well worth seeing, a really good insight into youth knife crime, and a 4*.

Carrying on with the dance theme was Still Breathing. This reminded me a lot of the excellent dance show we saw last year, who's name escapes me... It was an all male cast, something like 12 dancers, very energetic and impressive. The music and lighting leant towards electronic and dystopian, and the dancing itself was technically very impressive, almost mesmorising. Most styles got at least a mention, but it was mostly towards energetic modern, a mix of solo through formation. Very good, a 5*

Moving on from dance, we then had some comedy in the form of Sammy J (1999). We all really enjoyed Sammy J and the Forrest of Dreams last year, so were keen to experience his new show. This year we were treated to a slightly surreal journey through his penultimate year of school, following the ups and downs of a musical nerd. We're not sure quite how much of it was autobiographical, but it could be quite a bit...! His one man delivery style worked really well, and we all had a great time listening to the surreal but heartwarming story. Once again well worth a 5*.

Finally we had Dan Antopoloski, doing his standup routine. We normally don't go in for late night comedy, but Alex had seen him before, so we made an exception. Half of his material was really good, but the rest drifted between meh and unfunny. When it worked, it worked well though, so certainly someone to keep an eye out for in future years. Overall, 3*

Saturday



We began our day at Assembly George Street with Sylvia Plath - Three Women. This was three partly interwoven stories about motherhood, being a women, life and sanity. It sometimes seemed to lack a structure, though possibly those bits were the actual life and words of Sylvia Plath... When it worked, it was very powerful and though provoking, and well acted. 4*

A big change in style for our 2nd show of the day, The Early Edition. As ever, Marcus Brigstock and friends disected and satirised the days papers, in a part News Quiz, part Mock The Week, part rant way. The guests were largely great, and it was good fun (if slightly worrying, as ever, when you hear just what is in all the papers....). Wouldn't have minded seeing it another day if we'd been up for long enough! 4*

A British Subject charts the fairly recently ended story of a British/Pakistani dual national stuck on death row for half his life. It follows a Daily Mirror journalist and his wife as he starts to cover the story, and they are so moved by his plight that they take up his cause. It's a harrowing story, and pretty much only Prince Charles comes out looking good from it, which is an unusual situation! The acting is brilliant, with a cutting script that really has you feeling like you're stood there too in the jail, even with the understated minimal props. An excellent story, excellently performed. Well worth its 5*.

Carrying on the high scoring day came Circa. This dance show covered most styles, and really pushed the boundaries of what the human body can do. The balances and acrobatics were breathtaking, as they flung themselves joined across the floor, perched one atop another at improbable angles, walked over each other in stileto heals, or performed amazing feats with hula rings. An amazing show of what can be done, beautifully performed. I'd happy have watched it a second time in awe. 5*

Finally was Year of the horse, a very surreal show of the work of the Herald's editorial cartoonist. As his haunting cartoons were shown, the increasingly bizare stories that went with them were read out, to a jarring musical soundtrack. It was a compelling, and quite different show, and you could largely forgive some of his more ranty lefty conspiracy pieces because of what he could do when he really hit the mark. 4*

Sunday



I missed our first planned show, so I kicked off the day with the Rap Guide to Evolution. This was by Baba Brinkman, who did the Rap Canterbury Tales a few years ago, and was a biologist approved rap show on evolution, in hommage to Charles Darwin. Most of the songs really hit their mark, and I even learnt a few interesting things too! Not sure it'd be enough to convince a creationist of their error, though dragging one along could potentially be good fun to watch... The songs cover the topic well, and the rap related parts worked too, giving a solid 4* performance.

Beachy Head deals with the aftermath of one man's suicide at Beachy Head. His wife struggles to understand and come to terms with it, whilst a pathologist tries to explain, and two film makers try to document it despite their secret. There's heavy use of video and lighting, which produces a very powerful show. If they'd fixed a couple of plot wholes, and tweaked a few bits of characterisation, it could've been one of the best shows on the fringe. As it is, it seems they spent a tiny bit too long on the lighting and video, which while they give a stunning show, aren't everything. So, ends up a good 4*.

Stefan Golaszewski is a Widower is Stefan Golaszewski's second show (we caught his first last year and really enjoyed it). Again, it's a one man passionate account of part of a life, with a lot of wit and humour. This time he plays an old man, looking back over his time with the love his life. It's a funny, heart-warming, moving performance, largely just him with the odd prop. However, the story doesn't quite hold your attention quite as well as last year, and the lighter elements didn't always fit quite as well. Good, but not quite as good as last time, a 4*.

We finished with a £5; fringe show, Robin Ince - Bleeding Heart Liberal. We'd been impressed enough with Robin Ince from his free luncthime show, so decided to get an hour of just him. Being his last show of the fringe, it did often wander a long way off path, but on the whole the deviations were very funny and interesting. He's not quite got the anger or drive of someone like Mark Steel or Marcus Brigstock, but his geekier side makes him a little more like a non-musical Mitch Ben. It was funny and interesting though, certainly a 4*.

Monday



We started the day with some music in the form of Brel at Breakfast. This was roughly 50% Jacque Brel songs, in a mixture of English and French, along with a number of other pieces. The singer was good, though from the between-song discussions clearly has a lot of issues... His two pianists were great fun, and good players too. Some of the song juxtapositions were a little surreal, but overall the hour passed pleasingly with good music well done, with the odd funny moment. The pastries before were very nice too! Solid 4*.

Then for a quick trip over the mound for Shappi Khorsandi (A Beginners Guide to Acting English). This was supposed to be just a book reading, but was actually part book reading, part interesting reminiscance, and part standup material. It made for an excellent fun lunchtime session, even if sometimes random or meandering. Shame we couldn't get tickets for her main show, but this was a great show in its own right too. 4*

The Other Side was a powerful set of intertwined tales in Israel and Palestine, using a minimal but well used set. It followed families on both sides through tragedy and loss, and finally a sort of understanding through phone calls and meetings across the side. The staging used the few props (mostly the metal door frames) well to create simple but powerful effects, and only once or twice overused them. However, the occasional welsh sounding liltings in many of the accents kept distracting you and pulling you out, which was a shame. Powerful, and almost worked brilliantly, 4*.

Odyssey was a one man attempted to the whole of Homer's great story in an hour, complete with some actions and sound effects. The slightly silly premise actually worked very well, and the story was well covered and engaging. Probably also not too far off how much of it would've been performed/told in ancient greece! Good fun and engaging, 4*.

News Review were back for their 30th year, and continued with their musical look into the news, both this year's and the last 30 years. Some things were great, some flopped totally, but on the whole good, and better than last year. 4*

Last year we saw one Red Room show from Belt Up, and wished we'd known in time to see more. This year, they were back again, with a partly burnt out room in C Soco, and on top form as ever. Our first show was The Tartuffe, which was supposed to be their best of last year and back even better. The audience begins sat on sofas, matresses etc in the squat in C Soco, as the company mingles and chats in character. Then the play begins, as an aragent and deranged company attempts and fails to tell the story of Tartuffe, through squabbling, mutterings, incompetance and mime. All throughout, the cast have completely smashed the 4th wall, sitting with the audience muttering about the other actors, getting audience members to play bit parts, cast-audience water fights and the rest! It's an absolutely brilliant show, in a very immersive and engaging experience. Crazy, veering wildly and filled with random references and in parts only barely recognisable, but utterlly brilliant none the less! Certainly my pick of the fringe by far,
a stunning 5*.

Our final show was another Belt Up show in the Soco squat, this time The Trial. It begins with the audience blindfolded and lead into the space one at a time, whilst the disorientating soundtrack plays, and the cast runs about. From then you join with the cast in a dark, smoke filled, random and confusing tale through Kafka's terrifying play. You're constantly moving thoughout the space to follow the action, struggling to understand as the lead character struggles with the boombing voices, smoke, light, dark confusion. However, I kind of feel they over did the immersive disorientating experience a tiny bit, and toning it down a little bit would've still kept the excellent atmosphere and effect, but allowed the story a little bit more room. Almost there, but in the end not quite as good as Tartuffe, so 4*.
linkpost comment

PostGreSQL xml extensions (pgxml / xml2) now much easier on windows [Jul. 30th, 2009|03:23 pm]
At work, we make quite heavy use of the contrib xml2 xml parsing functions in PostGreSQL (just a shame they're not in 8.4 as standard....). On linux, getting the module compiled in is fairly easy, and has been since 8.0. On windows, it used to be a bit of a faff.

I upgraded the first of our windows machines to postgresql 8.3 the other day, and start out with the usual process of part building under mingw / msys, then building just the module with mingw (see this for more details). The good news is that this method seems to work fine still. What's more, installing MinGW and MSYS has got quite a lot easier in the last couple of years :)

The even better news is that the binary windows downloads of postgresql 8.3 seem to come with the pgxml / xml2 contrib functions pre-built :)

For anyone still on PostGreSQL 8.0 or 8.1 on windows, you can download pre-built pgxml modules from http://encore.torchbox.com/pgxml/.
linkpost comment

Making sense of the flags in /proc/cpuinfo [Jul. 28th, 2009|10:26 pm]
One issue you often face when trying to make sense of the flags given in /proc/cpuinfo is that all the interesting things you can google about a cpu feature are by their full name (eg Enhanced SpeedStep) and not by their flag (eg est). So, how do you turn the cryptic short form into the googlable long form?

The first thing to try is everyone's favourite "what's in my server" tool, dmidecode . With a new enough version, it'll decode a lot of the flags for you, eg
Processor Information
        Socket Designation: Microprocessor
        Type: Central Processor
        Family: Pentium M
        Manufacturer: Intel
        ID: E8 06 00 00 FF FB E9 BF
        Signature: Type 0, Family 6, Model 14, Stepping 8
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                CLFSH (CLFLUSH instruction supported)
                DS (Debug store)
                ACPI (ACPI supported)
                MMX (MMX technology supported)
                FXSR (Fast floating-point save and restore)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                SS (Self-snoop)
                HTT (Hyper-threading technology)
                TM (Thermal monitor supported)
                PBE (Pending break enabled)

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc arch_perfmon bts pni monitor vmx est tm2 xtpr pdcm

The slight snag is that it tends not to decode some of the more interesting and recent flags - notice above that my CPU supports EST (enhanced speedstep) and yet dmidecode hasn't mentioned it. If we go to a really new xeon cpu, with flags like:
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
then the list of un-decoded bits is even longer.


For these outstanding flags that dmidecode doesn't know about, what you need is a recent set of linux kernel headers. Take a look in include/asm-i386/cpufeature.h (assuming your kernel is new enough to list all the flags that is!), you'll see things like:
#define X86_FEATURE_LM          (1*32+29) /* Long Mode (x86-64) */
#define X86_FEATURE_XMM3        (4*32+ 0) /* Streaming SIMD Extensions-3 */
#define X86_FEATURE_EST         (4*32+ 7) /* Enhanced SpeedStep */
#define X86_FEATURE_LAHF_LM     (6*32+ 0) /* LAHF/SAHF in long mode */

So, if we wanted to know what the est flag mean, we can see it's Enhanced SpeedStep, and google that (googling EST is largely unhelpful!). Likewise, we can see that the even more cryptic lahf_lm tells us LAHF and SAHF are enabled in 64 bit mode (long mode = 64 bit, as a few lines above explains), though you will still have to know that LAHF is a kind of cpu instruction...
linkpost comment

CPU power management with linux and recent CPUs [Jul. 28th, 2009|10:05 pm]
Intel (and AMD) cpus have been supporting some power management for some time now, especially on the mobile focused cpus. It used to be a slightly fiddly thing involving acpi states, and didn't always deliver quite the savings one might want. However, in the last few years, there are a couple of interesting new CPU features (and matching linux kernel code) which aim to deliver some fairly decent power savings when the cpu load is idle. Handy for making your laptop batter life go longer, but also very handy for cutting your datacenter power bill!

First up, can your x86 / x64 cpu support the more interesting new cpu power features? The answer hides in the flags section /proc/cpuinfo if you know what to look for. The two main ones of interest are tm2 and est. My core 1 laptop cpu supports both, all the core 2 laptops I've come across do too, all my xeon powered servers from the last 6 months do, and a smattering of the ones from the 12 months before that do too. Alas none of my other servers do, as it was slow to move from the laptop cpus to the desktop and server ones.

Of the two flags, est (Enhanced SpeedStep) is the more interesting, as tm2 (Thermal Monitor 2) simply delivers a better way for the CPU to slow down (and draw less power) when it gets too hot. Important to have, but ideally not something you'll normally be using! (As an aside, I'd recommend reading this article if you want to know more on EST, TM1, TM2, and especially see pretty graphs of what happens to the cpu throughput when you stop the fans etc).


With Enhanced SpeedStep (est), you can control the clockspeed of the cpu, picking the suitable one (from a list, the exact number of steps depending on the cpu model), so you can trade between power use and the amount of work that can be done every second. With the right modules loaded on linux, the cpu will throttle back its clockspeed when the load is low. As load increases, the clockspeed is increased to meet demand, and when load drops, the cpu is scaled back again. Magic!

In order for this to work, we need three sets of modules loaded. The first is the basic acpi interfaces to the processor, then the base speedstep modules (which delivers us an exciting set of entries under /sys/), then finally the ondemand governor (which does the magic of changing the cpu speed automatically with load). While you can go for a userspace governor, and change it yourself / via a daemon, almost everyone recommends that when using a newer kernel, you just let the ondemand governor do its thing, as so much work has gone into making it "just work".

The modules you'll probably want are: processor acpi-cpufreq cpufreq_ondemand cpufreq_userspace cpufreq_conservative cpufreq_powersave . The latest RHEL 4 kernels (2.6.9-89) seem to have all of these backported, RHEL 5 certainly has them all, and debian stable and ubuntu have them in too. If you're not sure, I'd say go with at least a 2.6.2x kernel so you can ensure you get all the best new bits.

Loading them on RHEL 4, I generally sling the magic lines in /etc/rc.local. For RHEL 5, I normally create a new file /etc/sysconfig/modules/powersaving.modules and put them there (plus make it executable). For Debian and Ubuntu, I create a new file /etc/rc.boot/power-management, make it executable and put the commands there. The snippet of shell code you'll want is:
for i in processor acpi-cpufreq ; do
        modprobe $i >/dev/null 2>&1
done
for i in processor acpi-cpufreq cpufreq_ondemand cpufreq_userspace cpufreq_conservative cpufreq_powersave; do
        modprobe $i >/dev/null 2>&1
done
echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

With those commands in and run, you should discover the directory /sys/devices/system/cpu/cpu0/cpufreq , and if you have multiple cpus, /sys/devices/system/cpu/cpu1/cpufreq etc. Within those directories are several files that tell you about your cpu scaling, and with usermode, let you check it. scaling_available_frequencies gives the list of the different frequencies your cpu can run at (I have one machine with only the one speed, which is really rather helpful, but most have 4 or 5 different speeds). scaling_min_freq and scaling_max_freq tell you the outer bounds of the scaling range. scaling_available_governors lists the governors available (largely dependent on the loaded modules), and scaling_governor lets you get/set the current one. As I've mentioned before, go for ondemand unless you have a really strong argument otherwise!

Let's see it in action, with a slightly made up test:
[root@grenache cpufreq]# cat scaling_available_frequencies
2000000 1667000 1333000 1000000 
[root@grenache cpufreq]# cat cpuinfo_cur_freq
1000000
[root@grenache cpufreq]# cat /dev/sda | gzip -9 > /dev/null &
[1] 22983
[root@grenache cpufreq]# sleep 30
[root@grenache cpufreq]# cat cpuinfo_cur_freq
2000000
[root@grenache cpufreq]# kill 22983
[root@grenache cpufreq]# sleep 10
[1]+  Terminated              cat /dev/sda | gzip -9 > /dev/null
[root@grenache cpufreq]# cat cpuinfo_cur_freq
1000000


Tada! Our machine started out idling at low power + cpu frequency, detected the rise in demand and sped up, then throttled back when the demand was gone. If you have a plug in power meter, you can watch the power use change accordingly too :)


Oh, and you might find some older howtos talking about things in /proc/acpi/processor/CPU0/. If your cpu supports EST, then you almost certainly don't want to be using the old acpi interface, as it doesn't let you control the new EST features, and the older states generally don't work as well. You'll only want the acpi settings if you have an old (probably mobile) cpu that only supports the original SpeedStep.


(note - I haven't gotten my hands on one of the new i7 CPUs to check what happens with them. Some googling indicates they support est and tm2, but I'm not sure if the new Turbo Boost and Zero Power features will be controlled transparently by the above, or if you'll need some new additional tweaks for them)
linkpost comment

Is it a 32 bit or 64 bit cpu? [Jul. 28th, 2009|02:15 pm]
One might expect that a good sysadmin will always know the answer to this question, but I've found myself asking the question quite a bit lately. Sometimes it's on a machine that's not been used in a bit, or sometimes on a machine that we're thinking of re-purposing. Sometimes, you've just been given a login to someone else's box to install some software for them. Or, in the case of my new mythtv machine, simply because I didn't believe what the internet suggested...!

So, we have two questions we actually want an answer to - does this machine support 64 bit (or only 32 bit), and is the running linux kernel support 64 bit (or is it only a 32 bit kernel)?

For the first question, cat /proc/cpuinfo will tell you all you need to know, albeit somewhat cryptically. Check the flags on your processor, you're looking for lm . If you see lm in that list (lm = Long Mode), then your cpu supports 64 bits, and you can run a 64 bit kernel if you'd like to. If doesn't have lm, but it supports cmov (Conditional move instruction), then it's only a 32 bit cpu, but you can run a i686 kernel. If you don't even see cmov, then it's a fairly old / basic cpu (eg an older low power via), and you'll need to run an i386 / i486 kernel

If we do find the lm flag, then how do we tell if the running kernel is 32 bit or 64 bit? Possibly uname -a will give us a hint, but alas not always (look for x86_64 in the output for 64 bit, or i686 for 32 bit). If that isn't conclusive, then run getconf LONG_BIT - that should return either 64 or 32, that being your answer.


If you're on 32 bit, then there's also the question of if your machine has PAE (Physical Address Extension) turned on. While turning on PAE won't let you get past the 4gb per process memory limit that drives so many Java users to 64 bit, it will at least let you have multiple 4gb chunks of memory, and use up to 64gb.

The easiest thing to check is uname -a - if you get an answer back like 2.6.18-128.2.1.el5PAE then it's fairly clear :) Next up, if your distro ships the config file used to build the kernel, then check for CONFIG_HIGHMEM64G=y - if you see that, you've got PAE. Failing that, try running "free" - if it reports more than about 3.75gb, you're on PAE! Finally, if it reports under that, run dmidecode and add up all the memory it detects. If it shows 4gb or more and you don't see that with free, then you don't have PAE. If dmidecode reports you have under about 3.5gb of real memory installed, then it doesn't really matter either way!


(note - all of the above is only applicable for x86 / x64 cpus, and won't apply to sparc, arm etc)
link2 comments|post comment

Oxford Geek Nights 13 [Jul. 19th, 2009|06:01 pm]
This Wednesday saw the 13th Oxford Geek Night, which I think was my 10th. The talk about graphing, public data and timetric.com was really interesting, and it's a shame the keynotes are only 15 minutes as I could've happily listened for longer. The HTML 5 one was good too, letting everyone know what's been going on since we all got bored and stopped following the comings and goings in detail as many of used to. Really must read up on XHMTL 5 when I get the chance! Some great microslots too, do check out the videos if you missed it :)

Pleasingly, the one free drink per attendee was back for this one (Google used to sponsor it, but sponsorship cuts meant it stopped a little while back). This time, the nice people at the Guardian Open Platform bought our beer, so a big thankyou to them!

The slightly hacky internet from a netbook, a 3g dongle and a fon router is getting quicker to put together, which is good as it looks like the previous pub-supplied wifi is gone for good. We now have a midly terrifying shell script to do the bulk of the setup, then all you need to do is bluetak the fontenna to the wall sufficiently high up, and find a safe spot for all the kit. Speed this time was bad as T Mobile sucks for reception, but we have a plan for next time (involving a number of 99p PAYG sim cards and an almost scientific signal strength test, followed by aquiring the appropriate PAYG 3g dongle for the winning network), so with any luck OGN14 will be back to fastish internet for all your tweckling....!

(As an aside, has anyone else noticed that T Mobile's reception is decidely rubbish anywhere an Oxford geek might want to use it? Firstly there's the lack of 3G coverage in most of Jericho - we were stuck on 2G / GPRS which is why the internet at geeknights was quite so slow. Their coverage along the Oxford - Reading railway line is almost non existant, and between Oxford and Charlbury they make Vodafone's patchy coverage look positively comprehensive. Finally this weekend I've noticed an almost complete absense of coverage on the Birmingham-Newcastle-Edinburgh route when not in a station, while my vodafone mobile manages more than half the time. All very dissapointing.)
linkpost comment

Fun with my Zotac Ion powered Mini-ITX Box [Jul. 9th, 2009|07:19 pm]
Last week, I finally got around to ordering my new small computer to hide under the TV. In the end, I opted for the higher-end machine - a Zotac Ion A motherboard bought from the lovely people at mini-itx.com. (The Zotac Ion A is a dual core atom CPU, with the NVidia Ion chipset powering it. Slightly higher power than the Intel board I'd been considering, but it has HDMI output, and really good hardware accelleration for video).

The motherboard (and case, memory, disk etc) all turned up Monday, and I've been playing with it off and on since then. I think this evening I'm almost there with it :)

First up, I opted for MythBuntu 9.04, as quite soon I'm planning to add USB tv tuners to it, and use it for TV watching. Having no DVD drive at the time (it only arrived today), I tried to do some hacky install using a usb stick and dd'ing the iso image on the hard disk. My advice now would be not to bother, and just boot mythbuntu in another machine, and use the tool on that to make a bootable usb stick to install from. Would've been much less effort if I'd done that originally...

Next up, the Ion chipset is too new for the default nvidia drivers. You need at least the 185 series ones, and maybe even 190. Go add the avernard repository to apt to get the newer drivers. With those, you get shiny hdmi etc, all good.

For the video playback accelleration, you need the vdpau libraries, and mplayer compiled for vdpau, and the right mplayer config (this last one caught me out!). Install nvidia-185-libvdpau and mplayer from avernard and you'll be most of the way there.

Next, go to here and grab qvdpautest, which'll let you test that your vdpau accelleration is working properly. You should get some numbers something like:

Intel(R) Atom(TM) CPU  330   @ 1.60GHz
NVIDIA GPU ION (C79) at PCI:3:0:0 (GPU-0)

VDPAU API version : 0
VDPAU implementation : NVIDIA VDPAU Driver Shared Library  185.18.14  Wed May 27 03:09:54 PDT 2009

SURFACE GET BITS: 287.579 M/s
SURFACE PUT BITS: 228.802 M/s

MPEG DECODING (1920x1080): 56 frames/s
MPEG DECODING (1280x720): 136 frames/s
H264 DECODING (1920x1080): 52 frames/s
H264 DECODING (1280x720): 114 frames/s
VC1 DECODING (1440x1080): 63 frames/s


DVD playback at 720p at 136 frames a second, yey! (My TV only does 720 alas). Finally, you need to tweak your mplayer config. By default, it won't use vdpau output, and more crucially, it won't use the vdpau video codecs. Your ~/.mplayer/config file should look something like:

ao=alsa

# HDMI doesn't do volume control, only on/off
softvol=true
softvol-max=200

# Should always work
#vo=xv
# Shiny nvidia accelleration
vo=vdpau

# Prefer vdpau
vc=ffh264vdpau,ffmpeg12vdpau


Finally, try watching a h264 video file (eg a HD show from BBC iPlayer) and watch it be all smooth and only at 10% CPU! If mplayer runs at 100% cpu and keeps saying your computer is too slow, then you're not really using vdpau, so check again.



Oh, and sound. I wanted to use the HDMI port for sound output, which wasn't the default. Run aplay -l to see the order of the devices, mine was

**** List of PLAYBACK Hardware Devices ****
card 0: NVidia [HDA NVidia], device 0: ALC662 Analog [ALC662 Analog]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: NVidia [HDA NVidia], device 1: ALC662 Digital [ALC662 Digital]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: NVidia [HDA NVidia], device 3: NVIDIA HDMI [NVIDIA HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0


In order to make the HDMI output the default, create a /etc/asound.conf file along these lines:

pcm.!default {
    type hw
    card 0
    device 3
}
ctl.!default {
    type hw           
    card 0
}


Now, run alsamixer and un-mute the digital channels at the far end, and sound should come out nicely through your TV just like it does on mine! :)

Next up, getting those pesky USB DVB-T dongles working...
link8 comments|post comment

Deciding on a new media pc [May. 17th, 2009|06:17 pm]
At the moment, I have a fairly new but increasingly temperamental DVD player, a dual tuner PVR that's about 3 years old with a number of annoying bugs, and an increasing tendency to not record things / eat old recordings, and a very nice but surprisingly power hungry desktop on the other side of the room. After quite a bit of pondering, I don't think it's sane to amalgamate the three, so I'm looking into getting a new media pc to go under the TV.

Having played with MythBuntu, I'm fairly sure that that'll be the right thing to run on it. An Atom based Mini-ITX system looks to be the right mix between power use, features, cost and size. Since it'll also be a real box running linux, I'll be able to move all the networking services onto it from my desktop. Couple that with some wake-on-lan magic that the desktop seems to support, and I should be able to put the desktop + screens to sleep when I'm out, ssh into the media box and wake it up the few times a week when I'm out but need to access it. All looking like a plan.

Next up, I need to pick the motherboard for the media pc. There seem to be three main choices. First, there's the current intel mini-itx atom motherboards, which come a really low power processor, but a really power hungry chipset, which does rather seem like a waste... Oh, and they don't seem to do dvi/hdmi, which I'll want for going into the TV. Then there's the new D945GSEJT from Intel, which has the new low power chipset, dvi support, but only a slower, single core atom. Oh, and it isn't for sale quite yet over here (it only came out in the states in the last week or two), but seems to go for about $120 over there. Finally, there's the NVidia ION atom motherboards, especially the Zotac Ion. These have DVI and HDMI, come with the dual core atom option, have wifi and a few more ports, but draw a bit more power (about the same as the older intel boards though - 21w single core, 25w dual core), and cost more (£140 for the single core, £180 for the dual core). They should be on sale over here in a week or so.

I figure the older intel boards are out, so it's a toss-up between the newer intel board, and the nvidia one. Do I want a slightly cheaper, slightly lower power, slightly less powerful intel one, or the the more powerful nvidia?

Oh, and then I need to pick a case, pick a harddisk for tv recording (can't just use my desktop as that'll soon be asleep much of the time) and decide between pci and usb digital tuners. Choices, choices....
linkpost comment

Creating a 8192 bit GPG key to replace my 1024 bit one [May. 9th, 2009|05:00 pm]
As some of you will have heard, there's been some new research into the security of the SHA-1 hash, which has found a faster way to locate collisions. Couple that with the NIST Guidance that US Government agencies should transition away from SHA-1 to stronger hashes by 2010, and you have a fairly strong reason to migrate away.

My GPG key from 2003 to know has been a 1024 bit DSA key, which uses a 160 bit SHA-1 hash. As per the Debian Guidance, this isn't ideal, and I (along with many) have decided to move to a new, stronger key with a stronger hash.

Bruce Schneier, in Applied Cryptography, has recommended a 8192 bit key if you want it to have a useful lifetime beyond 2015. Since changing GPG keys is a faff (new keys, revocations, telling people, attending PGP key signing parties etc), I decided to go for a nice long key this time. By default, GPG will only create keys up to 4096 bits long, but it's happier to work with larger ones if created. The reason for this, if you google, is that they think that for most people going to 8192 bits will just make things slower, without delivering much more security, so they feel that anyone who wants a longer key should have to think about it, and explicitly enable it themselves.

I decided that given the faff of a new key, the relatively infrequent use my key gets (most just code signing and the odd signed email for voting/contracts etc), going straight to 8192 bits for my key was a sensible choice, even given the slowness vs 4096. So, I followed the GPG list's guidance, grabbed the source code, and upped the limit for myself.


To do this, I downloaded GPG 1.4.9 from my nearest GPG mirror, and unpacked. Line 1516 of g10/keygen.c holds the default and maximum key size for generation. I upped the limit from 4096 to 8192 and compiled.

Next, you need to alter your hash preferences in GPG. This needs to be done before you generate the new key! As per the debian article, edit your ~/.gnupg/gpg.conf file and add in two more lines to set your hash to SHA256. You can do this with:
cat <<~/.gnupg/gpg.conf >>EOF
personal-digest-preferences SHA256
cert-digest-algo SHA256
EOF


With my hashing preferences changed, and a custom version of gpg now sat in the g10 directory, I ran ./g10/gpg --gen-key. First I selected a key type of RSA (sign only), as DSA+Elgamal is capped at 1024 bit key + 160 bit hash, and apparently the larger DSA keys are less well supported. Next, I entered a key size of 8192 bits, which my custom version of GPG allows me to.

After picking an expiry time, entering my name and email, and picking a nice secure password, it was time for the key to be generated. As generating a 8192 bit key takes a lot of entropy, don't expect it to happen quickly, and use your machine in the mean time to generate more entropy for it!

With my signing-only 8192 bit key created, it was time to add in my two extra sub-keys so I could use it for everything. GPG had output the new key id, so I edited the key (gpg --edit-key [id]). Running addkey, I first added a RSA (sign only) 8192 bit key, and then I ran addkey again to add a RSA (encrypt only). Both key generation processes take a while, so be patient!

At this point, I had a 8192 bit RSA PGP key, with signing and encrypting subkeys on it. Next I set the preferences to request stronger hashes if possible, by running
setpref SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
(this magic list comes from the debian docs). Finally, I saved my new key, and took a backup of it.

The last couple of steps were:
  1. Sign my new key with my old key, to show it's really my new one
  2. Upload my new key to a keyserver
  3. Generate a transition notice, signed by both my old and new keys. This tells everyone my new key's details, and is a handy + secure document to point people at
  4. To sign the notice, run gpg -a -u [old id] -u [new id] --clearsign transition.txt
  5. Put my new public key into the KEYS file for the open source projects I release code for
  6. Printed out a few little bits of paper with my key id and fingerprint on, for key signing events
  7. Emailed a few key friends to ask them to sign my new key, pointing them at the transition doc, and following up with a quick SMS to confirm it's really me
  8. Wrote this doc up for them to follow :)


There we are, and I have a shiny new 8192 bit PGP key. It doesn't take too long to do (maybe 20 minutes elapsed, 15 minutes work including printing and cutting up fingerprint slips), and my key + signing is hopefully now secure for another 5+ years!
link5 comments|post comment

Trying and failing to upgrade my AppleTV hard disk non-destructively [Apr. 12th, 2009|08:31 pm]
I bought my AppleTV a year or so back, and I've found it a really handy way to watch TV shows I missed / aren't on freeview. Unfortunately, I've been using it so much, that it had got full!

Most of the AppleTV upgrade guides (like this or this) assume you're an iTunes user, so after getting your new hard disk running, you'll just re-sync all your content from iTunes. However, being a Linux user, I don't have iTunes, and the great thing about the AppleTV was that it let me buy stuff from the iTunes store without needing to use windows.

So, I decided I wanted to try doing a non-destructive upgrade of the hard disk in my AppleTV, using only linux (so guides that require a mac and iPartition were out). Here's my tale...

Oh, and in case it matters, my AppleTV was running 2.0, but shipped with 1.0, had a 40gb disk before, I've bought a 160gb new disk, and I'm using debian stable (lenny). I've only got one usb hd caddy, so I needed plenty of disk space on my machine.




First up, took out the old hard disk, and took two backups of it. One was of the whole disk (/dev/sde in my case, ymmv), and another of each partition on it (/dev/sde1 - /dev/sde4, you should see all 4 if your kernel has GPT support, which you'll need)

The first thing I tried was using a GPT enabled version of parted to create 4 partitions on the new disk, the first three being identical to the EFI, Recovery and OSBoot ones on the old disk. The 4th was a new hfs+ Media partition to fill the rest of the disk. I then dd'd the 1st three partitions onto the new disk, and created a new hfs+ (non journaled) partition for the 4th, and cp'd the content from a mounted copy of the old media disk. This appeared to work fine, but the AppleTV wouldn't boot properly. It would fail to boot a few times, go to the recovery mode, and fail to do a factory restore if requested. This apparently simple route fails :(

Next, I tried dd'ing the whole of the old disk onto the new one. The AppleTV booed fine, played my content fine, but didn't see any of the new disk space (as one would expect). Then, I did a factory reset. This worked, and got me back to 1.0, and with an empty media disk, but still only using the 40gb.

With the disk in this state, I then tried using parted to create a new media partition. It warned me about the GPT not filling the disk, but after fixing that, was happy to create a ~150gb media partition to fill the disk up. mkfs.hfsplus ran fine on the new partition, and the AppleTV booted happily. It was then running 1.0, but with 150gb of free space, but no content.

This seemed promising, so next I tried restoring the OSBoot partition again (by dd'ing my save of the 3rd partition back over the top of the 3rd partition on the new disk). It booted into the recovery mode which worried me, but diagnostics passed fine, so I rebooted it. This (2nd) time, the AppleTV booted fine, and I had 2.0 with 150gb of free space (no content)

My next step then was to take the disk back out, mkfs.hfsplus to re-create the media partition un-journaled (the AppleTV had turned it into journaled, and Linux won't then write to it). I mounted this under linux, and did a "cp -ar" of all the content from the backup of my media partition onto the new one. This booted fine, and showed the extra space. All my content showed in the list. However, while my tiny bit of iTunes sync'd content played fine, all my AppleTV purchased stuff wouldn't play, giving the error "No Content Was Found". The icons for the purchased content had gone too. My best guess was that the factory reset wiped the keys.




So, I thought I'd start again, and see how far I could get before loosing access to purchased content. dd'ing the whole of the old disk over worked fine - no new space, but all my content. Next, after running parted to fix the GPT, it was all the same.

The test then was to use gpt to create a new larger media partition, mkfs.hfsplus to create a new fs on it, then to cp all the content over from the old disk's media partition. This results in an AppleTV that'll boot, see the new space, but refuse once again to play purchased content.

Now, the very odd thing is that if I use the file browser (ATVFiles) to browse to /mnt/Media/Purchased Files/, then I can play those just fine. So, whatever the required details are to play purchased files, they are still on my AppleTV.

I tried upgrading to the latest AppleTV software (2.3.1), to see if that might fix things. Alas no change though :(
linkpost comment

Django 1.0.2 and Google App Engine [Apr. 8th, 2009|02:46 pm]
It looks like my next project at work will involve a Google App Engine hosted Facebook application.

As I'm a big fan of Django, I figured I'd try to use quite a bit of django in the app. App Engine does ship with Django, but an older version, and it's a little fiddly to tie in all the way down.

Luckily, there's a handy bit of glue code to tie it all together, the cunningly titled Google App Engine Django. After a little bit of prodding, and using zip files to get around file limits, I have managed to get it playing nicely together. The steps roughly are:

  • Make sure you have a new enough copy of the app engine python SDK (1.2.0 or later)
  • Symlink the sdk to .google_appengine in the root of your application
  • Create a /lib/ directory in your app to hold things
  • Download Django 1.0.2, and zip the django directory to /lib/django.zip (zip -r ~/code/myproject/lib/django.zip django/)
  • Do a svn checkout of google-app-engine-django (http://google-app-engine-django.googlecode.com/svn/trunk/)
  • Zip the appengine_django directory to /lib/appengine_django.zip (zip -r ~/code/myproject/lib/appengine_django.zip appengine_django/)
  • Grab app.yaml, main.py, manage.py and urls.py from google-app-engine-django, and put them in the root of your application
  • Update app.yaml with your app's name
  • In main.py, after "import logging" but before the first of the appengine_django imports, add in:
    # Put our zips onto the path before loading anything else
    sys.path.insert(0, 'lib/appengine_django.zip')
    sys.path.insert(1, 'lib/django.zip')

  • (More info on why you want to import from zips, and how available here)
  • Do the same sort of thing to manage.py, by adding before the first import
    # Load in appengine_django and django from the packaged zips
    import sys
    sys.path.insert(0, 'lib/appengine_django.zip')
    sys.path.insert(1, 'lib/django.zip')

  • Start dev_appserver.py on your application
  • Begin coding your almost-django app engine application!
link3 comments|post comment

Building your own Aggregate Functions with PostGreSQL [Apr. 2nd, 2009|04:00 pm]
This used to be on the Torchbox Blog, but got lost in the site re-launch. As it's really handy, I've re-posted it here

In SQL, aggregate functions are powerful things that act upon a number of rows, and output a single value from them. With PostGreSQL 8.x, it's now possible to easily define your own aggregate functions, to aggreate your data however you need. One use of this is to generate powerful reports very easily.

Let us think of a very simple (if slightly stilted) database:
CREATE SEQUENCE s_types;
CREATE TABLE types (
id INTEGER NOT NULL DEFAULT NEXTVAL('s_types'),
name VARCHAR(16) NOT NULL,
CONSTRAINT pk_types PRIMARY KEY(id)
);
INSERT INTO types (name) VALUES ('coding');
INSERT INTO types (name) VALUES ('design');
INSERT INTO types (name) VALUES ('sysadmin');
CREATE SEQUENCE s_customers;
CREATE TABLE customers (
id INTEGER NOT NULL DEFAULT NEXTVAL('s_customers'),
name VARCHAR(255) NOT NULL,
CONSTRAINT pk_customers PRIMARY KEY(id) 
);
CREATE TABLE sales (
customer_id INTEGER NOT NULL,
sale_date timestamp NOT NULL DEFAULT now(),
type_id INTEGER NOT NULL,
CONSTRAINT fk_sales_customer FOREIGN KEY ("customer_id") REFERENCES "customers" ("id"),
CONSTRAINT fk_sales_type FOREIGN KEY ("type_id") REFERENCES "types" ("id")
);
CREATE INDEX sales_customer ON sales (customer_id);

All goes well with our database for a while. Someone asks for a report of the first and last sale for each customer, and we're able to produce that directly in SQL, with just one result row per customer. Then they ask for something fancier - they don't just want to know the first and last sale date for every customer, they also want the list of sale types too. The snag is they only want one line per customer, and ideally we don't want to have to do all the heavy lifting work in our own code.

 The first and last sale dates are easy:

SELECT id, name, MIN(sale_date) AS first_sale, MAX(sale_date) AS last_sale FROM customers INNER JOIN sales ON (id = customer_id) GROUP BY id,name

 id | name     | first_sale          | last_sale           
----+----------+---------------------+---------------------
1 | Torchbox | 2007-01-03 10:02:11 | 2007-10-06 09:44:21

But we also want an aggregate function that'll give us the list of sale types. We have a look through the list of standard postgresql aggregate functions, but we don't find anything that'll effectively turn a sequence of strings into a list or an array. After a quick google, we can't find a list aggregate or array aggregate function anywhere.

Luckily, it's possible to define your own PostGreSQL aggregate functions, without needing to resort to coding them in c (which is possible, by the way, just requires some work). We can ask PostGreSQL to build us a new aggregate function, simply by gluing together normal functions to act on the data as it goes past

For our use case, we want something that takes a sequence of strings (but possibly also numbers for a similar use), and returns us an array of the distinct values at the end. The distinct bit is easy - just do func(DISTINCT val) rather than just func(val) and PostGreSQL will do it for us. For the rest, we'll want a custom aggregate function that starts with a new array, and does an array append with new values. Say, something like:

CREATE AGGREGATE to_array (
sfunc = array_append,
basetype = anyelement,
stype = anyarray,
initcond = '{}'
)

We can now get our list of customers, their first and last sale, and all their sale types, all with one row per customer, all in sql:

SELECT customers.id, customers.name, MIN(sale_date) AS first_sale, MAX(sale_date) AS last_sale, to_array(DISTINCT types.name) AS sale_types FROM customers INNER JOIN sales ON (customers.id = customer_id) INNER JOIN types ON (type_id = types.id) GROUP BY customers.id,customers.name

 id | name     | first_sale          | last_sale           | sale_types       
----+----------+---------------------+---------------------+------------------
1 | Torchbox | 2007-01-03 10:02:11 | 2007-10-06 09:44:21 | {sysadmin,coding}


If we wanted to exclude null entries from going into our array, then we'd use a similar new aggregate function, with a custom function to power it:
CREATE FUNCTION array_append_not_null(anyarray,anyelement) RETURNS anyarray
  AS 'SELECT CASE WHEN $2 IS NULL THEN $1 ELSE array_append($1,$2) END '
  LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT;
CREATE AGGREGATE to_array_not_null (
sfunc = array_append_not_null,
basetype = anyelement,
stype = anyarray,
initcond = '{}'
)
There are lots of other possibilities too, and PostGreSQL provides loads of handy functions to build up your new aggregate with. If we start using our own functions to build up new aggregates, then the possibilities are almost endless!

Two handy links from the official postgresql documentation are Custom Aggregate Functions and Built-In Aggregate Functions
linkpost comment

ApacheCon Europe 2009 [Apr. 1st, 2009|06:41 pm]
Last week I was out in Amsterdam for my 4th ApacheCon Europe. Once again it was an excellent conference, with loads of interesting talks and people.

The Monday kicked the conference off with a BarCamp unconference. Despite only having the one room (ideally you'd have a few, and a breakout space), it went very well. We had a load of interesting talks and discussions, and it was a good learning experience ready for BarCamp Apache Oxford, which I'm helping to organise this coming weekend.

Wednesday's main theme was Hadoop and big data. In fact, one could say that the biggest theme and buzz of the whole conference was on open source technologies for large scale and cloud computing, and there certainly were a large number of excellent talks on these topics. The talks on Pig, HBase and CouchDb were very intersting on how you can store and query very large data volumes, and there were also good talks on building, scaling and dynamically managing your hadoop and similar clusters.

On the text/search side, the Lucene talks seemed even better than last year. Solr had a bigger presence than before, and the new clusterable, machine learning project Apache Mahout produced a lot of buzz.

Thursday saw the first of my talks, on the Django web application framework in python. I've been using this for several years, and really like it, so it was great to have a standing room only crowd come to the talk. Admitedly it was in the smaller room, but still, it's nice to see people standing and sat in the corridoor just to hear your talk!

I gave my second talk (well, 2nd proper talk, if we exclude the barcamp and lightning ones) on the Friday, covering the new features in my own project, Apache POI. A friday afternoon slot is always a harder one, as people are already heading off, so it was good to see a decent crowd staying on to listen. Many people seemed pleased to learn about our new features, which is always nice.

My slides should be going up on the main apachecon site shortly (they're already up in the all-slides bundle, but not yet individually), so in the mean time you can download the slides and sample django app from http://people.apache.org/~nick/Talks/ApacheConEu09/
linkpost comment

Lack of posts of late [Jan. 31st, 2009|10:06 pm]
In case you were wondering why I haven't posted here in a while, and why I probably won't be doing much posting here in the near future, it's a question of a miss-match between when I feel like writing, and when it's easy to post. With various things in my life over the last year or so, the times when I have been in the mood to post have become less frequent. More importantly though, these instances have never coincided with when I'm near a computer, and normally also when I'm not in a position + inclination to use my phone. When I have been in a suitable position to post, I haven't felt inclined. From this, out drops a lack of posts.
linkpost comment

Giving in to facebook [Jan. 31st, 2009|10:02 pm]
Today, I have finally given in, and joined facebook. So, all those of you who've been bugging me about it for ages can finally be happy!

To explain why I haven't been on facebook so far, one needs to go back quite a long way in time. I was on livejournal pretty much before the term "social networking" had come into being, and it worked well. Then, when the first generation of dedicated social networking sites came along, I joined most, and watched them burn brightly for a few weeks, then fade away.

So, when facebook came along, I decided I'd sit this one out, and wait for the next one to come by, which hopefully would stay around for a while. As time went by, I kept hanging on for facebook to fade away, though I have finally had to come to the conclusion that it probably isn't any time soon. With this in mind, I've finally joined, on the grounds that waiting for the next one is going to be quite a long wait after all!
linkpost comment

Fedora + mplayer being unhelpful [May. 17th, 2008|07:49 pm]
I've been having trouble lately with playing DVDs on my laptop, which runs Fedora Core 8. No matter if I tried mplayer, xine or vlc, I got the same errors in dmesg when trying to play video DVDs:
Buffer I/O error on device sr0, logical block 321
Buffer I/O error on device sr0, logical block 322
Buffer I/O error on device sr0, logical block 323
Buffer I/O error on device sr0, logical block 324
Buffer I/O error on device sr0, logical block 325
Buffer I/O error on device sr0, logical block 326
end_request: I/O error, dev sr0, sector 1464
end_request: I/O error, dev sr0, sector 25376
end_request: I/O error, dev sr0, sector 25416
end_request: I/O error, dev sr0, sector 25460
end_request: I/O error, dev sr0, sector 155408
I checked, and all the software I ought to need was installed, and all the rpm dependencies were present and satisfied. So, I thought it might be a hardware problem, maybe with DMA, as did google searches. Cue lots of prodding of libata options, different kernels etc. Didn't help.

Eventually though, I stumbled upon the answer. On Fedora, installing libdvdread isn't enough. If libdvdcss isn't installed, then it'll silently carry on without it, and access to css protected bits of the disk will fail with drive errors. No warnings, no messages, no dependency checks at install. Just cryptic failures that make you think there's a low level problem.

Doing a "yum install libdvdcss" fixes the problem nicely. Quite why it couldn't give me a nice friendly message to tell me to do that I don't know, or even just have it as a dependency. Bah :(
link1 comment|post comment

navigation
[ viewing | most recent entries ]
[ go | earlier ]