Strange Aeons

Well this was more effort than it should have been. I have been dabbling with Erlang for

a while, and after struggling with MySQL, Tomcat and JDBC I was looking for an alternative web-app stack. CouchDB looks to be perfect, although sufficiently new that there is not a lot of documentation, both of the books available right now are OK, but not brilliant. In general I prefer the style of CouchDB: The Definitive Guide but prefer the examples from Beginning CouchDB. My personal preference with programming and software tool books is that they should provide detailed, hand-held, speak-to-me-like-I-am-an-imbecile walk-throughs of the most common basic use cases. Anyhow, CouchApp looks like a great way to develop apps for CouchDB, although documentation is fairly sparse. The main problem I found is that the versions of Erlang and CouchDB available from repositories for both Ubuntu and Debian are way behind the cutting edge to the point that the examples in the books won't run. I found that the best way to set up my CouchApp development environment is to completely avoid the repositories and build from source in a clean Ubuntu server VM using the following steps:

• $ apt-get update
• $ apt-get clean
• $ apt-get upgrade

We are going to be building some software so the following tools are useful:

• $ sudo apt-get install build-essential subversion git-core openssh-server

Install Erlang

• $ sudo apt-get build-dep erlang
• $ sudo apt-get install sun-java6-jdk
• $ wget http://www.erlang.org/download/otp_src_R14A.tar.gz
• $ tar zxvf otp_src_R14A.tar.gz
• $ cd otp_src_R14A
• $ ./configure
• $ make && sudo make install

Install CouchDB

• $ sudo apt-get build-dep couchdb
• $ sudo apt-get install xulrunner-dev libicu-dev libcurl4-gnutls-dev libtool
• $ wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/couchdb/1.0.0/apache-couchdb...
• $ tar zxvf apache-couchdb-1.0.0.tar.gz
• $ cd apache-couchdb-1.0.0
• $ ./configure
• $ ./configure --with-js-lib=/usr/lib/xulrunner-devel-1.9.2.3/lib --with-js-include=/usr/lib/xulrunner-devel-1.9.2.3/include
• $ make CouchDB
• $make
• $sudo make install

Final Setup & Running CouchDB

• $ adduser --system --home /usr/local/var/lib/couchdb --no-create-home --shell /bin/bash --group --gecos "CouchDB Administrator" couchdb
• $ sudo chown -R couchdb:couchdb /usr/local/etc/couchdb
• $ sudo chown -R couchdb:couchdb /usr/local/var/lib/couchdb
• $sudo chown -R couchdb:couchdb /usr/local/var/log/couchdb
• $sudo chown -R couchdb:couchdb /usr/local/var/run/couchdb
• $sudo chmod -R 0770 /usr/local/etc/couchdb
• $sudo chmod -R 0770 /usr/local/var/lib/couchdb
• $sudo chmod -R 0770 /usr/local/var/log/couchdb
• $sudo chmod -R 0770 /usr/local/var/run/couchdb
• $ sudo ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb

Put xulrunner-devel on your LD_LIBRARY_PATH

• $ sudo touch /etc/ld.so.conf.d/couchdb.conf
• $ sudo cat > /etc/ld.so.conf.d/couchdb.conf
• /usr/lib/xulrunner-devel-1.9.2.6 (NB. Check the version that you have installed!)
• <CTRL-D>
• $ sudo ldconfig

Running CouchDB Manually

• $ sudo -i -u couchdb couchdb

Running CouchDB As a Daemon

• $ sudo /etc/init.d/couchdb start

I finally got around to getting Tor working on one of my workstations. The process is quite simple to set up to enable anonymous browsing. My workstation is running Ubuntu so I firstly had to install the Debian/Ubuntu APT repository (NB. More detailed instructions for steps 1-3 are available from the Tor Project Debian/Ubuntu Installation Instructions and more details on steps 4 - 7 are available from the Tor Project general *nix installation instructions).

1. To your /etc/apt/sources.list add (replacing <DISTRIBUTION> with the name of your distro which you can copy from your existing sources.list):

   deb http://deb.torproject.org/torproject.org <DISTRIBUTION> main

2. Now you need to add the crypto keys

   gpg --keyserver keys.gnupg.net --recv 886DDD89 gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add -

3. Now update your apt package list then install both Tor and the Tor Geographical IP database

   $ sudo apt-get update $ sudo apt-get install tor tor-geoipdb

4. Install Polipo:

   $ sudo apt-get install polipo

5. Configure Polipo by downloading the ready-made Polipo configuration file from the Tor Project, moving it to /etc/polipo and renaming it to config
6. Configure Tor with your webbrowser. I am using Firefox and configuration was as easy as installing the TorButton plugin and restarting firefox.
7. Verify that Tor is working correctly by visiting the Tor Detector.

Remember that this is not a panacea for privacy issues on the web and all of the technical protection offered by Tor can be squandered in a heartbeat through user action. Good personal online security and privacy practice is essential.

The rsync program relies upon accurate clocks on the various machines that it is synchronising between because it uses the modification timestamps to determine which is newer and therefore which was more recently altered. So as the first stage in getting a distributed and bi-directional rsync system up and running so as to synchronise my files between several workstations and a fileserver, I have had to install and setup some tools that correct the clocks on each of these machines. The default is to use ntpdate which is installed on many *nixes by default but is only set to update the system clock at startup. Therefore a cron job is required to be run to cause ntpdate to be automatically run more often. Create an executable (chmod 755) /etc/cron.daily/ntpdate file containing:

ntpdate ntp.ubuntu.com pool.ntp.org

Because a machine's hardware clock can drift over time it helps also to install the ntp daemon which calculates the drift in your system clock and adjust is. To set it up, all that is required is:

sudo apt-get install ntp

followed by adding the ntp server pool to /etc/ntp.conf:

server ntp.ubuntu.com server pool.ntp.org

This process needs to be followed on all machines that are going to be part of the synchronisation pool because they all need to have accurate clocks.

Whilst I use LVM without RAID to manage the drives on my media server, because the media rarely changes so the offline backup is easy to keep in sync, I have started using RAID 1 to mirror the contents of my file server to guard against the failure of a single drive. This is because the data changes sufficiently often on the file server that my regular backup might get out of sync by up to a day which could lose me a days work.

1. Using the Debian alternate install CD, at the partion disks screen, select manual
2. Select the free space on your first drive and create 3 new partitions as follows: 2GB for /boot, 2GB for SWAP, Remainder for /
3. Select "physical volume for RAID" at this point and ensure that the boot partition is marked as bootable.
4. Repeat the above steps for the second drive
5. Now select "Configure software RAID" from the main partition disks screen
6. Create 3 Multidisk (MD) devices with each configured as RAID 1 with 2 active and 0 hotspace disks, one MD device for each of the boot swap and root partitions. Select the correct corresponding matched partitions to include in each MD device, e.g. sda1 and sdb1, sda2 and sdb2, sda3 and sdb3
7. After returning once more to the main Partition Disks screen you should see the RAID devices that you just created listed alongside the partitions from before. Create file systems on each RAID device in the same fashion as for normal, non-RAID, partitions, e.g. RAID device #0: type ext3 mounted at /boot, RAID device #1: type SWAP, RAID device #2: type ext3 mounted at /
8. Write the changes to disk and continue with a normal installation

After installation you must also install grub to the master boot record on the second drive in the RAID array so that the system can boot from the other drive in the case of one drive failing.

$ sudo grub-install /dev/sda $ sudo grub grub> device (hd0) /dev/sdb grub> root (hd0,0) grub> setup (hd0) grub> quit

Verify that you have the correct filesystems in the correct places, e.g.

$ grep /dev/md /etc/fstab $ df -h /

The status of RAID devices can be checked using the /proc/mdstat file. Each mdN device contains two sdXN disks and each mdN device should have "2/2" and "UU":

$ cat /proc/mdstat

The mdadm utility gives more details:

$ sudo mdadm --query --detail /dev/md0

An aide memoire to myself for setting up Ubuntu with LVM at install time:

1. At the disk partitioning screen select Manual
2. Select a drive (if they are new drives then they will require partition tables)
3. Select free space on the drive and choose to create a new partition. First off we need a /boot partition of 100MB which is of the primary type and starts at the beginning of the disk. Select the /boot mount point for this partition and ensure that you have selected the bootable flag.
4. Repeat the last step to create a SWAP partition of about twice the size of your RAM.
5. The rest of the disks holding the /boot and SWAP partitions can now be used to hold an LVM containing the / filesystem for the machine.
6. Select the free space on your first disk.
7. Choose to make it a physical volume for LVM
8. Repeat this step for any other drives that will be part of the LVM
9. Select "Configure the Logical Volume Manager"
10. Select "Create volume group"
11. Give the group a name - this can be anything but should be easy to remember. I just used machinename_vg1
12. Select which physical volumes will be part of the volume group
13. Select "Create a logical volume"
14. Select the volume group that will contain the new logical volume
15. Name the logical volume, e.g. machinename_lv1
16. Enter the size for the logical volume
17. Select finish
18. In the main menu select the volume group
19. Select to use the volume as Ext3
20. Select to mount the volume as the / file system
21. Finish the regular Ubuntu install

Of course one gotcha with LVM appears to be that unless you are combining the volumes with RAID then you could lose your entire logical volume, spanned acroos all of the disks, if you lose a single physical volume to a hardware fault. I would like to be proven wrong on this, and don't want to find out the hard way so I don't currently have data solely in an LVM file system - I merely use it as a convenience for a live file system that is completely backup up elsewhere.

I have been working fairly hard on some new research directions involving automated methods for effective strategic play of dialectical games. Because this is quite heavy going I frequently take breaks to do something less mentally taxing, like beautifying my Linux desktop. Updating conky to look good, changing my desktop wallpaper, and installing Avant Window Navigator have all contributed to what I think is a quite nice looking desktop:

I have started using the Avant Window Navigator (AWN) which gives the Ubuntu desktop a great alternative to the standard window panel running along the base of the screen. AWN basically gives Linux GUIs an OSX like dock which is aesthetically more pleasing than the default in Gnome but not really any more functional. It is a good idea to make this a start up item so head to System -> Preferences -> Sessions then select "Add" from the startup programs tab. Enter a suitable title for your new startup program and enter avant-window-navigator into the command field.

After my rail trip earlier this week where I had free, but so slow as to be unusable, WiFi I resolved to load some media onto my N810 to help pass the time on future journeys. Early experiments showed that the N810 can play the average xvid avi downloaded from your favourite torrent site. However it didn't necessarily play them well. Firstly, you need to install mplayer which gets better performance out of the hardware than the stock media player. Then, so long as the scene isn't high-motion you can watch your tv show, or movie, or whatever. To get good performance though really needs a re-encode, especially to play media downloaded from iplayer (maybe via my earlier technique for timeshifting British TV) which is unwatchable. The best solution that I have found is tablet-encode which provides mencoder profiles for the N810 and is command-line based so is ripe for scripting. Usage is simple, as all good software should be:

tablet-encode tablet-encode --help tablet-encode --preset list tablet-encode input.avi output.avi tablet-encode --list file1.avi file2.avi file3.avi tablet-encode --preset best input.avi output.avi tablet-encode dvd: film.avi tablet-encode --preset best dvd://1 dvd://2 dvd://3 media/ tablet-encode --gui input.avi output.avi

Tablet-encode can be downloaded from Maemo.org.

A colleague asked today how to search a set of directories for files containing a particular string and this is what I came up with:

find . | xargs grep gnu

where gnu was my search term. The find command with the dot alias as input recursively descends the file hierarchy from the present working directory. The output of find is piped into xargs which is a very useful little utility that takes the input from find as a long list and splits it into sub-lists, calling grep for each sublist in turn, where in this case each sublist is a filename that grep searches. Obviously he was looking for a different search term than gnu and he was only wanting to search particular file types but this was the basics of his solution.

I tried to install pwsafe on my Nokia N810 running the Maemo Linux distribution. Unfortunately I got a library not found error which the package manager couldn't fix, even after an update to the Maemo distribution. The missing library was called libhildonfm2 and after a bit of googling I found a fix. Unfortunately it requires that software be installed outside of the package manager using apt-get at the command line. Because installing using apt-get requires root privileges the first thing to do is install the rootsh app via the package manager. This enables you to get a root shell via the following command:

sudo gainroot

Once you have a root shell you can issue an:

apt-get autoremove

to clean up unusable and unused packages, then:

apt-get install pwsafe

which causes pwsafe and all required dependencies, including the required version of libhildonfm2, to be installed.