Saturday, November 09, 2013

NetBSD on the Raspberry Pi

Long time since my last post.
What's more, it might be sometime before I can seriously focus on this blog again as I'll be away from home for three weeks starting on Tuesday next (today is Saturday).

Well, in the last while I've been playing around with NetBSD on my Pi.
I have never used NetBSD before although I do have some experience with FreeBSD.

However, I have now got it working very well on my 512 MB Pi and I'll outline here what I've done together with any problems I've noticed.
First, I'll say that NetBSD, like FreeBSD and Gentoo, only has a limited availability of precompiled binary packages.
That means that for most things you want to install, you're going to have to compile them.
Compiling in NetBSD is easy and well-documented.
However, it can be stupifyingly slow on the Pi because of the slow CPU speed and limited RAM.

I got my NetBSD rpi image from this link which I found in the RPi forum here.
Regularly updated rpi images are available here and the link looks like this.

I gunzipped the .img.gz file and burned it to an 8GB SD card using

# dd bs=1M if=2013-10-20-netbsd-raspi (2).img.gz.img of=/dev/sdd
The SD card booted fine in the Pi and brought me to login page (no GUI).
Log in as root and no password is needed.

Type startx at the (root) prompt to get to the Desktop.

Actually, there is no directory called Desktop as the image uses the lightweight IceWM as a window manager and this is very similar to Fluxbox.

At this stage there are three important "housekeeping" tasks to complete.

1. The keyboard layout in the image in Japanese. So for many people, including me, to enable fluent typing it's necessary to open /root/.xinitrc and find the line
setxkbmap -layout jp
to
setxkbmap -layout us

or whatever you want.

In the same file, find the line
export LANG=ja_JP.UTF-8
and change it to
 export LANG=en_US.UTF-8
or whatever is appropriate for you.

At this stage, reboot and you should be able to type a lot more easily.



2. Now set up the time for your time zone.
This requires merely setting up the following symlink:

# ln -s /usr/share/zoneinfo/Europe/Dublin /etc/localtime
and set the Continent/City as appropriate for your location.


3. Expand the file system to fill the SD card.

The NetBSD image I used provides a root partition of only 1.8GB size.
Indeed, it's almost full before installing anything.
Luckily, it's easy to expand the root partition to take up all remaining space on the SD card by following the very clear instructions given here (see section marked 'Growing the Root file-system')


At this stage, NetBSD should start looking very usable on the Pi.
But, there's a lot more to be done.

1. First, set up your own user account with the following commands:

# useradd -m -G wheel paul
# passwd paul

(where paul is my user)

2. Before booting to your own user account, make sure that you can use sudo in your account.
You need to do this from the root account.

For this, issue the visudo command and edit the /usr/pkg/etc/sudoers file
If you've put yourself in the wheel group as I did with the useradd command above, then you just need to uncomment the following line in the sudoers file.

%wheel ALL=(ALL) NOPASSWD: ALL
Remember, however, that this allows use of sudo without any password which might be dangerous if others have access to your account.

3. Next, enable the IceWM window manager in your account and get it to start up at boot.

# cp -r /root/.icewm /home/paul/ 
# cp -r /root/.xinitrc /home/paul
4. Edit ~/.profile to include the line
startx
at the bottom.
This will autostart the X server after login.

5. But wouldn't it be better just to autologin as well?

It took me a while to track down how to do this but I finally found it in the mailing-list post.
Basically, I did no more than add

#Autologin Paul 
paul_login:\          
    :al=paul:np:sp#9600:ig:ht:

at the bottom of the /etc/gettytab file.
And in /etc/ttys, comment out the line
ttyE0 "/usr/libexec/getty Pc" wsvt25 off secure 
and add the following line

         ttyE0 "/usr/libexec/getty paul_login" vt100 on secure

6. Finally, at this initial stage we might as well overclock the CPU just a little as NetBSD on the Pi is slow.
Note that no config.txt file exists in /boot.
So, I just created one and added the following two lines
arm_freq=860 
force_turbo=1

Now reboot and see how it looks.
You should get straight to the IceWM "Desktop" without logging in.

Here’s how mine looks.
Although what’s on the right-hand side looks like Conky, I couldn’t get Conky to compile which seems to be a known problem in NetBSD (Look at Conky Doesn’t Compile in NetBSD in the FAQs for Conky).
It’s actually Torsmo which, although unmaintained for years, still works reasonably well although it’s not as polished as Conky now.

There's still a lot more can be done and I'll add these points when I get some time.






Thursday, September 12, 2013

Modify groups of photos all at the same time in Gimp

I've used Gimp in both Linux and OS X for many years.
However, I'm not even remotely an expert in photo re-touching.

But I do a lot of simple things such as resizing photos to include them in this blog or in emails, for example.
Amazingly, I was used to resizing each individually which can be time-consuming if you have more than just a handful of shots.

Then I discovered BIMP.
No, that's not a typo but stands for Batch Image Manipulation Plugin
This link explains how to use BIMP.
To install it, just go to this GitHub link where you can download the source for compilation in Linux (very quick compile and install) or the binary for Windows (which I haven't yet used).

Interestingly, for Mac 10.8, Gimp 2.8.6 already has Bimp integrated into its structure.
To use it, simply click on File and then Batch Image Manipulation Plugin.
Thereupon, the window to the right will open.

This really is a very necessary addition to Gimp and it dramatically improves its usefulness for simple users like me.



Monday, September 09, 2013

Gmail alert on Pi with PiGlow

I use quite a few notifiers to keep me aware of incoming emails on my Gmail system.
On my Mac, I use Google Notifier in the upper panel as well as the Google Mail Checker and Mail Checker Plus for Google Mail as extensions in Google Chrome.

I also have a mail checker in Conky which I use in all my computers running Linux. And this includes my Raspberry Pi's.
(Yes, I know the apostrophe is both unnecessary and grammatically incorrect. However, without it the last term in the plural form of the little computer seems to refer to a bodily effluent).

Great, so I should never miss an incoming email?
Well, not necessarily.
A lot of times I'm working or reading at my desk without a computer which may therefore all be in sleep mode.
As a result, I don't see the Gmail alerts.

Well, the PiGlow add-on board provides a great and very spectacular solution for my RPi.

How to set up the PiGlow is fully explained here.

Next I wrote two bash scripts to provide the LED sequences I wanted for my email alerts.

Here they are:

Text File: new_mail

#!/bin/bash 
for i in {1..10} 
do 
sudo piglow leg 0 25 
sudo piglow off 
sudo piglow leg 1 60 
sudo piglow off 
sudo piglow leg 2 100 
sudo piglow off 
done 
for j in {1..5} 
do 
sudo piglow all 100 
sleep 0.25 
sudo piglow off 
sleep 0.25 
done

 Text File: new_mail


          #!/bin/bash

          for i in {1..10}

         do
sudo piglow ring 1 25 
sudo piglow off 
sudo piglow ring 2 40 
sudo piglow off 
sudo piglow ring 3 55 
sudo piglow off 
sudo piglow ring 4 75 
sudo piglow off 
sudo piglow ring 5 100 
sudo piglow off 
sudo piglow ring 4 75 
sudo piglow off 
sudo piglow ring 3 55 
sudo piglow off 
sudo piglow ring 2 40 
sudo piglow off 
sudo piglow ring 1 25 
sudo piglow off 
done 
for j in {1..5} 
do 
sudo piglow leg 0 35 
sudo piglow off 
sudo piglow leg 1 70 
sudo piglow off 
sudo piglow leg 2 100 
sudo piglow off 
done

To use these scripts, they must be made executable by running the command
$ chmod +x new_mail

They can now be tested by moving to the directory where they are stored and running with the command (I stored them in my ~/Scripts which explains why I'm not using sudo)
$ ./new_mail
If all went well, these scripts may now be used to alert you to incoming emails.

I have referred before to the gmail_parser.py script by Baishampayan Ghose that works perfectly in Conky to update my unread email number.

I also mentioned in the previous post that I had updated this parser script to show the output in a different colour depending on the number of unread mails.

To get conky to alert me through my PiGlow of incoming mail just required me to further modify the gmail_parser.py script.
Below is the "heart" of the parser script that contains the changes:

def readmail(feed, maxlen): 

'''Parse the Atom feed and print a summary''' 

atom = feedparser.parse(feed) 

print '${color2} % s new email(s)\n' % (len(atom.entries)) 

elif len(atom.entries) == 1: 

print '${color3} %s new email(s)\n' % (len(atom.entries)) 

import subprocess 

subprocess.call("/home/paul/Scripts/new_mail", shell=True) 

else: 

print '${color4} %s new email(s)\n' % (len(atom.entries)) 

import subprocess 

subprocess.call("/home/paul/Scripts/new_mail2", shell=True)

The changes made to gmail_parser.py to enact the gmail alerts on the PiGlow are marked in blue.

And here's what the alert looks like (in a disappointingly unsharp video, but you'll get the idea):

 




Sunday, August 25, 2013

Stop Google Chrome inadvertently moving to another page

I mentioned that I fully restored OS X to my MacBook by wiping the disk and reinstalling.
Of course, Google Chrome had to be reinstalled as it is my browser of choice on all platforms these days (except for the RPi where I use the minimal web browser or Uzbl).

Well, after the reinstall, I found that browsing in Chrome was troublesome as, while using the trackpad to move up and down the page, occasionally I would be moved to a previous or subsequent page completely unintentionally.
I found this very distracting.

This problem arises because moving up and down requires two fingers moving in a vertical orientation on the trackpad.
However, the default configuration in System Preferences > Trackpad > More Gestures is for two fingers moving horizontally to swipe between pages.
As a result, if while moving two fingers vertically, you inadvertently add some horizontal orientation to your movement, you may well find that you've swiped to another page.

Solution?
Go into preferences and select Swipe with Three Fingers to swipe between pages.
I've been using this configuration for a few weeks without even a single inadvertent swipe between pages.

Friday, August 23, 2013

Display RPi's physical parameters in Conky

I've been using Conky in all of the OSes that I use on my Pi from the very start.
Whereas on my Desktops, I always like to display Core and GPU temperatures in Conky, early versions of OSes for the Pi were incapable of displaying any physical parameters.

The one exception, perhaps, was the use of BogoMIPS from /proc/cpuinfo as an approximation for the CPU speed.

Now, however, there's the vcgencmd which provides quite a lot of physical information of the Pi.

The screenshot to the right shows
a very small part of my Conky display in Raspbian which shows the CPU Core temperature, the Core voltage and a more accurate representation of the CPU frequency.


Here's the coding I added to my .conkyrc file to add these parameters to the output:


Core Temp: ${alignr}${exec vcgencmd measure_temp | cut -c6-9}°C 
Core Voltage: ${alignr}${exec vcgencmd measure_volts | cut -c6-10} 
CPU Freq:${alignr}${exec printf "%0.0f" $((`vcgencmd measure_clock arm | cut -c15-24 | awk '{print $1}'`/1000000))} MHz
In reality, the only one of these that actually varies to any significant extent is the core temperature.
The CPU frequency might decrease by 1 MHz now and again but I don't believe I've ever seen the Core voltage change at all.
However, the CPU temperature changes a lot.
At first boot, from cold, this temperature starts at about 32ºC and rises up to as high as 56ºC
This parameter is, therefore, a candidate for color-coding in the Conky output to indicate when the temperature rises above comfortable levels.

Wednesday, August 21, 2013

RPi to 50" plasma display

Yesterday, I explained how I copied my Raspbian from one 4GB SDCard to another of size 8GB.
Now I wanted this for a new RPi that I intended to hook up to my new Panasonic 50" Plasma TV.

To get the display to show up on the TV (of course, it's a HDMI to HDMI connection), some changes were necessary to the file /boot/config.txt
There's a wonderful how-to that explains how this is done on the RPi forum.

In my case, I went through the following steps:

1. First, you need to get a display.
When I connected the RPi to one of the HDMI inputs (HDMI3) on the TV, I got no display.
So, open up /boot/config.txt. I did this by removing the SDCard, and transferring it to the card reader on my trusty (but quite old, now) EeePC901 and opened the /boot partition in Ubuntu.
There I was able to edit the config.txt file and uncommented the lines:

 #hdmi_safe=1
#hdmi_force_hotplug=1

Replacing the card in the RPi and booting it now gave me a display, albeit in 640x480 resolution.

2. Next, you need to set up a more usable screen resolution.
Make sure you're in VGA mode (by setting hdmi_group=1 and hdmi_mode=1 if you've changed them from the config.txt settings above) and run the following commands.

But, because the VGA mode is very awkward, I found it much more convenient to ssh into Raspbian from another RPi but it could be from anywhere (except, of course, Windows) and run the commands from there.

tvservice –m CEA
which gave me

Group CEA has 1 modes:  (prefer) mode 31: 1920x1080 @ 50Hz 16:9, clock:148MHz progressive 3D:TopBot|SbS-HH
The next command
tvservice -m DMT
gave me the following output
Group DMT has 0 modes:

Then, as suggested in the linked HDMI setup post, I ran this command
tvservice –d somefile
followed by
 edidparser somefile
which gave me the following very large output:
Enabling fuzzy format match...Parsing plasma...HDMI:EDID version 1.3, 1 extensions, screen size 128x72 cmHDMI:EDID features - videodef 0x80 !standby !suspend !active off; colour encoding:RGB444|YCbCr422; sRGB is not default colourspace; preferred format is native; does not support GTFHDMI:EDID found monitor name descriptor tag 0xfcHDMI:EDID monitor name is Panasonic-TVHDMI:EDID found monitor range descriptor tag 0xfdHDMI:EDID monitor range offsets: V min=0, V max=0, H min=0, H max=0HDMI:EDID monitor range: vertical is 23-61 Hz, horizontal is 15-68 kHz, max pixel clock is 150 MHzHDMI:EDID monitor range does not support GTFHDMI:EDID found preferred CEA detail timing format: 1920x1080p @ 50 Hz (31)HDMI:EDID found CEA detail timing format: 1920x1080p @ 60 Hz (16)HDMI:EDID established timing I/II bytes are 21 08 00HDMI:EDID found DMT format: code 4, 640x480p @ 60 Hz in established timing I/IIHDMI:EDID found DMT format: code 9, 800x600p @ 60 Hz in established timing I/IIHDMI:EDID found DMT format: code 16, 1024x768p @ 60 Hz in established timing I/IIHDMI:EDID standard timings block x 8: 0x3140 4540 6140 8180 0101 0101 0101 0101 HDMI:EDID found DMT format: code 4, 640x480p @ 60 Hz (4:3) in standard timing 0HDMI:EDID found DMT format: code 9, 800x600p @ 60 Hz (4:3) in standard timing 1HDMI:EDID found DMT format: code 16, 1024x768p @ 60 Hz (4:3) in standard timing 2HDMI:EDID found DMT format: code 35, 1280x1024p @ 60 Hz (5:4) in standard timing 3HDMI:EDID parsing v3 CEA extension 0HDMI:EDID monitor support - underscan IT formats:yes, basic audio:yes, yuv444:yes, yuv422:yes, #native DTD:2HDMI:EDID found CEA detail timing format: 1920x1080i @ 50 Hz (20)HDMI:EDID found CEA detail timing format: 1920x1080i @ 60 Hz (5)HDMI:EDID found DMT detail timing format: 1366x768p @ 60 Hz (81)HDMI:EDID found CEA format: code 31, 1920x1080p @ 50Hz (native)HDMI:EDID found CEA format: code 16, 1920x1080p @ 60Hz (native)HDMI:EDID found CEA format: code 20, 1920x1080i @ 50Hz HDMI:EDID found CEA format: code 5, 1920x1080i @ 60Hz HDMI:EDID found CEA format: code 32, 1920x1080p @ 24Hz HDMI:EDID found CEA format: code 33, 1920x1080p @ 25Hz HDMI:EDID found CEA format: code 34, 1920x1080p @ 30Hz HDMI:EDID found CEA format: code 19, 1280x720p @ 50Hz HDMI:EDID found CEA format: code 4, 1280x720p @ 60Hz HDMI:EDID found CEA format: code 18, 720x576p @ 50Hz HDMI:EDID found CEA format: code 3, 720x480p @ 60Hz HDMI:EDID found CEA format: code 22, 1440x576i @ 50Hz HDMI:EDID found CEA format: code 7, 1440x480i @ 60Hz HDMI:EDID found audio format 2 channels PCM, sample rate: 32|44|48 kHz, sample size: 16 bitsHDMI:EDID found HDMI VSDB length 26HDMI:EDID HDMI VSDB has physical address 3.0.0.0HDMI:EDID HDMI VSDB supports AI:yes, dual link DVI:noHDMI:EDID HDMI VSDB deep colour support - 48-bit:no 36-bit:yes 30-bit:yes DC_yuv444:yesHDMI:EDID HDMI VSDB max TMDS clock 190 MHzHDMI:EDID HDMI VSDB content type support: game|cinema|photo|textHDMI:EDID HDMI VSDB supports 3D formatsHDMI:EDID HDMI VSDB 3D_Mask: 0x019cHDMI:EDID HDMI VSDB 3D_Structure_All: 0x8141HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 0 (CEA 31), 3D struct 0x06, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 1 (CEA 16), 3D struct 0x06, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 0 (CEA 31), 3D struct 0x08, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 1 (CEA 16), 3D struct 0x08, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 9 (CEA 18), 3D struct 0x06, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 10 (CEA 3), 3D struct 0x06, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 9 (CEA 18), 3D struct 0x08, 3D detail 0x00HDMI:EDID HDMI VSDB explicit 3D mode: 2D VIC order 10 (CEA 3), 3D struct 0x08, 3D detail 0x00HDMI:EDID found Video Capability DB length 2HDMI:EDID video capability: CE:3 IT:2 PT:0 QS:1HDMI:EDID extended data block tag 0x05 length 3 not supportedHDMI:EDID adding mandatory support for CEA (1) 640x480p @ 60HzHDMI:EDID adding mandatory support for CEA (2) 720x480p @ 60HzHDMI:EDID adding mandatory support for CEA (17) 720x576p @ 50HzHDMI:EDID VSDB 3D legend:FP=frame packing, F-Alt=Field Alternative, L-Alt=Line AlternativeSbS-Full=Side by Side Full, Ldep=L Depth, Ldep+Gfx=L Depth + Graphics DepthTopBot=Top and Bottom, SbS-HH=Side by Side half horizontalSbS-OLOR=Side by Side odd left odd right, SbS-OLER=Side by Side odd left even rightSbS-ELOR=Side by Side even left odd right, SbS-ELER=Side by Side even left even rightHDMI:EDID CEA (32) 1920x1080p 24Hz 3D supports: FP|TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (4) 1280x720p 60Hz 3D supports: FP|TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (5) 1920x1080i 60Hz 3D supports: FP|TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (19) 1280x720p 50Hz 3D supports: FP|TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (20) 1920x1080i 50Hz 3D supports: FP|TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (31) 1920x1080p 50Hz 3D supports: TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (16) 1920x1080p 60Hz 3D supports: TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (33) 1920x1080p 25Hz 3D supports: noneHDMI:EDID CEA (34) 1920x1080p 30Hz 3D supports: noneHDMI:EDID CEA (18) 720x576p 50Hz 3D supports: TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (3) 720x480p 60Hz 3D supports: TopBot|SbS-HH|SbS-OLOR|SbS-OLER|SbS-ELOR|SbS-ELERHDMI:EDID CEA (22) 1440x576i 50Hz 3D supports: noneHDMI:EDID CEA (7) 1440x480i 60Hz 3D supports: noneHDMI:EDID filtering formats with pixel clock > 162 MHz or h. blanking > 1023HDMI:EDID best score mode initialised to CEA (1) 640x480p @ 60 Hz with pixel clock 25 MHz (score 0)HDMI:EDID best score mode is now CEA (1) 640x480p @ 60 Hz with pixel clock 25 MHz (score 61864)HDMI:EDID best score mode is now CEA (2) 720x480p @ 60 Hz with pixel clock 27 MHz (score 66472)HDMI:EDID CEA mode (3) 720x480p @ 60 Hz with pixel clock 27 MHz has a score of 66472HDMI:EDID best score mode is now CEA (4) 1280x720p @ 60 Hz with pixel clock 74 MHz (score 135592)HDMI:EDID DMT mode (4) 640x480p @ 60 Hz with pixel clock 25 MHz has a score of 43432HDMI:EDID best score mode is now CEA (5) 1920x1080i @ 60 Hz with pixel clock 74 MHz (score 3773832)HDMI:EDID CEA mode (7) 1440x480i @ 60 Hz with pixel clock 27 MHz has a score of 45736HDMI:EDID DMT mode (9) 800x600p @ 60 Hz with pixel clock 40 MHz has a score of 53800HDMI:EDID best score mode is now CEA (16) 1920x1080p @ 60 Hz with pixel clock 148 MHz (score 4898248)HDMI:EDID DMT mode (16) 1024x768p @ 60 Hz with pixel clock 65 MHz has a score of 72185HDMI:EDID CEA mode (17) 720x576p @ 50 Hz with pixel clock 27 MHz has a score of 66472HDMI:EDID CEA mode (18) 720x576p @ 50 Hz with pixel clock 27 MHz has a score of 66472HDMI:EDID CEA mode (19) 1280x720p @ 50 Hz with pixel clock 74 MHz has a score of 117160HDMI:EDID CEA mode (20) 1920x1080i @ 50 Hz with pixel clock 74 MHz has a score of 4232360HDMI:EDID CEA mode (22) 1440x576i @ 50 Hz with pixel clock 27 MHz has a score of 45736HDMI:EDID best score mode is now CEA (31) 1920x1080p @ 50 Hz with pixel clock 148 MHz (score 5336040)HDMI:EDID CEA mode (32) 1920x1080p @ 24 Hz with pixel clock 74 MHz has a score of 124532HDMI:EDID CEA mode (33) 1920x1080p @ 25 Hz with pixel clock 74 MHz has a score of 128680HDMI:EDID CEA mode (34) 1920x1080p @ 30 Hz with pixel clock 74 MHz has a score of 149416HDMI:EDID DMT mode (35) 1280x1024p @ 60 Hz with pixel clock 108 MHz has a score of 103643HDMI:EDID DMT mode (81) 1366x768p @ 60 Hz with pixel clock 85 MHz has a score of 3062945HDMI 
:EDID preferred mode remained as CEA (31) 1920x1080p @ 50 Hz with pixel clock 148 MHzHDMI:EDID has HDMI support and audio support 
edid_parser exited with code 0
As is indicated in this last output and the output from the very first command, Mode 31 (CEA) 1920x1080p @ 50Hz is the favoured mode for my set up.

So with that knowledge, I added the following lines to /boot/config.txt
sdtv_mode=2 
sdtv_aspect=3 
hdmi_force_hotplug=1 
config_hdmi_boost=4 
hdmi_group=1 
hdmi_mode=31
On reboot, the display showed as a very nice 1080p @ 50Hz with the sole exception that the display didn't quite stretch to the screen edges.
However, this was easily resolved by adding the following lines to /boot/config.txt
overscan_left=0 
overscan_right=0 
overscan_top=-24 
overscan_bottom=-24
Now, the display was perfect.......and very, very big.

Performance?
Well, so far I'm very happy with it.
I'm using a LogiTech K400r combined keyboard and touchpad which works perfectly and is ideal for for computer control from a reclining position on the living room sofa, or the floor for that matter.

Additionally, I have been able to successfully overclock, without any overvolting, considerably higher than on my other 512 MB RPi.
On this latter, I couldn't go higher than arm_freq=875 without a boot failure.
On the new Pi, however, I can run stably with arm_freq=940, core_freq=450 and sdram_freq=400

Sound works great with all sound going via HDMI to the TV's speakers.
Shell-FM sounds wonderful from the Pi.

I've mentioned before that I type in both English (layout=US) and Brazilian Portuguese (layout=BR) and always have both layouts available in any OS that I use.
In Raspbian on my older 512 MB RPi, where I use Fluxbox as the window manager, following this guide I modified the file /etc/default/keyboard to include the lines:
XKBMODEL="pc105"
XKBLAYOUT="us,br"
XKBVARIANT=","
XKBOPTIONS="grp:alt_shift_toggle"
and this, in conjunction with the keyboard switcher indicator works perfectly.

However, although what I use in my new RPi is an exact clone of Raspbian from the other RPi, and everything else works flawlessly, I can no longer use Alt-Shift to switch keyboard layouts.
And up to now, I have been unable to resolve or understand this problem.

However, I found a workaround here.
This involved making a simple bash script (~/Scripts/layout_toggle) with the following input:

#! /bin/bash 
CURR_LAYOUT=$(setxkbmap -query | grep layout | cut -d ":" -f 2 | sed -e 's/ //g') [[ $CURR_LAYOUT == "us" ]] && setxkbmap -layout br || setxkbmap -layout us
making it executable and then adding a line to ~/.fluxbox/keys as follows

Mod4 Mod1 k  :ExecCommand /home/paul/Scripts/layout_toggle
Now, by pressing Win+Alt+k I can toggle between my keyboard layouts when I want.
So, why didn't I use Alt+Shift to toggle the layouts?
Well, I did, but it just didn't work.
Again, I don't yet understand why










Sunday, August 18, 2013

Raspberry Pi: Copy contents of one SD Card to another of larger size.

I use many OSes on my RPi but my favourite has to be Raspbian with a Fluxbox window manager.
It's lightweight and lends itself very well to keyboard shortcuts which allow for optimum performance speed despite the significantly underpowered CPU.

Of course, I've overclocked as much as I can without overvolting.

Well, now I've got a new 50" plasma TV with internet, hdmi ports, usb ports and everything a modern TV should have.
So, I want to use it as a screen for another of my RPi's (I've got three now).

Rather than start from scratch, I wanted to just copy everything from my 4GB Transcend Class4 SD Card to a new 8GB Sandisk Extreme Pro card.

I fired up Ubuntu 13.04 on my Dell Desktop, plugged in my SD Card reader with the Transcend card therein (which came up as /dev/sdb) and ran the following command:

/home/paul# dd if=/dev/sdb conv=sync,noerror bs=64K of=/home/paul/raspbian.img
After a little less than 4 minutes, I got the following output to indicate the required image had been written to the home directory of my Dell Desktop

60504+0 records in60504+0 records out3965190144 bytes (4.0 GB) copied, 230.983 s, 17.2 MB/s
Next, I used the following command to write the image from the Desktop to my Sandisk Class10 card

/home/paul# dd bs=1M if=raspbian.img of=/dev/sdb
After a further 4 minutes, I got the following output
3781+1 records in3781+1 records out3965190144 bytes (4.0 GB) copied, 242.555 s, 16.3 MB/s
and my Sandisk card was ready for action.
Of course, I had to make some very small changes to my /boot/config.txt file to get the display to show up on the plasma screen.
I intend to write another post about this shortly.

Additionally, given that I had transferred a 4GB image to an 8GB card, I also needed to create a Swap partition and expand the /root partition to completely fill all of the available 8GB.
This is very easy to do using Gparted (I used it from Ubuntu on the Desktop).

Once I had this done and made the required config.txt changes, the Sandisk card worked perfectly on the TV.

The only potential problem here is the space required to store the image of the card to be copied.
So, for example, if you wished to image the contents of a 16GB card, even though only one or two GB are actually being used on the card, the image size will still be a whopping 16GB which may not be available on your Desktop.

This post explains how to image the card and zip it before it's written so that the space required to store the zipped image is considerably less.

Using the same Transcend 4GB card, I made the gzipped image using the following command:

/home/paul# dd if=/dev/sdb conv=sync,noerror bs=64K | gzip -c > raspbian.img.gz
which provided this output

60504+0 records in60504+0 records out3965190144 bytes (4.0 GB) copied, 319.907 s, 12.4 MB/s
The gzipped image had a size of 2.0GB which is exactly half of the unzipped image size.
So that's good.

Next, I unzipped and wrote this image to the same Sandisk 8GB Extreme Pro card with this command:

/home/paul# gunzip -c raspbian.img.gz | dd of=/dev/sdb conv=sync,noerror bs=64K
which produced the following output

39998+41012 records in81010+0 records out5309071360 bytes (5.3 GB) copied, 329.885 s, 16.1 MB/s
 That's strange, why were 5.3GB copied when the image was only 4GB in size?

In any event, I again used Gparted to expand the /root directory and create a Swap partition.
However, according to Gparted, the file systems in each of the two existing partitions were unrecognizable.
As a result, Gparted could do nothing with these partitions.

Additionally, the Sandisk card was unbootable.

I tried this technique two or three times and always got the same negative result.

Unfortunately, I have no explanation for this.
But, as I have ample room on my Desktop for any further SD Card images I may want to produce, I'm not particularly concerned.




 

Saturday, August 10, 2013

The very best OS X screensaver -- how did I forget that?

Before wiping my MacBook HDD, I copied everything I needed to an external hard drive.
Well, not quite everything.

For years I have used a superb clock as a screensaver which I rely upon to tell me the time when I'm reading beside my computer.

But, I didn't save it or even note its name.

But, it's so great it must be featured in lists of the "best OS X screensavers" or such like.
Amazingly, however, it's not.

I searched and searched but it just wouldn't appear.
Doesn't anybody know about this great and very well designed screensaver?
Why are they plugging all that crap in their lists of the "best ever ...."?

Anyway, I eventually tracked it down here.
And it's called the Illuminated Clock Screensaver.

Unfortunately, I get the impression from the site that it's no longer being developed.
However, it still works in Mountain Lion.

Restoring OS X to MacBook. How I overcame a forgotten password problem in the process.

I've been using a 13" MacBook (late 2009, 2.26 GHz C2D, 4GB RAM) for more than three years and it was starting to get uncomfortably slow.
Additionally, I was seeing just a little too much of that dreaded beachball.

While Onyx and iBoostUp helped marginally, I needed much more improvement in the performance of what has become my mainstay computer over the last few years.

Nothing for it but to wipe the hard disk and reinstall OS X.
Well, now that OS X features a recovery system, nothing could be simpler, right?

First, of course, I copied all of my important personal stuff, my iPhoto library, documents, music, my calibre library and others to a HFS+ formatted partition on an external hard drive.

Now, just reboot and hold down Ctrl-R during the boot up.
The first "recovery" option I chose was to delete the hard disk.
Well, I just wanted to make sure that absolutely none of whatever was slowing up my computer was remaining.
Next, re-install OS X (Mountain Lion).
So far, everything is going exactly as expected.

Then I'm asked to enter my Apple ID and password.
OK, that's easy.
However, I then get a message telling me that that was not the Apple ID I used to install Lion.
Really?
What's Lion got to do with it. I want Mountain Lion.
But I upgraded from Snow Leopard to Lion online and from there to Mountain Lion.
And this has not been forgotten.
Nevertheless, its relevance is still unclear to me.

However, the truth is that I have three Apple IDs that I've used over the years.
Two are regularly used and have known passwords.
The third, however, for reasons I just cannot explain, has an unknown password. Plus, none of the password recovery options work for this Apple ID. The password change options don't appear in any of my usual email addresses and the password recovery does not accept my DOB.

I honestly cannot explain what happened here.

In any event, of course it was the third Apple ID that was used to upgrade Snow Leopard to Lion.

So, now I'm stuck.
I've wiped my HDD and I can't reinstall Mountain Lion because I cannot use the Apple ID used to purchase Lion.
Oh boy.
Now what can I do?

Well, one option is just to buy Mountain Lion.
But wait, I have a Snow Leopard install DVD somewhere.
Let's install that just to have something to work with.
After the install, I then used the Applications Bundle DVD to install stuff like iPhoto and Garage Band.

Now do I have to install, and pay for, Lion?
Well, you can upgrade Snow Leopard 10.6.8 directly to Mountain Lion without having to go through Lion.
So, I just update Snow Leopard to the latest version which is very easy.

But wait, when I upgraded to Mountain Lion, I kept the Install Mountain Lion image and copied it to the external HDD.
So, copied it back to my Mac and started the upgrade to Mountain Lion.
This went without a hitch and I'm back where I started but with a HDD which is very much less loaded than it was previously.

And performance is very significantly quicker and smoother.
Nice job but must remember to write down all passwords in future.




Thursday, May 23, 2013

Time-Lapse photography using the RPi camera module.

I've already mentioned this excellent post on this topic which, for me at least, is truly groundbreaking.
Everything in this post works to perfection, but I'll just add some relatively minor personal observations here.

First, the red LED that turns on for every shot can be a nuisance if you're taking your time-lapse shots through a pane of glass as the LED reflects from the glass into the camera lens and shows up on your shots.
A potencial solution is a very minor addition to /boot/config.txt.
Unfortunately, this no longer works (look here, too).
However, as this is undoubtedly a firmware issue, I'm certain it'll be fixed in short order.
In the meantime, I'm using a couple of layers of insulating tape over the LED to avoid this problem.

The command used in the DesignSpark post to join up all of the time-lapse photos into a single .avi film is the following:

$mencoder -nosound -ovc lavc -lavcopts vcodec=mpeg4:aspect=16/9:vbitrate=8000000 -vf scale=1920:1080 -o tlcam.avi -mf type=jpeg:fps=24 mf://@stills.txt
This works fine but ONLY as long as all of your .jpg files contain data.
I have found that, occasionally, a dud file results with essentially 0 bytes of data.
The mencoder command will fail if it meets up with one of these files among the thousands, at least, that you may have.

What I like to do, before running the
# ls *.jpg > stills.txt
command is to cd to the directory with your .jpg's and run
$ ls -lR | sort -bnr | tail
This will give a list of the ten smallest (in terms of Bytes content) files.
If any of these contain 0 bytes (or really any ridiculously small number such as 3 B), delete them before producing your stills.txt file.
If you have more than ten dud files, you can append any number (as a negative) you want after "tail".
So, this command will list the smallest twenty files in your .jpg list.
$ ls -lR | sort -bnr | tail -20






Tuesday, May 21, 2013

Camera module not happy on overclocked Raspberry Pi

I got my camera module from Farnell last week.
Notification and delivery went very smoothly.
On Sunday (12/05/2013) I indicated my interest in the camera module on the Farnell website.
The next day, they sent me an email telling me the camera was now available for purchase.
And, just two days later, it was delivered right to my door.
If only, things were always like that.

The documentation for this module is excellent and provides in a very lucid way all you need to get started.

Connecting the camera ribbon cable is very easy although take note that about 3mm on metal contact wires still protrude above the connector even with a good contact.

However, operation didn't at first go smoothly for me.

First, running as my user the command

raspistill -o photo1.jpg
gave this error
failed to open vchiq instance
The command ran without error when I used sudo, however.
This error is referred to in this guide for installing Quake3 in Raspbian.
Interestingly, after I installed Quake3, sudo was no longer required to run the raspistill command.
Nevertheless, the first of the suggested solutions didn't work for me.
The second, which is the following
# chmod a+rw /dev/vchiq
did but apparently this is only a temporary fix until the next reboot.

The biggest problem, however, was that this command did NOT produce any file named photo1.jpg.
raspistill -o photo1.jpg
In general, a screen opened on my monitor showing the object to be photographed. However, it didn't close and acted like a CCTV system.
At other times, the screen froze and the only way out was to pull the plug.
Sometimes, the screen remained live but the Desktop underneath froze (the clock stopped, the CPU monitor stopped, and that's as much as I could see.

However, if I reduced the wait time by something like
raspistill -t 5 -o photo1.jpg
(just 5 milliseconds delay until taking the shot), now I got a photo.

The video command
raspivid -o video1.h264
was even more temperamental.
The only way I could get and video at all was to severely limit the length of the video. And we're talking of half-second videos which are unlikely to be in much demand.
The command I used was
raspivid -t 500 -o video1.h264
I had seen some forum posts about the high power consumption of the camera module (nearly 300 mA for the camera alone).
So, I tried all of the four PSU's that I have for my Pi.
None gave me any significant improvement.

Now, I run all of my OSes on the Pi overclocked.
Generally I use cpu_freq=875 with force_turbo=1
So, I commented out these lines in my /boot/config.txt and now the camera worked absolutely perfectly.

Now that it's working fine, I can do some experimentation.
There is a wonderful blog post on how to use the Pi camera to do time-lapse photography.
And, everything works perfectly.

While the time-lapse script in the post works perfectly, I used an abbreviated script for my own meagre attempts at time lapse.
This is my script which is shamelessly taken from the post:

#!/bin/bash

while [ true ]; do

sudo raspistill -t 10 -o /media/stills/$(date +"%d_%m_%Y_%H_%M_%S").jpg

sleep 16

done;
The /media/stills/ directory is where I mounted a 16GB usb key (plugged into one Pi's USB ports) to hold the huge number of photos that can result from time-lapse.
And the 'sleep 16' merely ensures that shots are taken about every 20 seconds.

For €25, this is a powerful addition to my RPi.











Tuesday, April 23, 2013

Upgraded to Fedora 18 on my Pi and discovered the Fluxbox WM

I've used Fedora 17 on my Pi for some time and it works well although there are some problems.
My experience is detailed in this post and this follow-up.

Well, Fedora 18 has been available for a while now and a RPi version also appeared thereafter, so it's time for an upgrade.
However, rather than a clean install, I first tried upgrading from a terminal.
This is well documented in the Fedora wiki where two methods are suggested.

The recommended method is to use FedUp which is more beginner-friendly.
Unfortunately, although the app is available for use on the Pi, it doesn't work primarily because the methodology invokes the Grub bootloader to complete the upgrade.
However, there's no Grub on the Pi and no workaround seems to be available.

Ok, how about the Yum approach to upgrading Fedora to 17 to 18 which is detailed in the last link.
Unfortunately, this approach also didn't work on my Pi due to many errors.

Nothing for it but a clean install using the compressed image found here.
This went very smoothly and I used Gparted to increase the /root partition to take up all remaining space on my 8GB Sandisk Extreme Pro SDCard.
Note that as the image includes a 512 MB folder called /swap0 which serves as a swap partition, no further partition is needed on the SD Card.

As I had not installed over my Fedora 17 install, I was able to copy all essential stuff (mainly my elaborate Conky setup) from my old Fedora to the upgraded version.

Fedora 18 uses the xfce4 DE as before but the default DM has changed from GDM to LightDM.
However, although it's well-documented, I could not get lightdm to autologin by editing /etc/lightdm/lightdm.conf
Even invoking /etc/pamd/autologin did not work for me on the Pi.

I hate having to enter a password every time I boot into an OS so the lack of autologin in xfce4 forced me to look at the Fluxbox WM which I had recently played with in FreeBSD on my Pi

In summary, Fluxbox is wonderfully light, yet feature-full and ideally suited, I think, to the low resource boxes like the Raspberry Pi.

I could write pages on what I've done but because I'm short of time (traveling again tomorrow), I'm just going to summarize the main points of my experience so far.

i) Absolutely no problem getting Fluxbox to aulolog me in.
The three steps were:

a) to the file ~/.xinitrc add the line

exec startfluxbox
to start the WM at boot.

b) add the line
[[ -z $DISPLAY && XDG_VTNR -eq 1 ]] && exec startx
to ~/.bash_profile to start the X server at boot.

c) follow these excellent ArchLinux instructions to autologin your user at boot.

Actually, although I'm using Fedora 18 (the version I'm using is called Pidora 18), all of the references I've made to get complete autologin to the X desktop came from ArchLinux documentation.

ii) Conky works perfectly on Fluxbox in Pidora 18 with none of the transparency problems I had in xfce4.
The .conkyrc file I use is exactly the same as I had previously used in Fedora 17 on the Pi.
Additionally, there's no loss of transparency of the Conky window when new mail is received.

iii) Setting your Desktop wallpaper is very easy.
Unfortunately, when I wrote this the fluxbox wiki was down which restricted what I could reference.
However, a very lightweight configuration guide is also available here.

I added the following line to the ~/.fluxbox/startup file to get my selected wallpaper to load at boot time.

fbsetbg -f /home/paul/Pictures/fluxbox/fluxbox_wallpapers_by_deviantvicky-d33ory3.jpg


Well, there's still a lot more to report but it'll have to wait as I've run out of time for the moment.
But, I'll complete this post in just over a week's time.












Wednesday, April 17, 2013

Raspberry Pi: No boot after rpi-update

This is my first post in a long time as I was away in Brazil for a few weeks.

Anyway, when I got back I settled down to do some updating on my Pi OSes (of which I have quite a few).
I tried Raspbian first.
The command "sudo apt-get update && sudo apt-get upgrade" went without a hitch as did "sudo rpi-update".
That is until I tried to reboot.
It just didn't boot.
The red power light came on but the three internet lights never even flickered.
The green ACT light, however, was flashing three times, then stopping, then flashing three times and so on.
The RPi troubleshooting guide explains this nicely here.
So, it seems that start.elf was corrupted.

As usual, I'm not the first to have noticed this and a thread in the RPi forums already addressed the problem.
This post from the same thread suggests how to get Raspbian booting again.

OK, so I put my Raspbian SD card into my little, but exceedingly useful EeePC901 netbook (remember those?) and downloaded four files from the rpi-update github page.
These were bootcode.bin, start.elf, fixup.dat and kernel.img.
Then I copied all to /boot on my Raspbian SD card without doing anything further.

Back to the Pi and now the SD card booted.
Great!
Well, not quite, as for reasons I don't understand, my Logitech K260 keyboard and mouse did not work.
Same thing after a reboot.
So, now what?

Actually, the solution was quite simple.
I ssh'ed into my Pi from my MacBook (I autostart Conky in all my Pi OSes and from Conky I could see the IP) and ran "sudo rpi-update" from there.
Now, after a reboot, everything was perfect.


Saturday, March 02, 2013

A fast browser for the Pi

Came across this thread today, although it's a week or so old, dealing with a lightweight webkit and supposedly very fast browser for the Pi.

The Pi is a wonderful machine but, of necessity, it's lacking in resources such as CPU speed and RAM.
For this reason, browsing is at times painfully slow.
And I've tried a lot of browsers including QupZilla, Midori, Luakit, UZBL and many others.

I first tried the downloadable binary in Rasbpian but this just didn't work at all for me.
No idea why not.

Nothing for it but to compile from source.
The home page for this project is here in Google Code from where you can find out how to build from source.

In a fully-updated Raspbian, you run this command

git clone https://code.google.com/p/minimal-web-browser
Next, to avoid compile errors I had to install these two dependencies

libgtk2.0-dev
libwebkitgtk-dev

Thereafter, I was able to cd to minimal-web-browser/web-1.0, run make and sudo make install.
The compile and install were very quick and took less than a minute or two for a full install.

Running the command "web" brings up the browser showing Google in about 1 second.
Impressive.
Unfortunately as I'm traveling tomorrow, I didn't get much chance to really check it out but it certainly looks very promising.

Incidentally, I tried the compile also in Gentoo on my Pi.
Although it compiled there was an error (something to do with sudo install -g dialout).
Nevertheless, the command "web" did open up the browser with Google open.
However, after a few seconds, it seg faulted and disappeared.










Tuesday, February 26, 2013

Free up some space on your Pi SDCard

One of the problems about running an OS on your Pi (Raspberry Pi) is that the OS runs from an SDCard which is probably going to be either 4GB or 8GB.
That's not a lot.
Nevertheless, you can probably get away with as little as 2GB to run a fully-fledged (with full Desktop)  OS on your Pi as long as you take steps to manage the available free space.

Here's how I freed up quite a lot of space in some OSes I use on my Pi.

Raspbian
Delete the files syslog, kern.log and messages from /var/log/ and I retrieved 1.8GB of space.
I chose these simply because running the command (from /var/log/)

sudo du -csBM * | sort -rn | head -11
showed these three to be by far the largest log files.

Most people don't need, and never use, these logs but they consume a lot of space.
After deletion, they will be recreated next time you boot and start storing data about the computer's performance.
So, it's probably a good idea to remove these files from time to time.

Arch Linux
In Arch I got back 600 MB when I deleted all files from /var/cache/pacman/pkg/
These package files are stored simply to allow possible downgrading of installed packages.
Read about it here in the Additional Commands section (search for /var/cache/pacman/pkg)

Gentoo
Deleting everything in /var/tmp/portage/ freed up 600 MB of space.
Deleting everything in /usr/portage/distfiles/ gave me back a whopping 1.2 GB

Fedora
In Fedora, removing everything in the yum cache (/var/cache/yum) gave me 200MB of free space.
In addition, however, make sure you know how your swap is set up in Fedora.
I had created a 500 MB swap partition.
But, it seems Fedora prefers to set up a special Swap file in the root partition (which was designated as /swap0 and of 512MB size).
In /etc/fstab, only /swap0 was being mounted and not the swap partition that I had created myself independently of the config tool.
So, I revised fstab to mount /dev/mmcblk0p3 as swap and subsequently deleted /swap0.
Another 500 MB saved.




Tuesday, February 19, 2013

RPi boots when usb device plugged in

I came across this interesting post today which shows a Pi (powered but shutdown) will boot when a wifi dongle is plugged in.

Really?
Will it work for me?

I first tried it on my "old" (delivered in May 2012) Model B with 256 MB of RAM and it just didn't work no matter what usb device I plugged in (I tried three wifi dongles, 2 different usb keys, a SDCard reader, a powered usb hub and the usb plug from my X-Mini speaker).
No sign of any green light from any of these.

A bit disheartened at this stage but I still tried my new Model B (512 MB RAM) and this time it worked........with nearly everything.

The Pi booted without a murmur of complaint when I plugged in each of the following:

Wifi dongle MicroNet N150
Wifi dongle Belkin F5D7050 with zd1211 chipset
Wifi dongle Linksys AE1200
Usb from X-Mini speaker
Belkin powered usb hub
PNY 1GB usb key

However, it didn't boot when I plugged in 

Dreamgear SDCard reader
Any of a range of HP and Kingston usb keys.

Interesting certainly, but not sure if I'm ever going to find a use for this finding.
It's just so easy to flick the power switch on and off.



Saturday, February 16, 2013

Fedora + xfce + RPi + Conky

Yes, a captivating title if, perhaps, a little mysterious.
What am I talking about?

Well, I been using the Raspberry Pi for almost a year now and although it's very limited in terms of hardware power, it's an awful lot of fun trying to get acceptable performance out of this tiny, and very cheap, box.
The immense success of this little machine is testimony to the inventiveness of the computer hobbyist community.

I've played with a number of OSes on the RPi and these include Fedora 17 RC2.
It works well although it's not quite as smooth and polished as is Raspbian.
I've already posted about the problems I had getting Conky to "play nice" in Fedora/xfce.
Actually, whatever problems I had with Conky are almost certainly the fault of the xfce DE rather than Fedora itself.

When I posted before, the main problem with Conky in this environment was launching Conky always made the desktop icons invisible. In addition, with certain .conkyrc configurations the Conky colors were very different from what I had intended.
As I use a color-coded Conky to alert me to unusual activity in my OS, this is a serious defect.

Well, I've made some, but not complete, progress in this area.
Here are the results of my latest investigations in summary form:

i) As usual it is always critical for Conky to have the compositor enabled.
(Applications Menu -- Window Manager Tweaks -- Compositor Tab -- Check 'Enable display compositing'

With no other 'own window' options in .conkyrc than the following:

own_window                        yes
own_window_type               override
own_window_transparent   yes

conky displays in a black window as shown in the screenshot with all of the desktop icons fully visible.

But, by doing no more than restarting the window manager (xfwm4) by means of the shell command

killall xfwm4

the conky background becomes gloriously transparent as the next screenshot afirms.
Unfortunately, for reasons I cannot as yet explains, when the Gmail portion of the conky display shows a change in status (a change in number of new emails including when it reverts to '0' after opening new mail) the background turns black yet again.
Of course, issuing the above 'killall xfwm4' command brings back the transparency although this can be a bit of a pain if you get a lot of emails.
You can ease the problem to a small extent by making a keyboard shortcut (see section vi) for the above command.

ii) If the own_window options are changed to


own_window                 yes
own_window_type        normal
own_window_hints       undecorated,below,sticky,skip_taskbar,skip_pager

the conky window shows as 'partially' transparent, and once again without losing the icons.
See the third screenshot opposite.

The degree of transparency of the 'normal' window can be varied by moving the Inactive Windows Opacity slider which is found in Applications Menus -- Window Manager Tweaks -- Compositor Tab

Unfortunately, by reducing the opacity of the window, you also reduce the opacity (and therefore the visibility) of the font too.

Nevertheless, this option not only keeps the desktop icons but also is unaffected by incoming email.

So, as yet a 'perfect' solution is not available but it's not at all too bad.
My personal preference is for the own_window_type override option despite the annoyance of having to refresh the WM on receipt of emails.

Friday, February 08, 2013

Display web page data in Conky

A friend, who writes a small blog, asked me for some help taking the page view data from StatCounter for his blog and displaying it in Conky.

I haven't seen much on this topic other than this CrunchBang post.

Anyway, I decided to try my own approach and this is what's described here.
It's fairly tricky and involved so pay attention, please.

1. Get the data into a Conky-readable format (basically a text-file).

A LibreOffice or OpenOffice spreadsheet has a Link to External Data command which can be used for this purpose.
Open a new spreadsheet file, place cursor on cell A1 (of course, it can be anywhere, but let's assume you want it to look tidy) and open Insert > Link to External Data.

Now, in the External Data dialog that opens, place the url of your webpage in the url box (well, where else would you put it?)
In the same dialog, check the 'Update every' box in the bottom right corner and select how often you want to update it (I chose the default 60 seconds).
I found this particular dialog just a little temperamental. So, you have to place the mouse cursor at the end of the url in the url box and hit Return.
Then wait for the Import Options dialog to open. I selected the default Automatic option and hit Return again.
Now, the External Data dialog should reappear with entries in the 'Available tables/ranges' box.
In my case, as I want the information displayed in the tables, I selected 'HTML_tables'.
Hit OK and the information you want should show up in the spreadsheet.

Save this file with an appropriate name in the .ods format (mine was called stat.ods and I saved it to my ~ directory).

Now, Conky cannot read the .ods file.
So, we save the same file as a .csv file (comma-separated-values)
File > Save As > File Type > .csv

2. Create script to generate .csv file at selected frequencies to display current blog hit count

For this script, I made extensive use of the xdotool function which is very powerful with an enormous 'vocabulary'.
Essentially, it allows a script to mimic a series of keystrokes (and mouse movements too but I haven't used these).

Here's the bash script I made to update the .csv file I needed every two minutes (you can choose whatever you want):

#!/bin/bash 
##Open the file stat.ods in LibreOffice
          localc --minimized /home/paul/stat.ods "$@" &
## The 'sleep' commands are to ensure the keystrokes have enough time to  
## activate before the next command is called 
sleep 3
## Because the file, when it opens, asks if you want to update the live links 
## (links to webpage). You need to hit Return three times for this 
xdotool key Return Return Return 
sleep 3
## The following series of keystrokes save the .ods file as a .csv file of the same name 
## and in the same directory 
xdotool key alt+f 
sleep 2 
xdotool key alt+a 
sleep 2 
xdotool key Right 
sleep 2 
xdotool type ".csv" 
sleep 2 
xdotool key Return 
sleep 5 
xdotool key alt+s alt+y 
sleep 2 
xdotool key Return 
sleep 2
## Start an endless loop to periodically save the stat.ods file as stat.csv 
## Remember that stat.ods updates from the webpage because of the External Data Link 
while true; do
## For the scripted keystrokes to enter ONLY the stat.csv file, you need to 'steal' focus 
## from whatever other window you might have open at the time. 
## First, we need to identify what window is currently focussed so we can refocus it 
## when the updated stat.csv has been saved 
window_id=$(xdotool getwindowfocus)
## Now focus the stat.csv window to receive the keystrokes 
xdotool search --name stat.csv windowactivate 

## Open the File and Save it 
xdotool key alt+f ctrl+s 
xdotool key Escape 
sleep 0.25
## Return focus to the window from which it was stolen at the start of this loop 
xdotool windowactivate $window_id
## Shade the stat.csv spreadsheet to 'get it out of the way' until needed again
wmctrl -r "stat.csv" -b toggle,shaded
## Repeat this loop every 120 seconds 
sleep 120
         ## go back to the start of the 'while' loop 
done

Save the script I called mine losave (an abbreviation for LibreOffice Save) and make it executable. Although not entirely necessary, I also copied my losave file to /usr/bin/ so it can be used from anywhere.
Now, assuming you have LibreOffice Calc installed, launching losave (either from a terminal or by typing Alt-F2 and entering losave in the box) should open stat.ods in LibreOffice Calc, activate the link to your selected webpage and save the file as stat.csv at whatever frequency you have selected.


3. Create similar script to display location of last hit

Not surprisingly, what's needed here is almost exactly the same as in the previous script.
The major difference is that a different .ods file is required.
StaCounter displays hit location in the Recent Pageload Activity page available for each blog in your account.
I called this spreadsheet loc.ods (loc=Location).
The script is shown below without any explanatory notes as there is no significant difference from the earlier script.


#!/bin/bashlocalc --minimized /home/paul/loc.ods "$@" & 
sleep 2 
xdotool key Return Return Return 
sleep 2 
xdotool key alt+f 
sleep 2 
xdotool key alt+a 
sleep 2 
xdotool key Right 
sleep 2 
xdotool type ".csv" 
sleep 5 
xdotool key Return 
sleep 5 
xdotool key alt+s alt+y 
sleep 2 
xdotool key Return 
sleep 5 
while true; do 
window_id=$(xdotool getwindowfocus) 
xdotool search --name loc.csv windowactivate
xdotool key alt+f ctrl+s 
xdotool key Escape 
sleep 0.25 
xdotool windowactivate $window_id 
wmctrl -r "loc.csv" -b toggle,shaded 
sleep 115 
done

4. Now, we should have two periodically updated .csv files in our home directory (or wherever you placed them).

Here's some examples of what they look like:

stat.csv

,Today?,Yesterday?,This Month?,Total?,Settings, 
MyBlog1,0,0,0,0,Config,                       
MyBlog2,68,106,652,85371,Config,                       
MyBlog3,0,3,5,1689,Config,                       
MyBlog4,0,0,20,15849,Config,                       
MyBlog5,0,0,0,282,Config,                       
5 projects,68,109,677,103191,,,,,,,,,Real-Time Visitor Stats,,Project Settings,,, 
,User & Public Access,,Email Reports,,,

Here, information for a number of blogs is included, most of which are not active.
The only one of interest is MyBlog2.

loc.csv looks like this in part (I've taken out details of what pages were viewed).


,,,WinXP,,"Arizona,",,,,1920x1080,,United States,  
,,,Win7,,"Washington,",,,,1600x1200,,United States,  
,,,MacOSX,,"Para,",,,,1440x900,,Brazil,  
,,,WinXP,,"Andhra Pradesh,",,,,1024x768,,India,  
,,,Linux,,"Pais Vasco,",,,,1440x900,,Spain,

OK, so everything we need is available in these two files but it's mixed up with an awful lot of other stuff.
So, the challenge facing Conky is to pare out what we don't need and print what we do.


5. Sifting the useful information from the .csv files for display in Conky

We're now trying to convert the jumble in the above .csv files into what appears in this partial screenshot of Conky (running in Arch Linux).








The first and second lines above are simply text inputs.
The third line is of primary interest and displays hits so far today, the total for yesterday, the total for the month so far and, finally, the total for all time.

The algorithm used in Conky is this:

$alignr ${exec cat /home/paul/stat.csv | grep MyBlog2 | awk -F\, '{print $2}'}       ${exec cat /home/paul/stat.csv | grep MyBlog2 | awk -F\, '{print $3}'}      ${exec cat /home/paul/stat.csv | grep MyBlog2 | awk -F\, '{print $4}'}     ${exec cat /home/paul/stat.csv | grep MyBlog2 | awk -F\, '{print $5}'}
This is fairly straightforward (at least, in comparison with the next one (:-).
Basically, each section of the algorithm opens out the stat.csv file and selects only the line containing the blog information of interest (MyBlog2).
The 'awk' command breaks up the line into fields while designating the ',' as the field delimiter.
Subsequently, it prints out the second, third, fourth and fifth fields as the hits appropriate to the various categories.

Great, but how do we separate out from loc.csv the location from where came the last hit.

Here's the algorithm:

Last Hit From: $alignr ${exec cat /home/paul/loc.csv | cut -d "," -f 6,7 | sed 's/"Korea, Republic of"/Korea, Republic of/' | awk '!/"/' |sed 's/\,//g' | sed -n 2p}
Possibly the best way to explain what each section of the algorithm does, is to present how the original loc.csv file changes as it goes through each stage of the algorithm:
First, I'll point out that the country names for each location were "almost" invariably without quotes while all other location data (city, state or whatever) were always within quotes.
The one exception to this rule that I've found so far is South Korea (or Republic of Korea). For some strange reason, Statcounter displays 'Korea, Republic of' within quotes.
This explains the rather strange third section of the algorithm which does no more than substitute the name 'Korea, Republic of' within quotes to one without quotes.
Without this, all hits from South Korea would have shown up as blank spaces.

OK, so here's what we start with:


,,,Win8,,"Champagne-Ardenne,",,,,1600x900,,France, ,,,WinVista,,United Kingdom,,,,1280x1024,,, ,,,Linux,,United Kingdom,,,,1280x800,,, ,,,Linux,,"Texas,",,,,1024x600,,United States,
,,,Win8,,"Mazowieckie,",,,,1920x1080,,Poland,
,,,MacOSX,,"Stockholms Lan,",,,,1440x900,,Sweden,
Once again, I've taken out anything that might be construed as private.

Now, after application of the algorithm section 'cut -d "," -f 6,7', we get this:
Location,Host Name/Web Page/Referring Link
"Vendeuvre-sur-barse,"
"Champagne-Ardenne,"
France,
"Luton,"
United Kingdom,
,
"Wolverhampton,"
United Kingdom,
,
"Dallas,"
"Texas,"
United States,
"Warsaw,"
"Mazowieckie,"
Poland,
"Upplands-v�sby,"
"Stockholms Lan,"
Now, it's starting to look more handleable.
Let's see what the next algorithm section does (leaving out the Korea one as this only applies in rare circumstances).
So after cut -d "," -f 6,7 | sed 's/"Korea, Republic of"/Korea, Republic of/' | awk '!/"/' 
we get:

 Location,Host Name/Web Page/Referring LinkFrance,United Kingdom,,United Kingdom,,United States,Poland,Sweden,
So, now we have, other than the heading, just countries and commas. Getting closer.
Let's see what's next.
cat /home/paul/loc.csv | cut -d "," -f 6,7 | sed 's/"Korea, Republic of"/Korea, Republic of/' | awk '!/"/' |sed 's/\,//g' 

gives us

LocationHost Name/Web Page/Referring LinkFranceUnited Kingdom
United Kingdom
United StatesPolandSweden
Looking better as all of those commas have gone.
Now all we need to do is to pick out the last hit location which is in the second line.

Here's what the whole algorithm (cat /home/paul/loc.csv | cut -d "," -f 6,7 | sed 's/"Korea, Republic of"/Korea, Republic of/' | awk '!/"/' |sed 's/\,//g' | sed -n 2p)  gives us:

France
And that's it.
Any questions?