Create Playlist for Bible MP3's

I bought a copy of the English Standard Version Bible in mp3 form. Unfortunately, once you load it onto you MP3 player or smartphone, you have a hard time skipping around. I like to read a chapter from the Old Testament, a chapter from the New Testament, a chapter from the "wisdom literature", and a chapter from Psalms. Doing this, skipping around, and remembering which chapter you're on is painful.

That is... until you can create your MP3 play lists! To do the following, you have to have cygwin, Mac OS terminal, or a PC with Linux installed.

1. Then, with file names like:

   [...]
   Bible - ESV 0934.mp3
   Bible - ESV 0935.mp3
   Bible - ESV 0936.mp3
   Bible - ESV 0937.mp3
   Bible - ESV 0938.mp3
   Bible - ESV 0939.mp3
   Bible - ESV 0940.mp3
   Bible - ESV 0941.mp3
   [...]
   

   You determine which ranges of files you want for your custom playlist. For me, that was
   • Old Testament 1-441,688-941 (without "wisdom literature" or Psalms)
   • New Testament 942-1205
   • Wisdom Literature 442-483,637-687
   • Psalms 484-636
2. Now I create the M3U header for a given range. Let's say we're doing the Old Testament range:

   echo "#EXTM3U" > old_testament.m3u

3. Construct a sed command for each range like:

   /BEGINNING_NUMBER/,/ENDING_NUMBER/p;
   

   Then, concatenate all these sed commands together.

   e.g., for the Old Testament range above,you would create these sedcommands:

   /0001/,/0441/p; /0688/,/0941/p;

4. Now, stick that filter into the following command after navigating to the ESV Study Bible MP3 directory:

   find . -type f | \
       sort -u | \
       sed -n -e 's/^\.\///; YOUR_SED_COMMANDS_HERE'\
       >> old_testament.m3u
   

   e.g. For the Old Testament example:

   find . -type f | \
       sort -u | \
       sed -n -e 's/^\.\///; /0001/,/0441/p; /0688/,/0941/p;'\
       >> old_testament.m3u
   

Stick that .m3u onto your MP3 player or smartphone and you're off to the races.

For Android install MortPlayer and it will help you keep track of where you are in each of the playlists.

Creating one List

If you want to combine all the lists into one mega list that automatically switches between your subranges, use a python script like the following:

class RangeItr:
  def __init__(self,ranges):
    self.ranges_ = ranges
    self.outer_range_idx_ = 0
    self.inner_range_val_ = ranges[0][0]-1
  def __iter__(self):
    return self
  def next(self):
    cur_range = self.ranges_[self.outer_range_idx_]
    self.inner_range_val_+=1
    if self.inner_range_val_ > cur_range[1]:
      self.outer_range_idx_= (self.outer_range_idx_+1)%len(self.ranges_)
      cur_range = self.ranges_[self.outer_range_idx_] 
      self.inner_range_val_ = cur_range[0]
    return self.inner_range_val_


ranges =( RangeItr( ( (1,441),(688,941) ) ),
          RangeItr( ( (942,1205) ,) ),
          RangeItr( ( (442,483),(637,687) ) ),
          RangeItr( ( (484,636), ) )
        )
r = 0
f_num = 0
f = None
for i in range(0,10000):
    if i %200 == 0:
        f_num += 1
        f = open('bible_%02d.m3u'%f_num,'w')
        f.write('#EXTM3U\n')
    f.write("Bible - ESV %04d.mp3\n"%ranges[r].next())
    r = (r + 1 ) % len(ranges)

How to find the time of the last access of any file under a directory in Linux.

Here is a one-liner to find the time of the last access of any file under a directory in Linux:

date -d @`find PATH_TO_DIR -type f -exec stat -c %X '{}' \; | \
  perl -ne 'use List::Util qw[min max]; BEGIN {$max=0;} chomp; $max = max($max,$_); END { print $max ."\n";}'`

Here's a bash script to do this on each file or directory underneath a specified directory:

#!/bin/bash
# use like:
#   least_used_nodes.bash [PATH_TO_DIR]
#
# If PATH_TO_DIR is omitted, the current directory is used.
#
# leaf nodes with spaces in them are not supported

dir="."
if [ "$#" -ne 0 ] ; then
    dir=$1
fi

declare -a access_times
for n in `\ls -1 $dir | egrep -v '^\.{1,2}\$'`  ; do
    echo "Analyzing $n"
    d=`find $dir/$n -type f -exec stat -c %X '{}' \; |\
      perl -ne 'use List::Util qw[min max]; BEGIN {$max=0;} chomp; $max = max($max,$_); END { print $max ."\n";}'`
    #echo $d
    access_times[$d]="${access_times[$d]} $n"
done
echo ""
echo ""
#echo ${access_times[@]}

echo "Printing access times youngest to oldest"
for d in `echo ${!access_times[@]} | sed -r 's/\\s+/\\n/g;' | sort -r` ; do
    #echo $d
    list=${access_times[$d]}
    for n in $list ; do
        echo `date -d @$d +"%Y.%m.%d %kh%Mm"` $n
    done
done

Perl one-liner for only printing lines of standard in that are unique

Only print lines if they have never been seen on standard in:

perl -ne 'if (not defined %d) { %d = ();} if (not exists $d{$_}) { $d{$_} = 1; print $_}'

Used like:

printf "hi\nhi\nbye\n" | \
  perl -ne 'if (not defined %d) { %d = ();} if (not exists $d{$_}) { $d{$_} = 1; print $_}'
hi
bye

Martin Pechanec's slay - process killer in the console

Martin Pechanec's slay is the easiest way to kill processes from the Linux console without having to manually do all that ps ux| grep foo followed by kill -9 666 boilerplate. This script was inspired by the very similar slay command in QNX O.S.

To kill some kate instances with slay, you simply type

% slay kate

and slay will prompt you, one at a time, to kill the processes that have "kate" in their command name:

> slay: Process list:
UID        PID  PPID  C STIME TTY          TIME CMD
ubuntu  5834  5481  2 11:33 pts/90   00:00:00 kate
ubuntu  5839  5481  2 11:33 pts/90   00:00:00 kate
ubuntu  5843 32672  0 11:33 ?        00:00:00 kio_file [kdeinit] file /tmp/ksocket-ubuntu/klauncherN2EMKb.slave-socket /tmp/ksocket-ubuntu/kateliTBzb.slave-socket
ubuntu  5844 32672  0 11:33 ?        00:00:00 kio_file [kdeinit] file /tmp/ksocket-ubuntu/klauncherN2EMKb.slave-socket /tmp/ksocket-ubuntu/kateb8tyxa.slave-socket

> slay: Terminate:
ubuntu  5834  5481 11:33 pts/90   00:00:00 kate ? (y/n)[n]

Alternatively, if you know you want to smash every process with "kate" in its process name, you can supply the -f switch:

slay kate -f
> slay: Process list:
UID        PID  PPID  C STIME TTY          TIME CMD
ubuntu  5834  5481  0 11:33 pts/90   00:00:00 kate
ubuntu  5839  5481  0 11:33 pts/90   00:00:00 kate
ubuntu  5843 32672  0 11:33 ?        00:00:00 kio_file [kdeinit] file /tmp/ksocket-ubuntu/klauncherN2EMKb.slave-socket /tmp/ksocket-ubuntu/kateliTBzb.slave-socket

To use this awesome script, put the following in a file at $HOME/bin/slay and then put $HOME/bin: at the front of your $PATH variable:

#!/bin/ksh -
#-----------------------------------------------------------------------------
#
#     FILE:    slay
#     AUTHOR:  Martin Pechanec, 03-SEPT-2002
#
#     $Header:$
#     $Log:$
#
#-----------------------------------------------------------------------------
#
#     DESCRIPTION
#       slay - interactively kill by process name by sending
#              SIGKILL (kill -9) or (kill -s KILL) to the process(es)
#              selected
#
#       USAGE:
#         slay [-a|-c] [-s] [-f|-i] [-h] []
#
#       If no name or substring is specified then just the process
#       information is printed as in ps command.
#       It works on the current user processes run on all terminals.
#       The information printed out about the processes is in long
#       form by default (ps -f). By default the process termination
#       is interactive.
#       The options:
#         -a .. all processes (default current user's only)
#         -c .. current terminal for current user (default all terminals
#               for current user).
#         -s .. short items of the process list (default long)
#         -g (-sg) .. same as -s, but long args (default long)
#         -f .. force process termination (non-interactive)
#         -i .. info only, do not terminate anything
#         -h .. help
#
#       The  is a space separated list
#       of strings and they are matched against the 'ps' listing
#       line by line (see -s option). Not only the process name
#       is matched, but the whole process information. So, for example,
#       to terminate all the processes run on a tty pts/2 by a current
#       user, run:
#         slay pts/2
#
#       Other examples:
#         slay -s 1543   .. terminate the process with PID 1543
#         slay 1543      .. as above, but also terminate all its children
#         slay /usr/     .. terminate all the current user processes
#                           which include /usr/ in the process location path
#
#       January 20, 2004: Update to recoginize Linux. Changed kill -s KILL
#       to kill -9, since Linux did not want to end the program by the
#       former command. Replaced the eval echo by better filtering command.
#
#-----------------------------------------------------------------------------
#

# Uncomment the set -v line for debugging
# set -v

function vPrintUsage
{
  echo "Usage: slay [-a|-c] [-s[a]] [-f|-i] [-h] []";
  echo "             -a[ll]     .. all processes (default current user's only)";
  echo "             -c[urrent] .. current terminal for current user (default all terms)";
  echo "             -s[hort]   .. short items of the process list (default long)";
  echo "             -s[ar]g[s] (-g) .. short items but full args (default long)";
  echo "             -f[orce]   .. force (non-interactive)";
  echo "             -i[nfo]    .. info only, do not terminate anything";
  echo "             -h[elp]    .. help";
  echo "  Running plain slay will return all processes running for a current user.";
}

# -- Check the echo command for flags to force it not to print
# newline at the end of the printed line.
if [[ "`echo 'ble_ble\c'`" = "ble_ble\c" ]] ;
then
  EFLAG="-n"
  ENDER=""
else
  EFLAG=""
  ENDER="\c"
fi
ECHO="echo ${EFLAG}"
UNAME=`uname`

# -- Set Unix type flags
integer LINUX_FLAG
let LINUX_FLAG=0

# -- Set Unix type
if [[ ${UNAME} = "HP-UX" ]] ;
then
  if [[ ${UNIX95:-"__Unix95NotSet__"} = "__Unix95NotSet__" ]] ;
  then
    echo "slay: WARNING: UNIX95 variable undefined .. exporting UNIX95=true"
    export UNIX95="true"
  fi

  if [[ -z ${UNIX95} ]] ;
  then
    echo "slay: WARNING: UNIX95 variable empty .. exporting UNIX95=true"
    export UNIX95="true"
  fi
else
  if [[ ${UNAME} = "Linux" ]] ;
  then
    let LINUX_FLAG=1
  fi
fi

# -- Check the existence of the dash parameter
# and set inputs to local variables. The dash option
# can be position anywhere in the line.
# PS_PARAM initial value is all processes for current user only
# on all terminals (-u ${USER})

SLAY_PID=$$
PROCESS_NAME_LIST=""
integer ALL_FLAG
integer CURRENT_FLAG
integer FORCE_FLAG
integer INFO_FLAG

PS_EXEC="ps"
PS_PARAM="-u ${USER}"
PS_SHORT_PARAM="-f"
PS_O_USER_PARAM=""
AWK_FIELD='$2'
AWK_FILTER='$2, $3'

ARGS_PS_SHORT_PARAM="-o user,pid,args"

let ALL_FLAG=0
let CURRENT_FLAG=0
let FORCE_FLAG=0
let INFO_FLAG=0
let SHORT_FLAG=0

# Parse input arguments
while [[ $# -gt 0 ]] ;
do
  ARGUMENT=$1
  case ${ARGUMENT} in
    -a|-all|--all)
      if [[ 0 -ne ${CURRENT_FLAG} ]] ;
      then
        vPrintUsage
        return 0
      fi
      PS_PARAM="-e"
      PS_O_USER_PARAM="-o user="
      let ALL_FLAG=1
      shift
      continue
      ;;

    -c|-current|--current)
      if [[ 0 -ne ${ALL_FLAG} ]] ;
      then
        vPrintUsage
        return 0
      fi
      PS_PARAM=""
      let CURRENT_FLAG=1
      shift
      continue
      ;;

    -s|-short|--short)
      PS_SHORT_PARAM="-o user,pid,ppid,stime,tty,time,fname"
      AWK_FIELD='$2'
      AWK_FILTER='$2, $3'
      let SHORT_FLAG=1
      shift
      continue
      ;;

    -g|-sg|-sargs|-shortargs|--shortargs)
      PS_SHORT_PARAM="-o user,pid,args"
      AWK_FIELD='$2'
      AWK_FILTER='$2, $3'
      let SHORT_FLAG=1
      shift
      continue
      ;;


    -f|-force|--force)
      if [[ 0 -ne ${INFO_FLAG} ]] ;
      then
        vPrintUsage
        return 0
      fi
      let FORCE_FLAG=1
      shift
      continue
      ;;

    -i|-info|--info)
      if [[ 0 -ne ${FORCE_FLAG} ]] ;
      then
        vPrintUsage
        return 0
      fi
      let INFO_FLAG=1
      shift
      continue
      ;;

    -h|-help|--help)
      vPrintUsage
      return 0
      ;;

    -*)
      vPrintUsage
      return 1
      ;;

    *)
      ;;
  esac

# No dash parameter .. store the 
  PROCESS_NAME_LIST="${PROCESS_NAME_LIST} ${ARGUMENT}"
  shift

done

# -- Parameters and input process substring list set
# If there is not name list, then just list the names
# of the processes according to the -a flag.
# If the -a is not specified, then list processes
# for current user only. If -s is not specified, then
# list long listings.

# ps options:
#   -e .. every running process
#   -f .. full listing
#   -u ${USER} .. current user on all terminals
#
# All users setting implies all terminals.
# ps -e    .. short item,  all users -->  all terminals
# ps -e -f .. long item,   all users -->  all terminals
# ps       .. short item,  current user,  current terminal
# ps -f    .. long item,   current user,  current terminal
# ps -f -u ${USER} .. long item,  current user,  all terminals
# ps -u ${USER}    .. short item, current user, all terminals
#
# The ps command line will look like this:
#   ps ${PS_PARAM} ${PS_SHORT_PARAM}

# Check whether there are processes to be slayed
if [[ -z ${PROCESS_NAME_LIST} ]] ;
then

# No process names to process .. just print the processes info.
# Remove the new lines from the list .. second line!
  PROCESS_ID_LIST=`${PS_EXEC} ${PS_PARAM} ${PS_SHORT_PARAM} | awk "{print ${AWK_FILTER}}" | grep -v ${SLAY_PID} | awk '{print $1}' | grep -v PID`
  PROCESS_ID_LIST=`echo ${PROCESS_ID_LIST} | sed 's/^[ ][ ]*//'`

# This has to be there since HP does not allow more than 32 items in the list
  if [[ 0 -ne ${ALL_FLAG} ]] ;
  then
    ${PS_EXEC} ${PS_PARAM} ${PS_SHORT_PARAM}
  else
    ${PS_EXEC} ${PS_SHORT_PARAM} -p """$PROCESS_ID_LIST"""
  fi
  return 0
fi

# -- Some processes were listed .. create a list of all
# the process id's for processes to be slayed
PROCESS_ID_LIST=""
for PROCESS_NAME_ITEM in ${PROCESS_NAME_LIST} ;
do
  ITEM_PROCESS_ID_LIST=`${PS_EXEC} ${PS_PARAM} ${PS_SHORT_PARAM} | grep ${PROCESS_NAME_ITEM} | awk "{print ${AWK_FILTER}}" | grep -v ${SLAY_PID} | awk '{print $1}' | grep -v PID`
  ITEM_PROCESS_ID_LIST=`echo ${ITEM_PROCESS_ID_LIST} | sed 's/^[ ][ ]*//'`
  PROCESS_ID_LIST="${PROCESS_ID_LIST} ${ITEM_PROCESS_ID_LIST}"
done
PROCESS_ID_LIST=`echo ${PROCESS_ID_LIST} | sed 's/^[ ][ ]*//'`

# -- Check the process id list
if [[ -z ${PROCESS_ID_LIST} ]] ;
then

# If the long process list was used (default) and the process
# was not found, try the short with long arguments
  if [[ 0 -eq ${SHORT_FLAG} ]] ;
  then
    PROCESS_ID_LIST=""
    for PROCESS_NAME_ITEM in ${PROCESS_NAME_LIST} ;
    do
      ITEM_PROCESS_ID_LIST=`${PS_EXEC} ${PS_PARAM} ${ARGS_PS_SHORT_PARAM} | grep ${PROCESS_NAME_ITEM} | awk "{print ${AWK_FILTER}}" | grep -v ${SLAY_PID} | awk '{print $1}' | grep -v PID`
      ITEM_PROCESS_ID_LIST=`echo ${ITEM_PROCESS_ID_LIST} | sed 's/^[ ][ ]*//'`
      PROCESS_ID_LIST="${PROCESS_ID_LIST} ${ITEM_PROCESS_ID_LIST}"
    done
    PROCESS_ID_LIST=`echo ${PROCESS_ID_LIST} | sed 's/^[ ][ ]*//'`
  fi

# Check the new process ID list again
  if [[ -z ${PROCESS_ID_LIST} ]] ;
  then
    echo "slay: No such process(es)."
    return 0
  fi
fi

# -- Filter the process id list. Linux must not have the triple quotes
if [[ 0 -ne ${LINUX_FLAG} ]] ;
then
  PROCESS_ID_LIST=`${PS_EXEC} ${PS_SHORT_PARAM} -p ${PROCESS_ID_LIST} | awk "{print ${AWK_FILTER}}" | awk '{print $1}' | grep -v PID`
else
  PROCESS_ID_LIST=`${PS_EXEC} ${PS_SHORT_PARAM} -p """${PROCESS_ID_LIST}""" | awk "{print ${AWK_FILTER}}" | awk '{print $1}' | grep -v PID`
fi
PROCESS_ID_LIST=`echo ${PROCESS_ID_LIST} | sed 's/^[ ][ ]*//'`

# -- Check the filtered process id list
if [[ -z ${PROCESS_ID_LIST} ]] ;
then
  echo "slay: No such filtered process(es)."
  return 0
fi

# -- Some processes do exist .. print the processes info any case
# Print the header only if there will be some action
if [[ 0 -eq INFO_FLAG ]] ;
then
  echo "> slay: Process list:"
fi
${PS_EXEC} ${PS_SHORT_PARAM} -p """${PROCESS_ID_LIST}"""

# Check whether the information only .. if so, then stop here
if [[ 0 -ne INFO_FLAG ]] ;
then
  return 0
fi

# Echo process termination loop
if [[ 0 -eq FORCE_FLAG ]] ;
then
  echo ""
  echo "> slay: Terminate:"
fi

for PROCESS_ID_ITEM in ${PROCESS_ID_LIST} ;
do
  if [[ 0 -ne FORCE_FLAG ]] ;
  then
    kill -9 ${PROCESS_ID_ITEM}
  else
    PROCESS_ID_STRING=`${PS_EXEC} ${PS_O_USER_PARAM} -o user= -o pid= -o ppid= -o stime= -o tty= -o time= -o args= -p ${PROCESS_ID_ITEM}`

    if [[ ! -z ${PROCESS_ID_STRING} ]] ;
    then
      ${ECHO} "${PROCESS_ID_STRING} ? (y/n)[n] ${ENDER}"
      read CHECK
      if [[ "${CHECK}" = "y" || "${CHECK}" = "Y" ]] ;
      then
        kill -9 ${PROCESS_ID_ITEM}
      fi
    fi
  fi
done

return 0

#
#-----------------------------------------------------------------------------
#

Detect duplicate files in Linux or cygwin

In doing some back up of pictures and movies, I wanted to detect duplicate files and delete them. I found this post, but it doesn't run efficiently on large file sets since it md5sum's all files regardless of whether a size match amongst all the other files has been found. A stat command on a file is much quicker than actually computing the md5sum since a stat command looks at the file's header information, whereas an md5sum command has to read the entire file off the disk. Thus, I only want to do an md5sum of files where the length matches other files, since only files with a length equal to another file's length can possibly be a duplicate of another file.

We come to the following "one-liner", which only md5sum's the files that have identical length to some other file in the system:

find . -type f -exec stat --printf='%32s ' {} \; -print |\ sort -rn |\ uniq -d -w32 --all-repeated=separate |\ awk '(NF > 0){ system( \ "md5sum \"`echo \"" $0 "\"|\ sed -r \"s/^ *[0-9]+ *//\" `\" |\ cut -c 1-32 | tr -d [:space:] " ); printf " %32s %s", $1, $2 ;\ for (i = 3; i <= NF; i++) printf $i " "; \ printf "\n;\ sort -r |\ uniq -d -w65 --all-repeated=separate |\ awk '{for (i = 3; i <= NF; i++) printf $i " ";print "";}'

# Print size of every file to find duplicated file sizes # sort to put duplicate files sizes together ( for uniq ) # print all files that have file sizes equal to any other file # for the remaining files, insert the md5sum after the file size # do the md5sum while allowing for spaces in the file names # ( More than 1 consecutive space in a file name is not allowed..) # print out the file size with a fixed width for future comparisons and also print the file name # re-sort to catch duplicates of _different_ files with _identical_ sizes # Compare the file size and md5sum for every file and print multiples # get rid of the file size and md5sum and simply print out the duplicate file names by themselves

If you have a ton of files that you're analyzing, you may want to see progress happening. Due to the nature of sort, it won't generate output until all its input lines are received. We can intercept and log the intermediate output so you can see progress by modifying the command with tee:

find . -type f -exec stat --printf='%32s ' {} \; -print |\
    tee find_stat.log |\
    sort -rn |\
    uniq -d -w32 --all-repeated=separate |\
    awk '(NF > 0){
        system( \
            "md5sum \"`echo \"" $0 "\"|\
            sed -r \"s/^ *[0-9]+ *//\" | tee -a f.log`\" |\
            cut -c 1-32 | tr -d [:space:] " );
"        printf " %32s %s", $1, $2 ;\
        for (i = 3; i <= NF; i++) printf $i " "; \
        printf "\n;\
    }' |\
    tee md5sum_generation.log |\
    sort -r |\
    uniq -d -w65 --all-repeated=separate  |\
    awk '{for (i = 3; i <= NF; i++) printf $i " ";print "";}' | \
    tee repeated_files.log

Then, in a seperate window from the above command, you can run tail:

tail -f find_stat.log

or

tail -f md5sum_generation.log

Using boost's replace_all_regex_copy

#include <iostream>
#include <string>
#include <algorithm>
#include "boost/algorithm/string/regex.hpp"

using namespace std;

int main(void) {
    string orig_str="FUBAR";
    boost::regex re("UBA");
    std::string replacement="ooba";
    string result = boost::algorithm::replace_all_regex_copy(
        orig_str,re,replacement);
    cout << result << endl;
    return 0;
}

Yields:

FoobaR

Tabata Protocol Soundtrack MP3

I've recently become interested in the Tabata Protocol ( See 1, 2, and 3.) I tried to use a number of Tabata Protocol apps in Android, but all the apps I tried required the Android phone to not be in locked screen mode. If you are in locked screen mode, then these apps won't properly tell you when to start and stop running. If you're running while holding an Android phone, you clearly need to be able to lock the screen, otherwise your grip on the phone will punch all sorts of buttons or turn off the app.

What about the tried and true method? The mp3. Thankfully, you can play an mp3 on an Android phone with the screen locked. Now, I couldn't just generate one mp3 manually in Audacity. I had to completely solve the problem for everyone who wants to create a timing or HIIT mp3 with any combination of starts and stops with any kind of spacing between them.

BEHOLD! The Tabata Protocol Soundtrack generator!

This little web page will generate a HIIT soundtrack or a Tabata Protocol soundtrack according to your specification. If you simply go to the page and click the generate button, you'll get the classic Tabata Protocol timing that Mr. Tabata himself used whilst doing his study.

Shoot me some feedback in the comments if you find it useful.

Python one-liner to convert hex numbers to decimal

A one-line to convert all hex numbers from standard in to decimal:

cat log | \
  python -c 'import sys,re;[sys.stdout.write(re.sub("0x([a-fA-F0-9]+)",lambda m : str(int(m.group(0),16)),l)) for l in sys.stdin]'

How to automatically click ok on "Authentication Required" popup

Lately, I've been doing some web scraping automation. For the most part, I'm using Selenium, but Selenium doesn't seem to be able to automatically click "OK" when an authentication window pops up.

This python code, works with Redhat's Dogtail to auto-magically click "OK" button on an "Authentication Required" popup:

from dogtail import tree
from dogtail.utils import run
from time import sleep
from os import environ, path, remove
import sys
environ['LANG']='en_US.UTF-8'

#print tree.root.dump()
firefox = tree.root.application('Firefox')

#print firefox.dump()
#print dir(firefox)
#print firefox.findChildren()
auth_required = firefox.childNamed("Authentication Required")
#auth_required = firefox.child(name="Authentication Required")
ok_button = auth_required.childNamed("OK")
ok_button.click()
print 'ok button clicked now'
#print dir(ok_button)

NOTE:

Doing gui-automation this way may be a "bad idea", but I'm only doing this automation to scrape an internal webpage, and it only has to work on a very specific configuration -- Ubuntu 10.04 with Firefox.

Script to nuke specific onclick() behavior in Rally tool.

When you're editing your user stories in Rally the most irritating part is that you can't open all your stories in new tabs by holding down the ctrl key and clicking all the story links. The following greasemonkey script enables you to do that:

// Makes Rally tool do what it is supposed to.
// 2009.03.17, Ross Rogers
// Copyright (c) 2005, Mark Pilgrmim
// Released under the GPL license
// http://www.gnu.org/copyleft/gpl.html
//
// --------------------------------------------------------------------
//
// This is a Greasemonkey user script.
//
// To install, you need google's Chrome or 
// Greasemonkey: https://addons.mozilla.org/en-US/firefox/addon/748
// Then restart Firefox and revisit this script.
// Under Tools, there will be a new menu item to "Install User Script".
// Accept the default configuration and install.
//
// To uninstall, go to Tools/Manage User Scripts,
// select "Hello World", and click Uninstall.
// --------------------------------------------------------------------
//
// ==UserScript==
// @name          Rally Onclick Nuke
// @namespace     http://diveintogreasemonkey.org/download/
// @description   Nukes the "onclick" attribute from user story links so you can 
//                CTRL click a link and have it open in a new tab. Also creates
//                "accesskey" shortcuts for Save (alt+a), "edit_tags" (alt+t),
//                and "apply" button (alt+a)
// @include       https://*rally.sp*
// @include       https://*slm/ar/edit.sp*
// ==/UserScript==
                                                                                                                      
function remove_story_onclicks() {
  
  console.log("running remove_story_onclicks\n");
  var links = document.getElementsByTagName("a");
  for (i = 0; i < links.length; i++) {
    var node = links[i];
   var link = node.getAttribute("href");
   var className = node.getAttribute("class");
   var is_edit_button = (className && className.indexOf("sprite-edit") > -1);
   if (is_edit_button) {
       var newHref = node.getAttribute("onclick").replace(/.*popup\('([^']*?)'.*/, "$1");
        node.setAttribute("href",newHref);
    }
   if (link && link.indexOf("slm/detail/ar/") > -1 || 
       is_edit_button ) {
      if (node.getAttribute("onclick")) {
        node.removeAttribute("onclick");
        console.log("removed attribute onclick\n");
      }
   }
  }
  var buttons = document.getElementsByTagName("button");
  for (i = 0; i < buttons.length; i++) {
    var button = buttons[i];
    console.log("Iterating through buttons\n");
    console.log(button.innerHTML);
    if (button.innerHTML.match(/^Save$/) ) {
        button.setAttribute("accesskey","s");
        console.log("setting Save button's accesskey to 's'\n"); 
    };
    if (button.innerHTML.match(/^Apply$/) ) {
        button.setAttribute("accesskey","a");
        console.log("setting Apply button's accesskey to 'a'\n");
    };
  };
  
  var edit_tags_link = document.getElementById("edit_tags");
  if (edit_tags_link) {
      edit_tags_link.setAttribute("accesskey","t");
      console.log("setting edit_tags-link's accesskey to 't'\n");
  };
  
  
}


// override onclicks everywhere 
var links = document.links;
for (i = 0; i < links.length; i++) {
  var node = links[i];
  console.log(node.onclick);
  if (node.getAttribute("onclick")) {
    console.log("wrappering onclick\n");
    var original_onclick = node.onclick;
    function wrapper_onclick(e ) {
     var ret = original_onclick(e);
     remove_story_onclicks();
     return ret;
    }
    node.onclick = wrapper_onclick;
  }
}

remove_story_onclicks();
function document_onclick() {
    console.log("document_onclick() called\n");
    remove_story_onclicks.apply(this,arguments);
}
document.onclick=document_onclick;

To install this, put it in a file named rally_onclick_nuke.user.js, then load it in chrome by dragging the file onto the browser. It should ask you at the bottom of the page if you want to install the extension, and you press yes.

How to debug environments in Unix or Linux

How to tkdiff two environments

This scenario occurs frequently:
Frank can run program Chutney Aggregator, but Bob cannot.

What differs between Frank's and Bob's environment variables such that Frank can run Chutney Aggregator and Bob cannot?

If you run the command env, your shell will dump out all the environment variables. e.g.

KDE_MULTIHEAD=false
SSH_AGENT_PID=5349
DM_CONTROL=/var/run/xdmctl
TERM=xterm
SHELL=/bin/bash
XDM_MANAGED=/var/run/xdmctl/xdmctl-:0,maysd,mayfn,sched,rsvd,method=classic
...

Unfortunately, if we were to compare the output of command env on Frank's and Bob's, it would be extremely tedious, because env seems to just dump the environment variable hash in whatever fashion it pleases.

In order to be able to compare the environments, we need to sort the output from env.

env | sort

COLORTERM=
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-aEV1NDyS2z,guid=c31c98d5c8ad88de33dbfa0046ec2f30
DESKTOP_SESSION=default
DISPLAY=:0.0
...

If we put the output of this one-liner into a file, we can use tkdiff to compare the Frank's and Bob's environments in a visual way.

Frank performs this command in his terminal:

 env | sort > ~/tmp.shell_env

Bob does the same command in his terminal:

 env | sort > ~/tmp.shell_env

Now, we can compare the two environments:

 tkdiff ~Frank/tmp.shell_env ~Bob/tmp.shell_env

I find that the above process determines will crack most environment issues.


How to import a different environment

In tcsh, if I have to copy another person's environment variable wholesale, you can use this one-liner to create a tcsh script from their env which you may then source in another environment in order to reproduce their issues.

env | sort | perl -ne 's/'"'"'/'"'"'"'"'"'"'"'"'/g; \
   s/^([\w_]+)=(.*)/setenv \1 '"'"'\2'"'"'/; print $_' > ~/tmp 
source ~/tmp

If you back up your environment in the same way and then iteratively load part of their environment ( binary search?) you can pin point the difference that causes 1 environment to succeed and another to fail.


Root-Causing an Environment Issue

If you do have an environment dump from a person who is able to successfully run the program that you want to run, then you can possibly determine the root cause of the issue by iteratively importing their environment and testing whether you are now able to run the program.

To start, make sure that the problem you're encountering is caused by environment variable issues. Back up your environment as a script which you can use to restore it.

 env | sort | perl -ne 's/'"'"'/'"'"'"'"'"'"'"'"'/g; \
   s/^([\w_]+)=(.*)/setenv \1 '"'"'\2'"'"'/; print $_' > ~/my_env

Take the dump that the other person created from their environment and import all variables into your shell by creating a script from their environment per How to debug environments in Unix or Linux#How to import a different environment.

Import their environment by sourcing the script you created.

source ~/other_env

Run the program and see if it passes. If it does, you know that the root cause of your issues is the difference between your environment variables and the other person's environment variables.

Now comes the binary search part.

Restore your environment:

source ~/my_env

* Comment out the first half of the other person's environment script*. Then source

their modified environment script:

source ~/other_env

Run the program. If the program runs, then the environment variable that was causing the issue is in the 2nd half of the ~/other_env script. *You have just eliminated the first half of the script as the source of your environment problem!*

If the program doesn't run, the bad environment variable is in the 1st half of the ~/other_env script. *You have just eliminated the 2nd half of the script as the source of your environment problem!* Uncomment the first half of the script and comment out the 2nd half.

Keep repeating this process with the remaining un-commented sections of environment variables by restoring your environment ( source ~/my-env ) and smashing over your environment with different pieces of ~/other_env until you find the problem environment variable.

Tricks
Identifying and Removing Unused Variables with a 'one-liner'

Unused struct members can clutter and obfuscate Specman code. For the purpose of eliminating this annoyance I have created the following one-liner that may help you identify unused Specman members.

grep -i --perl-regex -h '^ *[a-zA-Z][\w_]+ *:.*;' *e |           \
 grep -v var |                                                \
 perl -ne                                                     \
    's/^ *//;                                                 \
     s/([\w_]+).*/$1/;                                        \
     print "$_";' |                                           \
 sort -u |                                                    \
 perl -ne                                                     \
    'chomp;                                                   \
     $ret_val = `grep -h $_ *e |                              \
         grep -v --perl-regex '"'"'^\\s*$_\\s*:.*;'"'"' |     \
         grep -v --perl-regex '"'"'^\\s*keep\\s*(soft\\s*)?$_.*==.*'"'"'`; \
     if( $ret_val =~m/^\s*$/ ) {                              \
         print "$_\n";                                        \
     }'

This little 'one-liner' will look through all the *e files in your current directory and look for all struct or unit members and then try to see if each of the members is used in any other code in the current directory.

Aliases

Aliases for the cshell.

diff_dirs  

Pull up tkdiff for each file that is different between two directories:

alias diff_dirs 'diff --brief "\!:1" "\!:2" |\\
 perl -pne '"'"'if (m/([\w-_\.\/]+) +and +([\w-_\.\/]+) +differ/)\\
 { `tkdiff "$1" "$2"` ;}'"'"

svn_diff_dirs

tkdiff the current directory with what all files used to be

alias svn_diff_dirs 'svn status |\\
 grep M |                     \\
 perl -ne '"'"'s/^M *//;      \\
     chomp;                   \\
     `tkdiff .svn/text-base/$_.svn-base $_`'"'"

Installing Wireless Drivers on Lenovo 3000 n100 Laptop w/ Intel's IPW3945ABG wireless setup on Fedora Core 6 ( FC6 )

Installing Wireless Drivers on Lenovo 3000 n100 Laptop
w/ Intel's IPW3945ABG wireless setup
on Fedora Core 6 ( FC6 )

Yammering's
I recently got a warranty replacement on a Sager laptop machine I'd bought ~4 years ago for college. Sager build's a hard core machine. It had a p4 with 1 GB ram and a radeon 9000 graphics card and a 17" monitor. It weight probably around 10 lbs and had a 15 minute battery life (literally), but it was worth it. I needed the power, because it was my only machine and I was still very much a gamer. Over time it showed other issues. Specifically it had a massive heating issue. After 2 years of use, with a significant amount of protein folding, it started having heat failures. It had to be babied constantly -- propped up on stuff to allow air flow -- big pain. I started sending it in for warranty repair. Huzzah for warranties. I'm so hard on stuff I can't imagine buying a laptop without a warranty. Finally, one of the warranty requests they gave me a new computer! Which happened to be much cooler than my old one. A Lenovo 3000 n100 laptop. The rest of this post describes how to get wireless going on this laptop after having installed Fedora Core 6.


Installing Wireless Drivers
The easiest way to install the ipw3945 driver is to use Axel Thimm's repository at atrpms.net . This is what I'm going to show you how to do.

-Update-
Assuming you have a wired connection to the internet. Update your machine using yum. This is because the repository from which you will be downloading later on uses the latest released Fedora Core 6 kernel.

yum update # <-- requires root permission. If you don't want to answer 'yes' use 'yum update -y' -Install Repository-
Start a new terminal and log in as root. As root, create a new file named
/var/yum.repos.d/atrpms.repo
atrpms.repo using your favorite editor (my favorite is jedit )
insert the following text ( per atrpms.net's instructions )
[atrpms]
name=Fedora Core $releasever - $basearch - ATrpms
baseurl=http://dl.atrpms.net/fc$releasever-$basearch/atrpms/stable
gpgkey=http://ATrpms.net/RPM-GPG-KEY.atrpms
gpgcheck=1

save and exit

-Install your version of the kernel module-
This should be installed first ( as opposed to letting yum auto-resolve the dependency ), because you'll want to specify which kernel module. Especially, if you're using the kernel modified for xen. In the console with root permissions, issue the commands:
yum install 'ipw3945*kmdl*' # <-- with root permissions

If you want to install the xen kernel module use the command:
yum install 'ipw3945*kmdl*xen*' # <-- with root permissions

-Install the ipw3945 module-
Now that you have chosen the kernel module, use yum to install the driver
yum install ipw3945 # <-- with root permissions

Once yum is done, the driver is installed! Reboot your machine.

-Using the ipw3945 driver-
Using the ipw3945 is not pretty, but it works. To get things going you'll need to load the ipw3945 kernel module and start the ipw3945 daemon:
modprobe ipw3945 # <- load ipw3945 kernel module
/sbin/ipw3945d # <- start ipw3945 daemon

Now you'll have to use some linux tools to connect to a wireless station.

Use

/sbin/iwlist scan

to find an network. Example output

eth1 Scan completed :
Cell 01 - Address: 00:12:17:32:B3:06
ESSID:"free_but_dont_abuse"
Protocol:IEEE 802.11bg
Mode:Master
Channel:6
Encryption key:off
Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 6 Mb/s; 9 Mb/s
11 Mb/s; 12 Mb/s; 18 Mb/s; 24 Mb/s; 36 Mb/s
48 Mb/s; 54 Mb/s
Quality=59/100 Signal level=-72 dBm Noise level=-72 dBm
Extra: Last beacon: 72ms ago

Note the 'essid' of your prefered network.

Then connect to the access point using

/sbin/iwconfig eth1 essid free_but_dont_abuse

For some reason you also have to 'activate' the interface too.

system-config-network

highlight your wireless connection on the first tab and click activate.