17 August 2009

My .bashrc

So I expect that this blog will contain a lot of moaning about the sorry state of production computing, of systems administration and of the world in general. But the reason that I'm in this field is because it's way too much fun. Specifically, I fell in love with GNU/Linux and The UN*X Way, and when it turned out that I could actually get paid to fiddle with this stuff… well, you can guess the story.

So here's the first post in what I hope will be a series comprising the majority of content here, a post about the details that make working and playing in a UN*X environment such a joy. Rather than simply pasting the whole stupid thing, I'm going to pull out and comment on the interesting bits. So here we go:

I get incredibly annoyed when I start some random terminal emulator (e.g. Emacs's own terminal-mode) and it doesn't accept ANSI color escape sequences. So here's a Boolean variable that I test for when setting various color options:
 1:  use_color=false
2: safe_term=${TERM//[^[:alnum:]]/.} # sanitize TERM
3:
4: # A good guess of whether this $TERM can display color
5: if [[ -f /etc/DIR_COLORS ]] ; then
6: grep -q "^TERM ${safe_term}" /etc/DIR_COLORS && use_color=true
7: elif type -p dircolors >/dev/null ; then
8: if dircolors --print-database | grep -q "^TERM ${safe_term}" ; then
9: use_color=true
10: fi
11: fi

I use a colorized prompt, but I couldn't decide which color I liked best – so I randomize it. I also have a little function that alerts me if the previous command exited with non-zero (i.e. abnormal) status:
12:  function returncode {
13: returncode=$?
14: if [ $returncode != 0 ]; then
15: echo "[$returncode]"
16: else
17: echo ""
18: fi
19: }
20: if ${use_color} ; then
21: COLORS=(30 32 33 34 35 36 37)
22: n=$(($RANDOM % ${#COLORS[*]}))
23: b=$(($RANDOM % 2))
24: if [[ ${EUID} == 0 ]] ; then
25: PS1='\[\033[01;31m\]$(returncode)\[\033[01;31m\]\h \[\033[01;34m\]\W \$ \[\033[00m\]'
26: else
27: PS1='\[\033[${b};31m\]$(returncode)'"\[\033[01;${COLORS[n]}m\]\u@\h \[\033[01;34m\]\w \$ \[\033[00m\]"
28: fi
29: else
30: if [[ ${EUID} == 0 ]] ; then
31: # show root@ when we don't have colors
32: PS1='$(returncode)\u@\h \W \$ '
33: else
34: PS1='$(returncode)\u@\h \w \$ '
35: fi
36: fi

``less'' can do a lot more1 than you think. Here I set environment variables so that manpages are in color, and alias my invocation of less so that it can interpret ANSI color escape sequences – so now ``ls –color | less'' doesn't spam you with garbage… and the color option to grep works as well:
37:  export LESS_TERMCAP_mb=$'\e[01;31m'
38: export LESS_TERMCAP_md=$'\e[01;31m'
39: export LESS_TERMCAP_me=$'\e[0m'
40: export LESS_TERMCAP_so=$'\e[01;44;33m'
41: export LESS_TERMCAP_se=$'\e[0m'
42: export LESS_TERMCAP_us=$'\e[01;32m'
43: export LESS_TERMCAP_ue=$'\e[0m'
44:
45: export PAGER="less -R"
46: alias less='less -R' # better than -r

Wait, you didn't know there was a –color option to grep!?
47:  if ${use_color}; then
48: export GREP_COLOR='01;33'
49: alias grep='grep --color=yes'
50: fi

My first-ever blog post showed how to do this; I'm including it here for completeness' sake: here's how to slow down your computer when compiling:
51:  if [ -a /proc/cpuinfo ]; then
52: export CONCURRENCY_LEVEL=$(($(grep -c processor /proc/cpuinfo) * 2 + 1))
53: export MAKEOPTS="-j${CONCURRENCY_LEVEL}"
54: fi

And last but not least, here are some handy aliases:
alias ipgrep='egrep -o "(([0-9]{1,3}\.){3}[0-9]{1,3})"' # iptables -L|ipgrep
alias rot13='tr A-Za-z N-ZA-Mn-za-m' # sbe terng whfgvpr
alias cg='egrep -v "^($|[[:space:]]*#|;)" ' # strip out comments

I do want to give credit to the authors of one of the best UN*X books I've ever read: Unix Power Tools from O'Reilly. Not only did this book teach me lots of *fu, but it advocates this very culture of hacking, documenting & sharing .bashrc. I recommend it to almost everybody that I know who expresses an interest in using UN*X better; it's great stuff.

14 August 2009

Publishing to Blogger with... Emacs!

Here is how I write blog entries using Emacs, and publish them to Blogger:

I write raw entries using the amazing Org-Mode for Emacs. I'll not take you to Org Mode kindergarten; there are dozens of good summaries and introductions out there. I keep an outline of blog entries - ideas, notes and in-progress drafts - and work on the just as I would any other project.

The Org export function can generate beautiful HTML – for example I can preformat code examples using font-lock-mode with simple tags. My .emacs defines a simple wrapper around org-export-as-html which only exports the body code without the header and footer:

(defun bloggerPublish()
"Render the region as HTML for publishing."
(interactive)
(org-export-as-html 0 nil nil nil t nil))

I highlight (using transient-mark-mode) the "body" of the post, invoke that function, and paste the result into a new Blogger post. Fill in the title, add some keywords, and I'm done. Blogger takes care of the CSS and if I cared about presentation enough I could modify the element classes that Org spits out.

Since I basically live in Emacs, I can get a heck of a lot more writing done by just tearing open a Remember template to jot down the idea for a post, intermingle my drafts/notes/etc. with the rest of my Real Work, and basically draft a post as quickly as an email or project notes or whatever.

Here's a public thanks to Carsten for such an awesome package!

13 August 2009

Migrating to RAID-1

Here's an old post, which hopefully nobody will ever need because everybody's installing RAID-1 or better on every system that ever gets built. Back in 2007, I migrated a RHEL system with a single hard drive to a software RAID-1 array -- without losing any data :-) Here's how to do it -- the instructions are specific to whatever version of RHEL that was way back then, but the technique will be useful on any reasonably-modern distro:



# edit /etc/fstab and /boot/grub/grub.conf to use /dev/sda instead of labels
e2label /dev/sda1 ''
^1^3
# run fdisk and change all partitions on /dev/sda to type fd
sfdisk -d /dev/sda | sfdisk /dev/sdb
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb3
mkfs.ext3 /dev/md0
^0^1
mkinitrd -v --preload=raid1 /root/initrd-$(uname -r).img $(uname -r)
mv /boot/initrd-$(uname -r).img{,.orig}
mv /root/initrd-$(uname -r).img /boot
reboot # This is the magic step! Don't skip it!
cat /proc/mdstat # Make sure that md0 and md1 are running
# reboot w/ rescue disk
modprobe raid1
for i in $(seq 0 9); do mknod /dev/md${i} b 9 $i; done
cd mnt
mkdir {sda1,sda3,md0,md1}
mount /{dev,mnt}/sda1
^1^3
mdadm -A /dev/md0 /dev/sdb1
mdadm -A /dev/md1 /dev/sdb3
mount /{dev,mnt}/md0
^0^1
# edit /mnt/sda1/grub/grub.conf to boot from /dev/md1
# edit /mnt/sda3/etc/fstab to mount /dev/md0 @ /boot & /dev/md1 @ /
cp -a /mnt/sda1/* /mnt/md0 # This is why we use the rescue disk; we probably
cp -a /mnt/sda3/* /mnt/md1 # don't want to try copying a running system
# The above will take a loooooong time.
umount sda1 sda3 md0 md1
reboot
cat /proc/mdstat
mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda3
watch -n 1 'cat /proc/mdstat' # The above will take a looooooong time.
grub --no-floppy
device (hd0) /dev/sda
root (hd0,0)
setup (hd0)
device (hd0) /dev/sdb
root (hd0,0)
setup (hd0)
quit
rm /boot/initrd*.orig
reboot
The procedure draws heavily on Warren Togami's document at http://togami.com/~warren/guides/remoteraidcrazies/. There are a few typos in his and I've made some enhancements, so use mine instead of his :-)

Custom Debian/Ubuntu Kernel Packages -- and Fast Compiling

This is no news, but I got sick of forgetting how to build an Ubuntu kernel package:
tar -xjf kernel*.bz2
# cd to new kernel tree
make-kpkg clean
cp /boot/config-$(uname -r) ./.config
make oldconfig
make-kpkg -initrd --revision= kernel_image kernel_headers modules_image
# sleep || eat || whatever
dpkg -i ../kernel-image*.deb

Good magic is that the -j option to make sets the number of concurrent jobs so as to take advantage of multiple CPUs. This number should be (numCPUs * 2 ) + 1. So before doing your make-kpkg, do this:
export CONCURRENCY_LEVEL=5  # a 2-CPU system

You can get really cute and put the following in your .bashrc:
if [ -a /proc/cpuinfo ]; then
export CONCURRENCY_LEVEL=$(($(grep -c processor /proc/cpuinfo) * 2 + 1))
export MAKEOPTS="-j${CONCURRENCY_LEVEL}"
fi

Maybe my .bashrc will be the subject of a future post.


Followers