BASH Hacks: Using 2 Comma Delimited Lists to Replicate Simple 2D Array

As we have stated, the data stuctures in BASH are extremely limited. BASH has just introduced a single dimensional array data structure, but it is tricky and hard to use. They have yet to implement a multi-dimensional array data structure, and therefore we are forced to do a lot of trickery in order to replicate the functionality of multi-dimensional arrays. A very simple type of 2 dimensional array that is highly useful for writing test scripts, is a 2 dimensional array with on dimension holding any type of data, and the second dimension relating a boolean value to the values in the first dimension of the array. Say for example we have a bunch of IPs that represent webservers in a cluster, and we're writing a script to test to see if the webservers are up or down. It would come in handy to know wether the last time through the loop the servers were up or down, so we might want an array like this:

192.168.1.1	192.168.1.2	192.168.1.3
1	1	0

Unfortunatly with BASH, this isn't possible. To replicate the simple two dimensional array of this type I have developed a little bit of code that makes extensive use of the Comma Delimited list code that I wrote about in a previous post: BASH Hacks: The Comma Delimited List. For this solution we implement 2 of these lists, the first list holds all of the variables that you are going to run a series of actions on. The second list will be populated by values from the first list that fail these actions or tests, and each item in this list will be equivilant to having a value of "0" in the boolean dimension of the simple 2D array, and conversely a lack of a value on this list is equivilant to a "1" in the boolean dimension. This is a template of the code that I use:

#!/bin/bash
###################################
## 2commas.sh
##
## A template for implementing 2 comma
## delimited lists to replicate the functionality
## of a simple 2D array
###################################

#######VARIABLE DEFINITIONS########
LIST=1,2,3,4,5

# We just initialize this variable here, it won't be used untill much later
DOWNLIST=''
###################################

# loop continuously
while [ 1 ]
do

# At the beginning of every cycle, tack a trailing comma to the end of the list
LOOPVAR=${LIST},

# Loop through the contents of the comma delimited list
while echo $LOOPVAR | grep \, &> /dev/null
do

# grab the first server name out of the comma-delimited list
LOOPTEMP=${LOOPVAR%%\,*}

# remove the first server name from the LOOPVAR comma-delimited list
LOOPVAR=${LOOPVAR#*\,}

if <your test>
then

## possibly some actions here
## possibly not

# if it's on the downlist, take it off and do some stuff
if echo $DOWNLIST | grep $LOOPTEMP &> /dev/null
then

## Some actions
## maybe no actions
## who knows?

# Initialize the NEWDOWNLIST variable,
# which is a temp var that will hold the downlist minus the now up server
NEWDOWNLIST=''

# This loop will take the now up server off the downlist
while echo $DOWNLIST | grep \, &> /dev/null
do

# grab each value in the downlist from the comma delimited list
DOWNLOOPTEMP=${DOWNLIST%%\,*}

# trim the value we just grabbed from the downlist
DOWNLIST=${DOWNLIST#*\,}

# This variable will be empty if the DOWNLOOPTEMP
# variable is different than the LOOPTERM variable
# if they are the same the variable will contain the value of DOWNLOOPTEMP
CONDITION=$(echo ${DOWNLOOPTEMP} | grep ${LOOPTEMP})

# If the variable is empty the value stays on the downlist
# and we add the server to the new list
if [ -z $CONDITION ]
then

# add the server to the new list
NEWDOWNLIST=${NEWDOWNLIST}${DOWNLOOPTEMP},

fi

done

# Set the downlist variable to the value of the New downlist variable
DOWNLIST=${NEWDOWNLIST}

fi

# If the test fails then
else

# This variable will be empty if the DOWNLIST variable
# doesn't contain the LOOPTEMP variable
# otherwise the value of the variable will be the value of the DOWNLIST variable
CONDITION=$(echo ${DOWNLIST} | grep ${LOOPTEMP})

# If the above variable is empty, the value needs to be added to the DOWNLIST
# and some actions performed, otherwise we already noted that it's down, you might
# still want to do some actions here though
if [ -z $CONDITION ]
then

# Add the value to the downlist
DOWNLIST=${DOWNLIST}${LOOPTEMP},

## maybe do some more actions here if you want

fi

fi

done

done

exit 0


For some reason Blogger doesn't respect any attempt on my part to use tabbing, so this code looks like crap, appologies for this. Unless i use the pre tags it won't work, so when i get an elegant solution for tabbing with blogger my script code is going to look pretty nasty. Appologies.

BASH Hacks: The Comma Delimited List

BASH is by no means a complete language, and for all but a very few things it's not the right one to go with. But for scripts that make intense use of external programs it's really the best, and so this will be the first article in a series of many that deals with simple BASH hacks to make your scripting life easier.

One of the biggest problems with BASH is that the data structures are fairly primative, and so writing a script that expands with your infrastructure is often times very hard. You end up having to do massive hacks to the script every time you add a new server, or customize the script per server etc. I've developed a bit of code that I use in any script that deals with potential areas for growth using a comma delimited list to perform a set of actions on each member of that list. This makes the code much more portable and transparent, and as your infrastructure grows or shrinks all you have to do is edit a variable definition at the beginning of one file, and the script is ready to go again. Here's the basic template:


#!/bin/bash
#################################
## commadelimited.sh
##
## A template for implementing a comma
## delimited list in BASH
#################################

######VARIABLE DEFINITIONS#######
LIST=1,2,3,4,5
#################################

# Add a trailing comma to the list variable
LOOPVAR=${LIST},

# Loop as long as there is a comma in the variable
while echo $LOOPVAR | grep \, &> /dev/null
do

# Grab one item out of the list
LOOPTEMP=${LOOPVAR%%\,*}

# Remove the item we just grabbed from the list,
# as well as the trailing comma
LOOPVAR=${LOOPVAR#*\,}

# some action with your variable
#
# echo $LOOPTEMP
#
# for example

done

exit 0


So what exactly does this do . . . well, here we go in detail. After defining the comma delimited list in a variable:


LIST=1,2,3,4,5


We have to add a comma to the end of this list, because we grep for a comma in the loop, and the loop would therefore terminate before processing the last item in the list, as there would no longer be a comma, because we trim the trailing comma from the list with the item itself:


LOOPVAR=${LIST},


Now we're ready to rock. If you havn't had the pleasure of using an if grep or while grep statement, then I would suggest you start to use it, this is one of the more usefull features of BASH:


while echo $LOOPVAR | grep \, &> /dev/null
do



This will loop for as long as the loopvar variable contains a comma in it.


LOOPTEMP=${LOOPVAR%%\,*}


This grabs the first item from what's left of the LOOPVAR list. The way this works is the % in the ${} indicates that we want to trim the string from the end. The fact that there are two of them (%%) tells it to trim greedy, and match as much as possible. The regular expression that follows the %% says to chop from the furthest comma to the right, followed by anything.


LOOPVAR=${LOOPVAR#*\,}


This statement is similar to the last one, but this one takes out the first value and the trailing comma and shortens the list by one. The # tells it to trim the string from the beginning, and only one # tells it to be non-greedy and grab everything up through the first comma from the left, as defined by the regular expression that follows.

So as we loop, the list gets shorter by one item per iteration, and eventually cycles down to an empty string, where the grep \, fails and the loop breaks.

The following script is an application of the comma delimited list idea that I use in my everyday work:


#!/bin/bash
################################
## sendadminmail.sh
##
## This script is called by monitoring scripts
## to send mail to all of the administrators
## of our company should something go
## wrong. Which given my skills, should
## never happen.
################################

#######VARIABLE DEFINITIONS########
#
# Comma Delimited List of Administrative E-mail addresses
ADMINLIST=test@test.com,test@test2.com,test@test3.com
#
###################################

# initialize the loop variable, the trailing comma is
# so that the last entry will be honored
LOOPVAR=${ADMINLIST},

# this loop loops through each individual entry on the list
while echo ${LOOPVAR} | grep \, &> /dev/null
do

# Grab the first e-mail address out of the comma-delimited list
LOOPTEMP=${LOOPVAR%%\,*}

# Remove the first e-mail address from the
# LOOPVAR comma-delimited list
LOOPVAR=${LOOPVAR#*\,}

# send the e-mail
echo "$1" | mail -s "$2" "$LOOPTEMP"

done

exit 0


A Hate Letter to MySQL

Normally I just throw up code snippets and cool stuff, but having to work with MySQL on a daily basis behooves me to throw up a rant to the MySQL developers here. I was strongly in the PostgreSQL camp, as that's what I learned on, and it had nice features that MySQL lacked . . . like referential data integrity for example. But with MySQL 5.0.x they announced that MySQL had matured and was ready to be a real SQL server with the purchase of the InnoDB storage engine.

So I switched to MySQL for 2 reasons: the first being that MySQL 5.0.x seemed to be up to snuff and ready to compete with the big boys (namely Oracle and PostgreSQL), and 2, unfortunatly it's called the LAMP stack and not the LAPP stack, and every web app out there has MySQL support, but only a few have support for PostgreSQL, and as my job forced me to deal with these programs, I was forced to migrate. So here I will bring my complaints against MySQL, and suggest directions for improvement.

1. Kill Off MyISAM

It would be a mercy killing. For fuck's sake, this thing doesn't adhere to much of anything in the SQL 95 or SQL 99 standard, so stop claiming that it's a storage engine. Despite the fact that InnoDB has nice features like referencial data integrity and Transaction handling, the majority of database designers out there (if we may even call them that) don't take the time to write InnoDB databases. Aparently that ENGINE=INNODB is just too much strain on the wrist to type, and in addition to this, the default storage engine in the my.cnf file is . . . you guessed it: MyISAM.

MyISAM is a disgrace to SQL, it doesn't support FOREIGN KEY constraints, which is, by the way, like the key feature of SQL, and it doesn't support transactions, which means if you write a web app with a complicated series of inserts, and the clown end user hits the stop button a half second into this, your data set is completely fucked. I realize that most web developers don't put transaction support in their code even when it's available, but let that be their mistake, don't make the lowly DBA . . . me . . . tell my hot shot developer that he can't put transaction support in because we're running MyISAM. You can of course specify a FOREIGN KEY in MyISAM, but as it notes in the manual, this does not enforce the constraint, it merely serves as a mental note to the database designer. ARE YOU KIDDING ME?! It's time to put this baby to bed . . . permanantly. I'll tell you what you do MySQL, you take the only good feature of this thing: full text indexing, and you fold it into InnoDB and turn out the lights.

2. Kill phpMyAdmin

If anyone intelligent is reading this, you can imagine my disgust when i walk into my first day on the job and see my first developer using phpMyAdmin to interact with the database. The idea behind this is pretty cool I'll admit, but it's this nasty little program that lead to the rise of the MyISAM storage engine, and we should put it to bed along with it's progeny.

Aside from the fact that this program is the single most notorious security hole since portmapper, and putting it on the front of your DB is like giving a stranger the keys to your car and hoping he doesn't steal it, this program sucks. If you know MySQL, it's about a hundred times faster to use the mysql CLI client program to build your database and query it. This thing is terribly slow and terribly insecure. I have no problem with the novice database designer using it for playing around, or for putting up a personal website, but I have personally seen this thing used all over the place in production level deployments. Luckily enough I killed it at my current job, but I can assure you that the University of Chicago uses it for it's mega-database, so if you want to change your grade . . . well, get inside the UofC network and find the phpMyAdmin folder, and do what you want.

This program is the number one reason why there are so many crappy databases around, and why they are mostly for MySQL. This thing allows a novice database designer to design a MySQL database, without knowing anything about SQL at all. If you want to design a database, pick up the PostgreSQL book by Bruce Momijian, or read the MySQL 5.0.x documentation from cover to cover . . . I've read both, and I assure you they are compelling reads, and you'll never make the clown mistakes that these phpMyAdmin designers make, and you'll code your database in 10% of the time.

3. MySQL Cluster Is Not, I repeat NOT Ready For Deployment

So stop claiming that it is. First of all the NDB storage engine is a big pile of trash, and they need to do a complete rewrite. NDB has several problems the first of which being that it is an in memory storage system. Yes, this does lead to faster query execution, but it comes at a costly price. Firstly, to scale this thing takes a lot of money, once your data set gets over about 5 gigs, which can happen very quickly indeed. Secondly, the system works by storing stuf in memory and periodically writing down to the hard drive, which means if you have a hard crash in between those two point your data is fucked, and have fun recovering from memory . . . I assure you that this is no picnic.

The next problem that comes from this are that limits on memory usage are hardcoded in the my.cnf file, in order to make changes to this, you are required to do a complete restart of the cluster . . . which means downtime, and avoiding downtime is the sole purpose you went with the cluster in the first place. And there are no utilities that easily track what kind of space you're using, so you'd better be a master of perl or python, because you have some nasty scripting ahead of you, to keep track of your data set . . . otherwise you'll max your memory usage and the database will refuse further updates to the dataset . . . oh yes it's fun.

In addition to this NDB seems to fail for a lot of the reasons that MyISAM fails, and more. NDB doesn't support foreign key constraints, nor does it support transaction handling, and in addtion to this it doesn't support multi-column unique indexes, or multiple auto_increments on a table. NDB also requires each table to have a primary key, which isn't so bad, but it's rediculous that this is required by the engine itself. NDB does not support full text indexing either, so as the bastard child of MyISAM it even lacks the one feature that is good about MyISAM. Please for the love of God do a rewrite, make it a non-memory based storage engine with transaction handling, and referencial data integrity and multi-column unique indices. I'm so tired of reading all these articles basking in the glow of MySQL cluster, these articles convinced me to deploy it, and really, for all it's faults, this is not nearly as good a solution as MySQL circular replication, which is sad.

4. InnoDB's Future

The reason for switching to MySQL in the first place is InnoDB, but this baby has a long way to go before i'm pleased with it. Firstly . . . the defining of a foreign key constraint is so unbelievably annoying it's absurd. The only way to define a foreign key is to do it at the end of a table definition with FOREIGN KEY (id) REFERENCES table(id). This kills me, allow us to do it the way PostgreSQL and Oracle and even MSSQL does it . . . id INTEGER NOT NULL references table(id). See how nice that is . . . see how you just do it when you define the column, so that you're not forced to look back and find out what the column name was and write a bunch of rediculous nonsense at the end. All of us DBA's and architects would give you a big kiss if you would just do this bit of syntatic tom-foolery for us.

Fold in Full Text indexing and make InnoDB the default storage engine in the my.cnf file. So what . .. an army of terrible database designers will have to learn InnoDB, oh my god that would be so terrible. Syntactically the basic functionality is no different, it just adds some sweet stuff so do it, for the love of code. Make anyone who wants to write MyISAM databases strain their wrists with the extra ENGINE=MyISAM, they deserve carpul tunnel if they're going to write databases with that nonsense storage engine anyway.

Finally, beef up PLmySQL, it sucks. For those of you unfamiliar with PLSQL it means Procedural Language SQL, and it's a language for triggers and such. Triggers are widely considered to be bad practice . . . but at the same time, they are very useful for administrative data collection and for lots of other things too. PostgreSQL has a robust PLpgSQL that works amazingly well and makes coding triggers really easy. PLmySQL sucks and could use a lot of beefing up. Specifically you need a much better system of passing data into the trigger, as the system right now is almost indeciferable and a lot of data just isnt' available to the trigger or procedure.

If you do all of this, you might just have me on your side of the debate because I want to be there . . . as opposed to having to be there because of your marketshare.

BASH Script Background Music

So I was sitting here at work, writting a massive script to handle the increasing complexity of our subversion repository for my co-worker Sean (of seancode.blogspot.com fame), and the script did so much I didn't know what to call it. For the sake of naming it, I just named it the most rediculously epic sounding thing I could think of . . . ultramega.sh, which, as it happens, is also the name of a PowerMan 5000 song. Sean and I went out for a smoke and I told him that the name of his script was ultramega.sh, after the PowerMan 5000 song, and I joked that i should have my script play the song in the background while it was chugging along . . . as this bad mother of a script takes a long time to do everything that it does. Sean was like . . . AWESOME! Have it start with the script and stop with the script . . . but we need something more epic than UltraMega, have it play "The Final Countdown" by Europe on repeat. Sweet god I thought . . . that is awesome, let's do it! So I did, with an assist from sean, who figured out how to put VLC into quiet mode, crank the volume and have it repeat. So if you've ever wanted epic background music for a script, or wanted to fuck with one of your programmers . . . here's how we pulled it off:

#!/bin/bash
##########################################################
## FinalCountdown.sh
##
## A template for adding background music to your script
##########################################################

#########VLC VARIABLES###########
VLCFILE=~/finalcountdown.mp3
VLCCOMMAND="vlc ${VLCFILE} --volume 500 -q -L"
GREPCOMMAND=$(echo "$VLCCOMMAND" | sed 's/\ /\\ /g')
##################################

# Start VLC
$VLCCOMMAND &

# PUT THE CONTENTS OF YOUR SCRIPT HERE
#
# do some crap
#
# some more crap
#
# finally back to VLC

# stop VLC
GREPRESULT=$(ps aux | grep "$GREPCOMMAND")
GREPRESULT=${GREPRESULT#*\ }
GREPRESULT=${GREPRESULT%%\ *}
kill ${GREPRESULT}

# end script
exit 0

In case you havn't figured it out yet . . . this is for linux, but it should work equally well for all *nix's. I'll go over the script in detail for those of you not fluent in the language of the gods . . . BASH.

First - the variables:

VLCFILE=~/finalcountdown.mp3

This is the location of the song you want to play with VLC.

VLCCCOMMAND="vlc $VLCFILE --volume 500 -q -L"

This is the command we want to execute . . . this is here in case we want to use a non-vlc audio program, it makes the code transparent enough such that you just enter in the command of whatever audio program you want and it will work. Specifically for VLC the volume directive tells VLC to put volume to 500%, which is the max VLC volume, the -q tells VLC to not output much to the terminal, and the -L loops the audio output, so that if our script takes longer than the song, we're not left with an awkward silence.

GREPCOMMAND=$(echo "$VLCCOMMAND" | sed 's/\ /\\ /g')

This variable is less obvious than the first two, but what it does is it optimizes the command string for a grep by replacing all of the spaces with \spaces, so that it works regular expression-wise.

We then simply start VLC in the background by executing the VLCCOMMAND variable followed by the trailing &.

When we want to kill VLC we use ps aux with our optimized GREPCOMMAND variable to grab the one process that we're looking for . . . afterall you don't want to issue killall vlc and kill all the other VLC processes running too . . . that would suck:

GREPRESULT=$(ps aux | grep "$GREPCOMMAND")

We then need to trim all the extra stuff off the result of this to get just the PID . .. we do that with the following two commands:


GREPRESULT=${GREPRESULT#*\ }
GREPRESULT=${GREPRESULT%%\ *}


Then kill the process with the PID that we just grabbed:


kill ${GREPRESULT}


Pretty friggin' sweet huh? We think so . . . for a real laugh, put this in a script on a machine in your data center . . . and drive those techs crazy with the seemingly random Europe rips coming from your boxes at all hours.

SSH, HTTP And SAMBA Tinkering With Parrallels

So I know what you're thinking . . . you're thinking I have a new intel mac, and I bought a copy of Parallels (offsite) and installed Ubuntu 6.06 Dapper Drake into a virtual machine, but now that I've combined these two majestic operating systems how do I go about sharing files between the two?

Well I'll tell you how. I was hoping that parallels was using something like chroot to partition the drive, mount it with the requested file system and then virtualize the OS in a jail. This is, rather unfortunately not the case. What parallels actually does is creates an 8.4 GB file in the /users/<username>/Library/Parallels/<name of virtual machine>/ folder named <name of virtual machine>.hdd. Crazy.

So there is no way you're going to copy and paste files in and out of the jail locally, not even with the terminal and the old standard cp command. So how do I share or access files in between Mac OS X and Ubuntu? Well there are three ways depending on what you want to share or access.

1. SSH

If you actually want to move files from Ubuntu to OS X, or from OS X to Ubuntu you'll need to do it through SSH's SFTP program. This is actually relatively easy to set up, though my set up will require that you have a router (I think you can get around this by editing the hosts files in Ubuntu and OS X to resolve to the proper IP without any routing, but I havn't tested that).

First you'll need to bring down the active network device. If you are connected to your network via a wired ethernet connection, your primary connection device is most likely eth0, if connected by your airport card it is most likely ath0 (though possibly eth1, and definitely eth1 if you're still running Breezy Badger).

To bring down the wired connection:


shell~$ sudo ifdown eth0


To bring down the wireless connection:


shell~$ sudo ifdown ath0


Then We're going to set Ubuntu up with a static IP. Edit the /etc/network/interfaces file:


shell~$ sudo nano /etc/network/interfaces


For a wired connection, find the section for eth0, should look something like this:


auto eth0
iface eth0 inet dhcp


We need to modify this to give it a static IP:


auto eth0
iface eth0 inet static
# this defines the interface eth0 to have a static IP
address 192.168.0.150
# this defines the static IP of the interface
gateway 192.168.0.1
# this defines the default gateway (its the IP of your router)
netmask 255.255.255.0
# this is the subnet mask of the device (probably 255.255.255.0)
# if your setup is so advanced that it isn't 255.255.255.0 you
# then you probably know what this value is anyway
broadcast 192.168.0.255
# if your netmask is 255.255.255.0, your broadcast is the first
# 3 values of the IP address of the router followed by
# 255, X.X.X.255 if its not this, see the above netmask comment



For a wireless interface you'll need to find your wireless section, which should look something like this:


auto ath0
iface ath0 inet dhcp


And change it to have a static IP:


auto ath0
iface ath0 inet static
# This defines the interface ath0 to have a static IP
address 192.168.0.50
# This defines the static IP of the interface ath0
gateway 192.168.0.1
# This defines the default gateway of ath0 (Router IP)
netmask 255.255.255.0
# Subnet mask of ath0
broadcast 192.168.0.255
# Broadcast address of ath0
wireless-essid BEER
# Name of Wireless Network Connected to
wireless-mode managed
# mode of wireless network connected to
wireless-key H3X1SFUN
# WEP-key for wireless Network connection (or ASCII key)


Then bring the connection back up with the new configuration.

For a wired connection:


shell~$ sudo ifup eth0


For a wireless connection:


shell~$ sudo ifup ath0


To see if your static IP is working properly you'll need to perform a couple of tests. First check to make sure the proper device is representing the ip.

For a wired connection:


shell~$ sudo ifconfig eth0


For a wireless connection:


shell~$ sudo ifconfig ath0


A working device will output the values that you set in the interfaces file for each of the address, netmask and broadcast fields. Next you'll want to ping the router to make sure you're connected to the LAN:


shell~$ ping <router ip>


Then ping a site on the internet to make sure you have proper WAN access (assuming you have WAN access at all):


shell~$ ping www.google.com


Awesome, now we have a static ip on our ubuntu OS running inside of the parallels. To SSH to the ubuntu OS, you'll next have to install SSH on ubuntu:


shell~$ sudo aptitude install ssh


After the installation, make sure the ssh daemon is up and running:


shell~$ ps aux | grep sshd


At this point you should be able to use SFTP from Mac OS X to Ubuntu, in order to put and get files. To make sure, switch over to your OS X partition and open a terminal:


shell~$ sftp <ubuntu-ip>
sftp-> get testfile.sh
sftp-> put testfile.sh
sftp-> exit


To be thoughrough you should also start up the ssh daemon on OS X. To do this go to System Preferences -> Sharing and turn on personal file sharing. To test this switch back to your Ubuntu OS and run the same commands above only using the OS X ip instead of the ubuntu ip. If you don' know the OS X ip, just switch back to OS X and use ifconfig to get it:


shell~$ ifconfig


That about wraps it up for SSH.

2. HTTP Sharing

For some stuff SSH just doesn't make sense. For example if for some reason you want to listen to your music, or watch your movies from the OS X area in the Ubuntu area, or vice versa, you just don't want to have to copy those large files from one OS to the other, especially with the movies. So for this purpose we're going to set up HTTP servers to stream media from one OS to the other. Incidentally, as a cool side-effect of this you'll be able to stream this to any machine on your LAN, virtual or otherwise. Though this is because anyone on the LAN will be able to stream or download any file that you share via HTTP, and therefore if you want to securely share files, make sure you do it with sftp, not http.

Before we continue, I should point out that I am assuming you are on a LAN for this, and understand that if you stream copyrighted content (even legally aquired) over the WAN you are in violation of several laws, however bogus they may be.

Onward . . . you will most likely be streaming from OS X into Ubuntu, so we'll start there. To turn on your apache server just go to System Preferences -> Sharing and turn on personal web sharing. With one click of a button apache is up and running (though it apache 1.3 unfortunately) and they even poke a hole in the firewall on port 80 and 443 for you as well. Lets all just take a minute here to thank the folks at apple for being so friggin' awesome . . .

The document root for the apache server is: /Library/WebServer/Documents/ and you'll need to put all of your files into this folder in order to serve them. But of course we don't want to take up unnecissary space so just soft link the files or folders you want into the document root.

For example, my movies are in /Volumes/FATMATT/Movies, so . . .


shell~$ ln -s /Volumes/FATMATT/Movies /Library/WebServer/Documents/


Make sure the permissions are ok for the www user to read the files that you want to stream:


shell~$ ls -l /Library/WebServer/Documents/Movies/
shell~$ chmod -R 755 /Library/WebServer/Documents/Movies


Now we need to switch to Ubuntu and use a web browser to browse to the folder that we just set up, found at <ip of OS X>/Movies. If you want to download the files shared over HTTP, you can just browse to the folder and click on the file to download and save it. If you want to stream the files however (if they are media files) you'll need to download and install VLC. To install VLC in Ubuntu you'll first need to edit the /etc/apt/sources.list file:


shell~$ sudo nano /etc/apt/sources.list


And uncomment all of the universal repositories. Then perform and apt update, and install the program:


shell~$ sudo aptitude update
shell~$ sudo aptitude install vlc


To stream with VLC, browse to the shared folder and right click on the file, copy the link location. Then open VLC, go to open file, and paste the link location into the MRL text bar. And now you stream files to your Ubuntu OS wihtout having to take up vital disk space on that 8.4 GB file system.

To share files via HTTP from Ubuntu to OS X we'll need to set up an apache server on the Ubuntu OS. To do this install apache2:


shell~$ sudo aptitude install apache2-mpm-prefork


This will take a little while, but while it goes just sit back and enjoy the glory of apt, and take a moment to thank the developers at debian for allowing us to never have to hand-resolve dependancies ever again. Once this has finished apache should be running, but make sure:


shell~$ ps aux | grep apache2


The document root for apache in ubuntu is /var/www/, so linkng files in will look like this:


shell~$ ln -s /target/directory/ /var/www/


Once you've done that switch to OS X and browse to the folder: <ip address of ubuntu OS>/<folder name>. You can again download the files by clicking on them, or stream them in VLC using the same technique discussed above. To get VLC for mac go to their website (offsite) and download it.

3. SAMBA

The third option will allow you to read and write files on a samba share, without having to move the file from one to the other. This option will also allow you to copy files in and out of your samba shares on either OS and you'll be able to move files in between the two using this.

Most likely you'll be accessing the files on OS X with Ubuntu, so first we'll go through how to mount a samba share in Ubuntu. First you'll need to initialize the Samba server on OS X. You do this by going to System Preferences -> Sharing, and then you turn on Windows File Sharing. Doing this will prompt you for a user name and password, and OS X will conveniently let you know that the way Samba stores your password is less secure than the way the OS X stores your password, but it is still sufficiently secure for a LAN. Once you've done this your home directory in OS X will be up on a network Samba share, you can verify this in the finder, in the network section.

To mount the share in Dapper Drake go to Places -> Network Servers and then click on the share that has the name of your OS X box. My box is called MINIMATT, so I go to Places -> Network Servers -> MINIMATT to access my Samba share. If you need to configure additional samba shares on OS X, please refer to my previous post Setting Up External Hard Drive Samba Shares On OS X.

There is no real need to configure a samba server on your Ubuntu machine, as you can copy and paste what you need into and out of Ubuntu using the OS X samba server share. However to be thorough I should probably put this information up here as well. Unfortunately I don't have the desire to do this at the moment, so at some point in the future I'll put up a post covering how to set up a samba server in Ubuntu.
Enjoy!

Setting up External Hard Drive Samba Shares on OS X

I have a large USB 2.0 external hard drive (300GB) connected to my brand spanking new mac mini, loaded with files that I wanted to be able to access on my Ubuntu box, which is all the way on the other side of the room. Initially I was just unmounting the volume, unplugging it and carrying it all the way accross the room to my Ubuntu machine. Well after doing this once, my arms got tired, and I thought to myself there must be a better way.

My initial thought on this was to create an alias in my home directory, and then open it up for sharing. So to do this I created an alias of the drive in my home directory (right-click on the folder -> create alias of "FATMATT" -> drag the alias into my home directory) and then allow my home directory to be shared with samba (system preferences -> sharing -> windows sharing). This all seemed to work fine, except that linux (and windows for that matter) doesn't know what the alias is.

Well I have a solution for this I thought, I'll just soft link the directory into my home directory, certainly Ubuntu can understand that. So I opened up my trusty terminal window and went to work:


ln -s /Volumes/FATMATT /Users/matt


I go back to my Ubuntu box and it understands the soft link, and I can go into the folder and browse the contents. When I try to open anything in the samba share however, it won't do it. Well it turns out that since the actual files are ouside of the samba share, and also because OS X prevents sharing USB drives over samba shares, you're unable to open the files despite the fact that you can see what files are on the volume. Though I would recommend using soft links instead of aliases whenever possible, as through this process I discovered that a soft link is actually much faster in file browsing than an alias is (especially when using front row).

So I'm left with my final option, the one I didn't want to go with: configure an extra samba share for the USB drive. As OS X is nice enough to come with samba installed and configured for the home directories, I didn't have to install a samba server on the machine (a task that is quite daunting in Linux), all I had to do was reconfigure samba a bit.

In OS X samba's primary configuration is stored in the /etc folder as the file /etc/smb.conf, to add extra samba shares you need to configure this file by hand, so I went once again to my trusty terminal and opened the file with sudo and nano (that's right I use nano . . . just leave it), backing the file up first of course:


sudo cp /etc/smb.conf /etc/smb.conf.bak
sudo nano /etc/smb.conf


Samba's configuration is quite simple and easy to understand, it is comprised of sections denoted by starting tags of the form [start tag]. To define a share you just think of what you want the share to be called and then create a section with that name, for example my drive is called "FATTMATT" so I create the following section with the tag [FATMATT]:


[FATTMATT]
comment = FATMATT
path = /Volumes/FATMATT/
browsable = yes
read only = yes


The configuration options in the section tell samba the following:


comment = FATMATT


This option gives samba the name of the share as it will be displayed to the samba client.


browsable = yes


This value allows the samba client to browse the contents of the folder.


read only = yes


This disallows the ability to write to the share from the client, to allow the share to be written to from a client merely exchange that line with the following line:


read only = no


Once you've added your section you need to make sure of a few things. Check the ownership and permissions of the USB drive to make sure that the user you will be accessing the share as has read permission to the drive (for read only) or read/write permission on the drive (for not read only)


ls -l /Volumes/


Then finally you'll have to reload samba with the new configuration file


sudo killall -HUP smbd


Enjoy!

Firefox CSS Work Around

We all love firefox, right? If your answer to that question is anything other than yes, then please let the door hit you in the ass on your way offsite. Unfortunately I discovered a CSS bug in firefox 1.0, and 1.5, in both the Linux and Mac versions (I assume it is also a bug in Windows firefox, but havn't had the software or desire to test it). Basically the problem is that sometimes (havn't been able to work out the exact circumstances of the bug) if you create a div with a width of 100%, the div itself actually is 100% wide, but all the attributes of the div only span as far as the content of the div. For example, lets say I created a title bar for a page like so:


<div style="float:left; background-color:#FFFFFF; border-bottom:2px solid #EEEEEE; width:100%">Hello World</div>


The background color and bottom border would only be as wide as the text "Hello World" in the div. Kind of a pain. Well as a simple work around to this problem you can add a span or a div with a blank space in it floating on the opposite side of the div. So for example, the above div has float left, so to rectify the problem we could do this:


<div style="float:left; background-color:#FFFFFF; border-bottom:2px solid #EEEEEE; width:100%">Hello World<span style="float:right;"> </span></div>

<!-- or -->

<div style="float:left; background-color:#FFFFFF; border-bottom:2px solid #EEEEEE; width:100%">Hello World<div style="float:right;"> </div></div>


This CSS bug doesn't happen with IE or Safari - I know, it doesn't make sense, a CSS error in firefox that doesn't happen in IE, but its true, I swear - and hopefully the crack firefox crew will get on fixing it and you won't have to use this fix sometime in the very near future.

Welcome

Welcome to the Code and the Fury, the blog of irrefutable code and philosophy god, Matthew Story. I'll try to keep this place updated as much as possible, as I'm sure I'll have plenty of fans and followers eagerly awaiting new articles with each passing day. In case you didn't read the "about" section, in which case you're totally daft, I'll use this first post to give you all a brief idea of what this blog is all about.

My two principle interests are web programming and philosophy. I am currently employed as a web-programmer and network administrator in Chicago. I graduated from the University of Chicago in June of 2006 with a degree in Philosophy. I know what you're thinking . . . biggest nerd ever . . . well yes, you're right.

So for the Code part of this blog, you'll find nifty web-app hacks up here, and also the occasional networking hack or awesome bash script. On the administrative side of things you should probably know that if you're looking for windows help, you won't find it here. I worked for a couple months as an OS X systems administrator, and in my current position I work on Debian GNU-Linux. On my LAN at home I am currently running a combination of Ubuntu Linux and OS X, and I love playing with them. So up here you'll get linux and OS X hacks, and maybe if i get around to installing it again, we'll get to some FreeBSD hacks later on.

For the Philosophy part of this blog I will be writing predominantly about modern continental philosophy, mostly Heidegger, but also Husserl, Camus, Sartre. Every once in a while I might reach back and pull out some german ideology, but we'll see where that goes.

As an added treat, I will occasionally combine the two parts of the blog into some sort of . . . superblog. I'm a big proponant of Open Source Software, and I have quite a bit to say on the subject. Because of this also you're free to use all the code snippets that appear on this site under the rules of the LGPL v 2.1 (offsite).

Enjoy!