Shell-3: loops, scripts

1. Data collection generates 1000s of data files and each one must be processed separately.
2. Project generates 1000s of pdf files and we are required to look through a large random sample, one-by-one.
3. Vacation photos are saved in super high resolution and we need to convert each one to something smaller for Instagram.

When I went on vacation, I saved photos that were way too huge. There is a handy free program named "convert" that can rescale images however you like, but it wants one file at a time. The syntax for that is

$ convert file.jpg -resize 1400x1050 -quality 85 file.new.jpg

To process a lot of files, it is necessary to take each file, create a new name, and then run convert on each one.

I want output that takes the old name file.jpg and creates the new name file-1400x1050.jpg. This file name part of the assignment creates a little detour on splitting the file name in two pieces. There is a handy program called basename that I use often.

We can write this out in 2 ways

1. The "jam it all into 1 line method" uses semi-colon (;) for puncutation.

   $ for fn in *.jpg; do base=`basename $fn .jpg`; convert $fn -resize 1400x1050 -quality 85 $base-1400x1050.jpg; done
   

2. The patient shell approach. BASH knows you are not

done typing when you hit return, it waits (signalling ">" to you)

$ for fn in *.jpg
> do 
>   base=`basename $fn .jpg` 
>   convert $fn -resize 1400x1050 -quality 85 $base-1400x1050.jpg
> done

If you type that in, the shell will wait until you hit enter after done to begin.

Tricky bits.

1. The first "fn" has no dollar sign, but when it is accessed, "$fn" it does! Among programming languages, that is unusual. The variable is created with no dollar sign, but to access it we require the dollar sign.
2. The "do" statement applies to all of the lines that follow, until done. The one line method makes it appear as though it only applies to the first.

Recall in the north-pacific-gyre folder, there are several data files to be processed.

ls N*.txt

NENE01729A.txt
NENE01729B.txt
NENE01736A.txt
NENE01751A.txt
NENE01751B.txt
NENE01812A.txt
NENE01843A.txt
NENE01843B.txt
NENE01971Z.txt
NENE01978A.txt
NENE01978B.txt
NENE02018B.txt
NENE02040A.txt
NENE02040B.txt
NENE02040Z.txt
NENE02043A.txt
NENE02043B.txt

Our Assignment: Run each txt files through a data processing script.

Recall the problems unearthed in data inspection

1. One file too small
2. The Z-ending files are not useful

The shell script "goostats" is used to process each file and create a new data output file.

• I strongly prefer to keep the output in a separate output directory.
• And I would almost certainly put the output directory outside the current working directory.

mkdir ../output

Why do I like that? No danger of disrupting original data.

In fact, I'd make the original data read only like so

chmod -w N*.txt

Our assignment is to run each file through goostats.

1. Test commands that work in the command line
2. Put those lines into a file
3. Work out the command line arguments problem
4. Consider adding "hash-bang" in line 1

The shell command line arguments can be handled in many ways. The simple way simply used arguments by their position in the command line. This is not the best way, since there is no error checking on the inputs. However, it is commonly used in small projects.

Using goostats on a single file, novice might be tempted to run

goostats infile.txt outfile.txt

However, we hit problem(s).

1. goostats.sh is not executable, and
2. goostats.sh is not in the path

So we can't run it

• My natural inclination would be to make goostats executable

chmod +x goostats.sh

• and then run with the "./" trick

./goostats.sh infile.txt outfile.txt

• however, for security reasons, the system administrators have forbidden us from creating new executable files.
• there is a work-around, however, because even though "goostats" is not executable, we are allowed to run the program "bash" and it can open goostats and run it for us.

• `bash` or `sh` or other shells

In the simple cases we are using here, it is virtually certain the program would run in a simpler shell than BASH, such as the much more generic `sh` or the "lightweight" derivatives like `zsh`

sh goostats.sh

I mention this because on some large multi user systems, the `bash` luxury might not be available, while `sh` almost certainly will be.

• If we could make this executable, then running it would not guarantee the correct shell is used to run it. We want to make sure that `bash` is used, for example.

  We showed above it is possible to launch a program by passing it to "bash". That's not unheard of, but it would be somewhat unusual to do this when the work is finished.

• #! is pronounced "hash bang".

  Often people would insert a line 1 message like this

#!/bin/bash

• Make the script executable, run

chmod +x dostats.sh

• If we could make that executable, then we could launch the script

./handlegoo.sh

• But if we are not allowed to do that the usual Unix say, we run like this

bash handlegoo.sh

• The "-x" flag activates a debug mode in bash

bash -x handlegoo.sh