Searching Exercises

For the following exercises, try using the files in /opt/corpora/nanc/wsj/1994 for your searches.

This means if you are connected to /opt/corpora/nanc/wsj/1994, the command for searching for lines containing the string 'find' is:

gunzip -c * | egrep 'find'

Where word lists are called for, it is useful to be able to turn a file and into one-word per line format. This can be done with the command:

cat file | tr -sc "[:alpha:]" "[\n*]"

This prints the file on standard out. To redirect it to a file, do:

cat file | tr -sc "[:alpha:]" "[\n*]" > outfile

To produce a vocabulary list from a file in one-word per line format, sort it (into alphabetic order) and then remove adjoining duplicate lines using sort and uniq:

sort one-word-per-line-file | uniq > vocab_list

Summarizing all of the above points. Here is one line that will produce an alphabetized vocabulary file in your home directory for a compressed Wall Street Journal file:

gunzip -c ws941116.gz | tr -sc "[:alpha:]" "[\n*]" | sort | uniq > ~/wsj94116_vocab.txt

Problem 1

A. Try to write a single regular expression pattern that matches all forms of the verb 'find' (meaning "to successfully conclude a search").

• find
• finds
• found
• finding

B. What other words match this expression? Here are some: Other words

C. Modify the pattern in answer to exercise 1a, so as to rule out matching 'founder' and 'foundation':

New pattern

D.

The revised pattern still gives you a lot of output to check by eye. Use a second regular expression search of that output to make sure there are no hits on 'foundation.' Use the same idea to check that the answer to part A DOES have hits on 'foundation'.

New commands

E.

Here's another way to revise the pattern:

Alternative

Problem 2

A. Try to write a single regular expression to find words that start with lower-case 's' followed by two lower-case consonants in the file 'ws940701':

Here's a nice answer: Egrep command

B. Here's an automatically generated list of the the words that come up: Word list

Here's the unix-ese I used to generate this list: Complex Unix command

Try out the commands between the vertical bars in sequence so you understand what each step of this sequence does. In each of the following sets of directions "bulba%" stands for the Linux prompt (which not be quite the same for all of you). What follows are commands you should actually type to the prompt, up until you get to the symbol "#", which is a standard Unix scripting language comment character. What follows "#" is a comment on or explanation of what you just did.

1. bulba% cd /home/ssrl/compling; cat reverse_alphabet # notice there 2 of each letter
2. bulba% sort reverse_alphabet # Look what happens
3. cd $HOME; cp /home/ssrl/compling/reverse_alphabet . # copy the file "reverse_alphabet" to your home directory
4. bulba% sort reverse_alphabet > alphabet # store the sorted file in a file
5. bulba% cat alphabet # inspect the file you just created
6. bulba% uniq alphabet # what changes?
7. bulba% uniq alphabet > uniqified_alphabet # Store the uniqified file
8. bulba% wc alphabet uniqified_alphabet # The first column is lines, the second words, the third characters
9. bulba% sort reverse_alphabet | uniq > new_unqified_alphabet # Doing it all in one step

Problem 3

A

Using the command line in Complex Unix command from problem 2 as a model, construct a list of words beginning with the prefix "exo" found in /opt/corpora/nanc/wsj/1994/ws940701.

Here's a good answer: Another complex Unix command

Here's an automatically generated list of the the words that come up: Word list

B

Now construct a list of words beginning with the prefix "exo" found in the Wall Street Journal in 1994.

Here's the revised command: Revised command

Here's the word list constructed from that command:

Revised word list

Do you think all of these words really use the prefix 'exo' or do some of them just start with the letters 'exo'? HINT: Think about the meanings? Do all of them have to do with the outside of something or getting to the outside of something?

C

Now look for all instances of words beginning with the prefix "intra" found in the Wall Street Journal in 1994.

Here's a good answer: Another complex Unix command

Here's the automatically generated list: Word list

Python

1. Running and modifying the 'hello world' program
   • Here is "hello world" as a Python program.
   • Make a python directory in your home directory
     bulba% cd $HOME; mkdir python
   • Connect to it.
     bulba% cd python
   • Copy the hello world program and change its permissions to be executable and run it
     bulba% cp /home/ssrl/compling/hello_world.py . bulba% chmod ugo+x hello_world.py bulba% hello_world.py hello world bulba%
     Notice that the "\n" is not printed. This just added a carriage return. We will now practice editing the file to take away the carriage return.
   • Call up your favorite text editor on the file $HOME/hello_world.py and change it to be:

     #!/usr/bin/python
     
     print 'hello world'
     

     Now save the change, rerun the program and verify that the change has taken effect.
   • The first line of the program file is not Python.

     #!/usr/bin/python
     

     It is an instruction to Unix about what to do with this file, in particular, what program to call to execute it. A full path name is given and is advisable so that the program will run anywhere.

     The second line is a Python instruction:

     print 'hello world\n'
     

2. Python is interpretive language
   • Start up Python while connected to your python directory (where you have placed your edited copy of "hello_world.py"):

     bulba% ls
     hello_world.py
     bulba% python
     

   • Import every function, method and variable from the sys module

     >>> from sys import *
     

     That was just for practice.
   • Now import hello_world

     >>> import hello_world
     hello world
     

     Notice the program actually RUNS when you import it. That's because that's how you wrote it. The instruction in the file is just executed when loaded.
   • Type Control-D (Hold down the control key while pressing "d") to exit Python.
   • Now modify the file as follows:

     #!/usr/bin/python
     
     def hello ():
         print 'hello world'
     
     if __name__=='__main__':
         hello()
     

     Indent as above! Indentation is important! The following will cause an error:

     #!/usr/bin/python
     
     def hello ():
     print 'hello world'
     

     The exact number of spaces indented is not significant, but the fact that there IS indentation matters. This is why it's useful to edit Python with an editor that knows Python.

     Here are two:

     • Idle
       To start:

       bulba% idle
       

       Click File>Open and select 'hello_world.py'. Then in the new window that opens with the file, click Run>Run Module.

       In the hello_world window, if you type "def hello():" and hit carriage return, the proper indentation will appear.

       If you hit a carriage return after the print line, indentation will also appear. But now you don't want it. Just hit the Backspace key.

     • Emacs
       To start:

       bulba% emacs hello_world.py
       

       Click Python>Start interpreter. Then click on the hello_world window (you'll have 2) and click Python>Execute Buffer.
   • Having made the editing change RELOAD your module into Python (through the menu).

     Loading the original file by menu in idle should look like this:

     >>> ================================ RESTART ================================
     >>> 
     Hello world!
     

     The program executes. Loading the revised file by menu should look like this in idle:

     >>> ================================ RESTART ================================
     >>> 
     

     The program does not execute.

     We need to execute the function to make it execute:

     >>> hello()
     Hello world!
     

   • Note doing an import has a different effect than loading by menu, Kill your old Python shell in idle and do Run>Python shell. Then

     >>> import hello_world
     

     Then:

     >>> hello()
     
     Traceback (most recent call last):
       File "", line 1, in -toplevel-
         hello()
     NameError: name 'hello' is not defined
     

     In Python importing a module defines everything in the name space for that module. So the function "hello" is not defined. The function "hello_world.hello" is defined:

     >>> hello_world.hello()
     Hello world!
     

     You can import things into the top level name space if you desire. The command for that is:

     >>> from hello_world import *
     >>> hello()
     Hello world!