Shell Programming and Scripting

Hello! I think this should be an easy solution. I have a large file with many fields of data. The first field has a unique identifier (a subject number) for every record for a chunk of data. Something like this:
There were ten experimental conditions (ec), but the ec is identified by only one field in one record for each subject. I already ran a script to create a list of all subject numbers and their corresponding ec. For simplicity's sake (or so I think) I divided up that list by ec, so i now have numerous lists containing all of the subject numbers for each ec. I hope you can picture this.

Lets call my main data file "all.inq" and my lists "[0-9].lst"

I want to grep all the records in all.inq that begin with the numbers listed in 0.lst and put them in a file 0.inq, and then repeat the process for all other lists. This is probably a one liner, but I just don't know much about grep.

Thanks!
Chris

I thought of a way to simplify the question and make it general.

grep syntax is:
Can you have grep read the pattern from a file, and go line by line through that file and just append all standard output to the intended file?

for name in list
do
egrep ^$name pattern > $name.out

done

to dump all in the same file use >> $name.out.

list would be the numbers.

I dont understand how this would work. It looks to me like list would be the file name of my list file, and name would just read in each record in the file (??? I dont know if that is the right interpretation). Then, the grep command looks for lines in the data file that start with the subject numbers in the list file. However, If I print to >> $name.out, then I am going to have a new file for every subject... which would not be good. I want to group by experimental condition.

If I am interpreting this correctly, I would be able to to this and just define the output files name to explicly be the ec number, if I read from my individual lists that are already grouped by condition.

Is there a way to read in column $1 from the list file (which is the ec) and have that govern the name of the .out, and then read the second column of the list file (the subject number) into the grep search pattern pattern?

So the grep line would be something like this, i think:

Code:

grep '^$2' all.inq >> $1.out

I just dont know how to read in the master list file so that it would work.

is 'ec' the first field in all.inq?

Pls post a sample all.inq, a sample 1.lst and a desired output of 1.inq

And indicate a 'common' field between all.ina and 1.lst files

Alright, this is a sample from all.inq:
The file is enormous so this is only a small chunk of block 6 from subject# 570613667. If you look in my first post, that simplified version should be enough to work with. The experiment condition is only in one field of one record at the top of each subject (that line also begins with the subject#, which is a good thing for me). I already handled this part of the process with awk. I found a unique quality of a field in the first line, and printed the ec and sn fields ($6=ec; $1=sn, if you care).

So, my master list looks like this:
There are no headers. $1=ec; $2=sn

In case it would be helpful, I also have made sub lists by grepping all lines that begin with 1, 2, 3, ... for all possible ec.

Ideally, I would not use these sub lists, and I would be able to read right from the master list.

My output would look exactly like the all.inq, except that the subjects with ec=1 would be in a file 1.inq, and so on. Again, the relationship is mandated by the records in the master list.

I hope this is clear enough for you to help me now! Thank you,

Chris

PS you can safely assume all spaces are tabs.

While we are at it, I need to remove a group of subjects from ec 1. I have their subject numbers listed, but I don't know how to use this list to expedite deleting them from the master list or 1.lst.