read in variable data from another file - grep


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting read in variable data from another file - grep
# 1  
Old 12-16-2008
Question read in variable data from another file - grep

Hello! I think this should be an easy solution. I have a large file with many fields of data. The first field has a unique identifier (a subject number) for every record for a chunk of data. Something like this:
Quote:
1 [Data] [Data] ...
1 [Data] [Data] ...
1 [Data] [Data] ...
1 [Data] [Data] ...
2 [Data] [Data] ...
2 [Data] [Data] ...
2 [Data] [Data] ...
2 [Data] [Data] ...
There were ten experimental conditions (ec), but the ec is identified by only one field in one record for each subject. I already ran a script to create a list of all subject numbers and their corresponding ec. For simplicity's sake (or so I think) I divided up that list by ec, so i now have numerous lists containing all of the subject numbers for each ec. I hope you can picture this.

Lets call my main data file "all.inq" and my lists "[0-9].lst"

I want to grep all the records in all.inq that begin with the numbers listed in 0.lst and put them in a file 0.inq, and then repeat the process for all other lists. This is probably a one liner, but I just don't know much about grep.

Thanks!
Chris
# 2  
Old 12-16-2008
I thought of a way to simplify the question and make it general.

grep syntax is:
Quote:
grep [options] PATTERN [files]
Can you have grep read the pattern from a file, and go line by line through that file and just append all standard output to the intended file?
# 3  
Old 12-16-2008
for name in list
do
egrep ^$name pattern > $name.out

done

to dump all in the same file use >> $name.out.

list would be the numbers.
# 4  
Old 12-16-2008
Quote:
Originally Posted by nj78
for name in list
do
egrep ^$name pattern > $name.out

done
I dont understand how this would work. It looks to me like list would be the file name of my list file, and name would just read in each record in the file (??? I dont know if that is the right interpretation). Then, the grep command looks for lines in the data file that start with the subject numbers in the list file. However, If I print to >> $name.out, then I am going to have a new file for every subject... which would not be good. I want to group by experimental condition.

If I am interpreting this correctly, I would be able to to this and just define the output files name to explicly be the ec number, if I read from my individual lists that are already grouped by condition.

Is there a way to read in column $1 from the list file (which is the ec) and have that govern the name of the .out, and then read the second column of the list file (the subject number) into the grep search pattern pattern?

So the grep line would be something like this, i think:
Code:
grep '^$2' all.inq >> $1.out

I just dont know how to read in the master list file so that it would work.
# 5  
Old 12-16-2008
is 'ec' the first field in all.inq?

Pls post a sample all.inq, a sample 1.lst and a desired output of 1.inq

And indicate a 'common' field between all.ina and 1.lst files
# 6  
Old 12-16-2008
clarification

Quote:
Originally Posted by vgersh99
is 'ec' the first field in all.inq?

Pls post a sample all.inq, a sample 1.lst and a desired output of 1.inq

And indicate a 'common' field between all.inq and 1.lst files
Alright, this is a sample from all.inq:
Quote:
570613667 6 191 compatible unpleasb 30 1 579 30 Anger
570613667 6 192 compatible europeanamericanright 76 1 490 35 g10cm.bmp
570613667 6 193 compatible unpleasb 30 1 805 49 Debt
570613667 6 194 compatible pleasb 76 1 803 3 Dinner
570613667 6 195 compatible pleasb 30 0 617 11 Merry
570613667 6 196 compatible unpleasb 30 1 638 15 Lice
570613667 6 197 compatible pleasb 76 1 595 43 Home
570613667 6 198 compatible pleasb 76 1 569 35 Yacht
570613667 6 199 compatible europeanamericanright 76 1 497 50 g28cm.bmp
570613667 6 200 compatible unpleasb 30 1 627 48 Broken
The file is enormous so this is only a small chunk of block 6 from subject# 570613667. If you look in my first post, that simplified version should be enough to work with. The experiment condition is only in one field of one record at the top of each subject (that line also begins with the subject#, which is a good thing for me). I already handled this part of the process with awk. I found a unique quality of a field in the first line, and printed the ec and sn fields ($6=ec; $1=sn, if you care).

So, my master list looks like this:
Quote:
1 72430987
1 48942358
3 95842375
2 89432754
6 94378257
8 98734193
4 74322245
...
There are no headers. $1=ec; $2=sn

In case it would be helpful, I also have made sub lists by grepping all lines that begin with 1, 2, 3, ... for all possible ec.

Ideally, I would not use these sub lists, and I would be able to read right from the master list.

My output would look exactly like the all.inq, except that the subjects with ec=1 would be in a file 1.inq, and so on. Again, the relationship is mandated by the records in the master list.

I hope this is clear enough for you to help me now! Thank you,

Chris

PS you can safely assume all spaces are tabs.
# 7  
Old 12-16-2008
While we are at it, I need to remove a group of subjects from ec 1. I have their subject numbers listed, but I don't know how to use this list to expedite deleting them from the master list or 1.lst.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

2. Shell Programming and Scripting

How to read each file grep a value from that?

Hi Team, in /tmp folder i have thousands of log files i want to read each file and grep a value called "Calling This". Each logfile name is different but it ends with .log. How can i achieve this? Please excuse if i did any mistake by not following forum standards. I will surely follow... (10 Replies)
Discussion started by: darling
10 Replies

3. Shell Programming and Scripting

Read csv file, convert the data and make one text file in UNIX shell scripting

I have input data looks like this which is a part of a csv file 7,1265,76548,"0102:04" 8,1266,76545,"0112:04" I need to make the output data should look like this and the output data will be part of text file: 7|1265000 |7654899 |A| 8|12660000 |76545999 |B| The logic behind the... (6 Replies)
Discussion started by: RJG
6 Replies

4. Shell Programming and Scripting

How to read file and load data into a script as variable

I need to read a text file that contain columns of data, i need to read 1st column as a function to call, and others are the data i need to get into a ksh script. I am quite new to ksh scripting, i am not very sure how to read each row line by line and the data in each columns of that line, set... (3 Replies)
Discussion started by: gavin_L
3 Replies

5. Shell Programming and Scripting

Perl: How to read text from file and process $variable in that data too.

In the hello.htm have the sentenses: Hello $name How are you? The perl script: $name = "David"; open(HEADER,"hello.htm"); while(<HEADER>) { $html .= $_; } close(HEADER); print "$html";I making something about template. But it can't process the $name variable. (4 Replies)
Discussion started by: natong
4 Replies

6. Shell Programming and Scripting

Want to read data from a file name.txt and search it in another file and then matching...

Hi Frnds... I have an input file name.txt and another file named as source.. name.txt is having only one column and source is having around 25 columns...i need to read from name.txt line by line and search it in source file and then save the result in results file.. I have a rough idea about the... (15 Replies)
Discussion started by: ektubbe
15 Replies

7. UNIX for Advanced & Expert Users

how to read the data from an excel sheet and use those data as variable in the unix c

I have 3 columns in an excel sheet. c1 c2 c3 EIP_ACCOUNT SMALL_TS_01 select A.* from acc; All the above 3 col shoud be passed a variable in the unix code. 1.How to read an excel file 2.How to pass these data as variable to the unic script (1 Reply)
Discussion started by: Anne Grace
1 Replies

8. Shell Programming and Scripting

Post Shell programming: Question about source a file and read data from the file

This is shell programming assignment. It needs to create a file called .std_dbrc contains STD_DBROOT=${HOME}/class/2031/Assgn3/STD_DB (which includes all my simple database files) and I am gonna use this .std_dbrc in my script file (read the data from the database files) like this: .... (3 Replies)
Discussion started by: ccwq
3 Replies

9. Shell Programming and Scripting

Read from data file

Hi, I have a data file formatted like this: Ex: Mike 3434 Jack 481 Peter 12 Alan 926 I want to get this data into 2 variables: "Names" and "Numbers" that I can using one "for" loop to get the value as Names and Numbers Like this: for i in 0 1 2 3 do echo $Names echo... (12 Replies)
Discussion started by: fongthai
12 Replies

10. Shell Programming and Scripting

Read data from a file into a variable

I am a FORTRAN guy and not a UNIX expert by any means so sorry if this sounds dumb, but all I want to do is have a UNIX script which reads data from a file (say 1000 lines worth, each row is a file name) and store it in an array to perform an operation on later. As maddeningly simple as this... (2 Replies)
Discussion started by: yorkdg
2 Replies
Login or Register to Ask a Question