Grep strings on multiple files and output to multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grep strings on multiple files and output to multiple files
# 1  
Old 04-19-2016
Grep strings on multiple files and output to multiple files

Hi All,

I want to use egrep on multiple files and the results should be output to multiple files. I am using the below code in my shell script(working in Ksh shell). However with this code I am not attaining the desired results.

Code:
#!/bin/ksh
(
a="/path/file1"
b="path/file2"

for file in $a $b
do
egrep -iv '513|519|532' "$file" > test1 > test2
done
)
exit

The above code is resulting 2 files 'test1' and 'test2' whereas 'test1' does not contain anything in it(its empty file). 'test2' contains values of '/path/file2' whereas 513,519,532 records were ignored.

I want my results should be like below:

‘test1' should contain values of ‘/path/file1' whereas 513,519,532 records should be ignored. And
‘test2' should contain values of ‘/path/file2' whereas 513,519,532 records should be ignored.

Can anyone of you suggest me any solution on this ?

Thanks in advance.
Regards,
am24
# 2  
Old 04-19-2016
First, note that /path/file2 and path/file2 are only guaranteed to identify the same file if you are sitting in the system's root directory when you invoke this script.

Second, that is not the way grep (or egrep) works. If you want different output files for different input file, you'll need to invoke grep once for each desired output file.

Alternatively, you could use something like awk to simulate multiple invocations of egrep -v (assuming that output files are sequentially numbered and all have the base test) using something like:
Code:
#!/bin/ksh
awk '
FNR == 1 {
	if(NR > 1)
		close(of)
	of = "test" ++nf
}
! /51[39]|532/ {
	print > of
}' "$@"

and invoke this script with pathnames of the files to want to process as operands. Or, you could change the last line of this script from:
Code:
}' "$@"

to include an explicit list of the files you want to process:
Code:
}' /path/file[12]

As always, if you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.

Although written and tested using the Korn shell, this will work with any shell that uses basic Bourne shell syntax.

Last edited by Don Cragun; 04-19-2016 at 05:44 PM.. Reason: Fix typo: s/30]/39]/ in "! /51[39]|532/ {" & s/he/the/.
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 04-19-2016
Hi Don,

Thank you so much for your suggestion. I have tried the below code and able to get the desired results.

Code:
#!/bin/ksh
nawk '
FNR == 1 {
	if(NR > 1)
		close(of)
	of = "test" ++nf
}
! /5130|532/ {
	print > of
}' /path/file[12]

I did not understand what is the relation between FNR and NR in this code. Can you Please brief me on that ?

Regards,
am24
# 4  
Old 04-19-2016
Quote:
Originally Posted by am24
Hi Don,

Thank you so much for your suggestion. I have tried the below code and able to get the desired results.

Code:
#!/bin/ksh
nawk '
FNR == 1 {
	if(NR > 1)
		close(of)
	of = "test" ++nf
}
! /5130|532/ {
	print > of
}' /path/file[12]

I did not understand what is the relation between FNR and NR in this code. Can you Please brief me on that ?

Regards,
am24
FNR is the current line number in the current input file. So, FNR == 1 is true when you are looking at the 1st line in a file. NR is the number of lines read from all input files. So, when FNR is 1 and NR is also 1, you are looking at the 1st line in the 1st input file. And, when FNR is 1 and NR is greater than 1, you are looking at the 1st line in the 2nd or 3rd or ... (but not the 1st) input file.

Note also that you used:
Code:
! /5130|532/ {

which will select any line that does not contain the string "5130" and does not contain the string "532". The code I suggested used:
Code:
! /51[30]|532/ {

which was a typo. If should have been:
Code:
! /51[39]|532/ {

which would select any line that does not contain the string "513", does not contain the string "519", and does not contain the string "532" (which is the same as what either of the commands:
Code:
egrep -v '513|519|532'
egrep -v '51[39]|532'

would select). I will edit my earlier post to correct that typo in a few minutes.

Last edited by Don Cragun; 04-19-2016 at 09:32 PM..
This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 04-20-2016
Hello Don,

Thank you for the brief explanation on the code. I have got a clear understanding now.

Thanks again for your time to look into this Smilie

Regards,
am24
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Single grep to multiple strings with separate output per string

I need to grep multiple strings from a particular file. I found the use of egrep "String1|String2|String3" file.txt | wc-l Now what I'm really after is that I need to separate word count per each string found. I am trying to keep it to use the grep only 1 time. Can you guys help ? ... (9 Replies)
Discussion started by: nms
9 Replies

2. Post Here to Contact Site Administrators and Moderators

Redirecting grep output to multiple files

Hi All, I am trying to redirect the grep output to multiple files, can you please help with that. Below is the command im using to match my pattern grep \<proxyType\>$PxyType $DIR/EndureFiles.json > File_Name*.json Note : $DIR and $PxyType is already defined in my script Im able... (0 Replies)
Discussion started by: Deena1984
0 Replies

3. Shell Programming and Scripting

Create Multiple UNIX Files for Multiple SQL Rows output

Dear All, I am trying to write a Unix Script which fires a sql query. The output of the sql query gives multiple rows. Each row should be saved in a separate Unix File. The number of rows of sql output can be variable. I am able save all the rows in one file but in separate files. Any... (14 Replies)
Discussion started by: Rahul_Bhasin
14 Replies

4. Shell Programming and Scripting

Grep multiple strings in multiple files

Hi, every one! I have a file with multiple strings. file1 ATQRGNE ASQGVKFTE ASSQYRDRGGLET SPEQGARSDE ASSRDFTDT ASSYSGGYE ASSYTRLWNTGE ASQGHNTD PSLGGGNQPQH SLDRDSYNEQF I want to grep each string in hundreds of files in the same directory, further, I want to find out the string... (7 Replies)
Discussion started by: xshang
7 Replies

5. Shell Programming and Scripting

Search & Replace: Multiple Strings / Multiple Files

I have a list of files all over a file system e.g. /home/1/foo/bar.x /www/sites/moose/foo.txtI'm looking for strings in these files and want to replace each occurrence with a replacement string, e.g. if I find: '#@!^\&@ in any of the files I want to replace it with: 655#@11, etc. There... (2 Replies)
Discussion started by: spacegoose
2 Replies

6. Shell Programming and Scripting

Grep multiple terms and output to individual files

Hi all, I'll like to search a list of tems in a huge file and then output each of the terms to individual files. I know I can use grep -f list main.file to search them but how can I split the output into individual files? Thank you. (6 Replies)
Discussion started by: ivpz
6 Replies

7. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

8. UNIX for Dummies Questions & Answers

Grep multiple strings in multiple files using single command

Hi, I will use below command for grep single string ("osuser" is search string) ex: find . -type f | xarg grep -il osuser but i have one more string "v$session" here i want to grep in which file these two strings are present. any help is appreciated, Thanks in advance. Gagan (2 Replies)
Discussion started by: gagan4599
2 Replies

9. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

10. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies
Login or Register to Ask a Question