Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Shell script to search all files for every string in another file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to search all files for every string in another file
# 1  
Old 11-09-2016
Shell script to search all files for every string in another file
Hello All

I have a pattern.txt file in source directory ((/project/source/) in linux server and data looks like:

Code:
123abc17
234cdf19
235ifg20


I have multiple log files in log directory (/project/log/) in linux server and data for one log file looks like:

Code:
<?xml version="1.0" processid is 123abc17
read successfully at 20161109093456
<?xml version="1.0" process id is 986bng21
read successfully at 20161109093459
message id aazzkk110 is 123abc17
message id aakjahsdk110 is 234cdf19
<?xml version="1.0" processid is 235ifg20
read successfully at 20161109093456
<?xml version="1.0" process id is 987skj29


I want to grep every single string in the source file - pattern.txt against all log files in log directory
and populate the output in a output.txt file in directory (/project/output/) data should look like below:

Code:
<?xml version="1.0" processid is 123abc17
read successfully at 20161109093456
message id aazzkk110 is 123abc17
message id aakjahsdk110 is 234cdf19
<?xml version="1.0" processid is 235ifg20
read successfully at 20161109093456


If a string is matched against any log file, I want to check if line starts with '<?xml version' then i want to populate second line also in the output file along with matching line
otherwise I just want to populate the matching line.


Please help me in writing a unix command / shell script to achieve it.
Last edited by Scrutinizer; 11-09-2016 at 08:37 PM.. Reason: code tags
# 2  
Old 11-09-2016
grep can use a pattern file:
Code:
grep -f /project/source/pattern.txt $(  find /path/to/files/ -type -name '*.log' )

# 3  
Old 11-09-2016
Hi Jim

I ran below command and I got the following message about arguments.

Code:
grep -f /project/source/pattern.txt $(  find /project/log/ -type -name '*.log' )

'find: Arguments to -type should contain only one letter'

Please suggest me what changes do I need to make.
# 4  
Old 11-10-2016
An f argument is missing.
# 5  
Old 11-10-2016
You might get by with:
Code:
grep -f /project/source/pattern.txt $(  find /path/to/files/ -name '*.log' )

as long as you only have regular files with names ending with .log, but I would guess that Jim accidentally just dropped the letter f to only apply grep to regular files with names ending with .log.
Code:
grep -f /project/source/pattern.txt $(  find /path/to/files/ -type f -name '*.log' )

If there are lots of log files to be searched and/or they have long pathnames, you might be safer using:
Code:
find /path/to/files/ -name '*.log' -exec grep -f /project/source/pattern.txt {} +

which should avoid any cases where the first two commands might fails with an "argument list too long" error.

Note, however, that any of these will just print the names of the files containing a line matching a line that from your pattern file. The grep utility that you requested be used does not contain sufficient power to select and print a varying number of lines following a matched pattern. For that you need something more like awk:
Code:
#!/bin/ksh
pattern_file="/project/source/pattern.txt"
log_dir="/project/log"

find "$log_dir" -type f -name '*.log' -exec awk '
FNR == NR {
	patterns[++patternsc] = $0
	next
}
/^<[?]xml version/ {
	n = 2
}
{	for(i = 1; i <= patternsc; i++)
		if($0 ~ patterns[i])
			break
	if(i <= patternsc)
		p = ($0 ~ /^<[?]xml version/) ? 2 : 1
}
p {	print
	p--
}' "$pattern_file" {} +

This was written and tested using a Korn shell on macOS 10.12.1, but will work with any shell that accepts Bourne shell syntax. If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
# 6  
Old 11-10-2016
Hi Don

Thank you for the script, when I executed it I was seeing all the matching lines against the patterns in the pattern.txt file but even if the line is starting with

Code:
 <?xml version

its not populating the second line. Could you please help me with with that.
# 7  
Old 11-10-2016
Hi,

For given input / output samples, Don solution nice solution works perfectly SmilieSmilie

Here is another try:
Code:
awk 'NR==FNR{a[NR]=$0;b=NR;next} { for(i=1;i<=b;i++) { if ( $0 ~ a[i] && $0 ~ /<[?]xml version/) { print $0;getline;print $0; }  else if ( $0 ~ a[i] ) print $0 } } ' pattern.txt file.log

It gives desired output as posted in #1

@Don : Can you help me to understand your code better ?
Code:
/^<[?]xml version/ {
	n = 2
}

Is it just for condition/action block because n is not used after that ?

Code:
p {	print
	p--
}'

Use of p in red here is unclear .
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to Search a string in a file in shell script?

I have a text file which is generated when the batch job is run. This batch may take few mins to run. When completed, the last line of the text file would be process completed. I need a shell script which will wait for this file till the process completed is printed in it, once found, it would move... (2 Replies)
Discussion started by: Lalat
2 Replies

2. Shell Programming and Scripting

Linux shell script, search by an input string

So, there is a large file where I have to conduct several search using bash shell scripting. The file is like this: TITLE and AUTHOR ETEXT NO. Aspects of plant life; with special reference to the British flora, 56900 by Robert Lloyd... (1 Reply)
Discussion started by: Philia
1 Replies

3. Shell Programming and Scripting

UNIX shell script to search a string in a file

Hi folks, I am new for shell script, I hope somebody to help me to write shell script My requirement is below steps 1. I have apache access.log i.e located in /var/log/httpd/ Ex. 127.0.0.1 - - "GET... (14 Replies)
Discussion started by: Chenchireddy
14 Replies

4. UNIX for Dummies Questions & Answers

Search for string in a file then compare it with excel files entry

All, i have a file text.log: cover6 cover3 cover2 cover4 other file is abc.log as : 0 0 1 0 Then I have a excel file result.xls that contains: Name Path Pass cover2 cover3 cover6 cover4 (1 Reply)
Discussion started by: Anamika08
1 Replies

5. Shell Programming and Scripting

Search string within a file and list common words from the line having the search string

Hi, Need your help for this scripting issue I have. I am not really good at this, so seeking your help. I have a file looking similar to this: Hello, i am human and name=ABCD. How are you? Hello, i am human and name=PQRS. I am good. Hello, i am human and name=ABCD. Good bye. Hello, i... (12 Replies)
Discussion started by: royzlife
12 Replies

6. Shell Programming and Scripting

Shell script to search all entries from 1 file to all other separated files.

Hi, I am trying to create a shell script in unix platform, hence will need info on how to start and any ideas from you guys. million thx. Objective: Shell script to search all entries from 1 file(a.out) to all other files and extract the search output to 1 file (c.out). Situation, 1)... (8 Replies)
Discussion started by: Mr_47
8 Replies

7. Shell Programming and Scripting

Need to build Shell Script to search content of a text file into a folder consist several files

Have to read one file say sourcefile containing several words and having another folder containing several files. Now read the first word of Sourcefile & search it into the folder consisting sevral files, and create another file with result. We hhave to pick the filename of the file in which... (3 Replies)
Discussion started by: mukesh.baranwal
3 Replies

8. Shell Programming and Scripting

Shell Script to Search for a particular String and copy the timestamp to a variable

Hi, We Perfrom Loads to the database through a Perl script which generates a statistics file. I need to read the statistics. the Statistics file looks something like below: Process Beginning - 08-26-2010-23.41.47 DB2 CONNECTION SUCCESSFUL! Ready to process and load file: FILENAME # of... (2 Replies)
Discussion started by: Praveenkulkarni
2 Replies

9. Shell Programming and Scripting

shell script to search a string and delete the content

Hi, I've a shell script e.g. #!/bin/bash echo "Enter the next hop id" read nhid echo "enter the IP address" read IP echo "enter the interface name" read name echo "enter the enable/disable state" read state exit 0 now from this script i want to search strings in another (.cam) ... (6 Replies)
Discussion started by: vic_mnnit
6 Replies

10. UNIX for Dummies Questions & Answers

count the number of files which have a search string, but counting the file only once

I need to count the number of files which have a search string, but counting the file only once if search string is found. eg: File1: Please note that there are 2 occurances of "aaa" aaa bbb ccc aaa File2: Please note that there are 3 occurances of "aaa" aaa bbb ccc... (1 Reply)
Discussion started by: sudheshnaiyer
1 Replies
Login or Register to Ask a Question