How to check 2 log files for a common pattern?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to check 2 log files for a common pattern?
# 1  
Old 08-24-2015
How to check 2 log files for a common pattern?

hi!

im new here and to unix.

I want to do something with our log files. to compare two log files for a certain pattern.

sample:

file1.log contains all the "successful" run of a procedure.
file2.log contains all the "current" running procedures.

sample line from file1.log
Code:
2015-08-25 08:41:23,507 INFO Successful run: some procedure @ poolname (setup: 19ms / exec: 21ms) (requestor: requestorip) (ProcessID: processID)



sample line from file2.log
Code:
2015-08-25 08:41:23,507 INFO Currently Running: some procedure @ poolname (ProcessID: processID)

what I want to do is get all "processID" in file2.log that are not in file1.log
this will be executed at random or for reporting purposes.

edit:
by the way the processID looks like this

Code:
ProcessID: 108786b:14f62721fb3:-7fa2 ProcessID: 108786b:14f62721fb3:-7fa1 ProcessID: 108786b:14f62721fb3:-7fa0 ProcessID: 108786b:14f62721fb3:-7f9f ProcessID: 108786b:14f62721fb3:-7f9e
ProcessID: 108786b:14f62721fb3:-7f9d

is this possible?

thanks! Smilie

Last edited by Scrutinizer; 08-24-2015 at 11:05 PM.. Reason: added some info; mod: CODE tags
# 2  
Old 08-24-2015
Hi, try:
Code:
awk -F'[()]' '!/ProcessID: /{next} {p=$(NF-1)} NR==FNR{A[p]++; next} p in A{print p}'  file1.log file2.log


Last edited by Scrutinizer; 08-25-2015 at 12:25 AM..
# 3  
Old 08-24-2015
hi!

Sir as per testing it works but it returned the processID that is existing in both files. I need the ones that are not existing in both files.

sorry i'm still reading about unix commands. as of writing im only familiar with tail/grep commands
# 4  
Old 08-25-2015
OK, to get the reverse, try:
Code:
awk -F'[()]' '!/ProcessID: /{next} {p=$(NF-1)} NR==FNR{A[p]++; next} !(p in A){print p}'  file1.log file2.log

This User Gave Thanks to Scrutinizer For This Post:
# 5  
Old 08-25-2015
thanks Sir! Smilie

i'll try to play with this to add the "some procedure" part of the line.

by the way would it be too much to ask for an explanation of the command?

i'm currently reading this grymoire.com/Unix/Awk.html tutorial.

thanks! again! Smilie
# 6  
Old 08-25-2015
Reformatting and adding comments to Scrutinizer's code:
Code:
awk -F'[()]' '		# Set input field separators to the open and close
			# parenthesis characters.
!/ProcessID: /{next}	# Skip any line that does not contain "ProcessID: "
{p=$(NF-1)}		# Set p to the last string on the line between "(" and
			#    ")".  Presumably this sets p to the processID on
			#    the current input line from either input file.
NR==FNR{A[p]++; next}	# If this line was read from the 1st input file,
			#    increment the count of the number of time we have
			#    seen this processID in the 1st input file, and
			#    skip to the next input line.
p in A{print p}		# If the processID seen on this line in the 2nd input
			#    file was also seen in the 1st input file, print
			#    the processID.
' file1.log file2.log	# End the awk script & name the 1st and 2nd input files.

My reading of your requirements was that you wanted to print processIDs that appear in file2.log that DO NOT also appear in file1.log. If that is what you want, I believe the line shown in red above should be changed to:
Code:
!(p in A){print p}	# If the processID seen on this line in the 2nd input
			#    file was not seen in the 1st input file, print
			#    the processID.

And, I don't see any need to count the number of appearances of each processID in the 1st file, so the line:
Code:
NR==FNR{A[p]++; next}

could be simplified to just:
Code:
NR==FNR{A[p]; next}

to just note that the processID identified by p was found in the 1st input file (without counting the number of times it has been found).
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Comparing two files and list the difference with common first line content of both files

I have two file as given below which shows the ACL permissions of each file. I need to compare the source file with target file and list down the difference as specified below in required output. Can someone help me on this ? Source File ************* # file: /local/test_1 # owner: own #... (4 Replies)
Discussion started by: sarathy_a35
4 Replies

2. Shell Programming and Scripting

Merging two files without any common pattern

Hi I have file1 as IJU_NSOMOW; SOWWOD_TWUIQ; and file2 as how are you?; fine there; Now my problem is i need the output file as IJU_NSOMOW; how are you?; SOWWOD_TWUIQ; fine there; (2 Replies)
Discussion started by: Priya Amaresh
2 Replies

3. Shell Programming and Scripting

Finding log files that match number pattern

I have logs files which are generated each day depending on how many processes are running. Some days it could spin up 30 processes. Other days it could spin up 50. The log files all have the same pattern with the number being the different factor. e.g. LOG_FILE_1.log LOG_FILE_2.log etc etc ... (2 Replies)
Discussion started by: atelford
2 Replies

4. Shell Programming and Scripting

common entries of first column in 2 or 3 files:kindly check

Hi all, I have 3 files with such data first files second file third file I have to find common entries of first column in two ways 1) between 2 files (2 Replies)
Discussion started by: manigrover
2 Replies

5. Shell Programming and Scripting

Request to check: find common and write before it

Hi all, I have 2 big files with such kind of inputs File I File II I want the output file shuld contain Please let me knw scripting regarind this (1 Reply)
Discussion started by: manigrover
1 Replies

6. Shell Programming and Scripting

Request to check:find out common entries

I have to compare 2 files which means 2 files with common entries in same column and separate those common entries in a diferent file as well right before those entries common so that I can separat common and Uncommon entries in rows in 2 different files. Is it possible For eg. one file ... (3 Replies)
Discussion started by: manigrover
3 Replies

7. Shell Programming and Scripting

Find required files by pattern in xml files and the change the pattern on Linux

Hello, I need to find all *.xml files that matched by pattern on Linux. I need to have written the file name on the screen and then change the pattern in the file just was found. For instance. I can start the script with arguments for keyword and for value, i.e script.sh keyword... (1 Reply)
Discussion started by: yart
1 Replies

8. Shell Programming and Scripting

pattern search for multiple log files and counting

I have 10 appservers and each appserver has 4 jvms . Each of these logs is archived and stored on a nfs directory . For example the files are /logs/200907/ap1-jvm1.server.log.20090715.gz /logs/200907/ap2-jvm2.server.log.20090714.gz /logs/200908/ap1-jvm1.server.log.20090812.gz I want to... (3 Replies)
Discussion started by: gubbu
3 Replies

9. Shell Programming and Scripting

Check for files and log report if missing

Hi, I need a shell program that will prompt the user to input a dept name to the script. The script should then check a specific directory called ‘report' in the home directory of each member of that dept. If the report directory does not exist, or there are no contents in the directory,... (2 Replies)
Discussion started by: johnnyvlme
2 Replies

10. Shell Programming and Scripting

selecting only few lines from many based on a common pattern

Hi, I have a file having some thousand records with the following sort of lines: Error: Failed to get order data Order: PO-BBBTGZE Error: No CLI Error: Failed to get order data Order: PO-SBDJUZA Order: PO-XBBIDEN Error: No CLI Error: Failed to get order data Order: PO-BBDJUTQ Order:... (2 Replies)
Discussion started by: damansingh
2 Replies
Login or Register to Ask a Question