Sponsored Content
Top Forums Shell Programming and Scripting Compare three files based on two fields Post 302748731 by PikK45 on Thursday 27th of December 2012 12:08:13 AM
Old 12-27-2012
Compare three files based on two fields

Guys,

I tried searching on the internet and I couldn't get the answer for this problem. I have 3 files. First 2 fields of all of them are of same type, say they come from various databases but first two fields in the 3 files means the same.

I need to verify the entries that are not present in all the 3 files.

Below are the files 1.txt, 2.txt & 3.txt respectively

Code:
2677|47876992|1|20:20:51|12/16/2012|1|1|496300|
2677|47877024|2|13:47:37|12/16/2012|1|1|008994|
2677|47877052|3|21:03:56|12/16/2012|1|1|647546|
2677|47877055|4|16:54:07|12/16/2012|1|1|133914|
2677|47877099|5|16:29:06|12/16/2012|1|1|379245|
2677|47877081|6|10:44:13|12/16/2012|1|1|014078|
2677|47877232|7|19:07:06|12/16/2012|1|1|242776|
2677|47877246|8|13:02:32|12/16/2012|1|1|623853|
2677|47877258|9|22:03:05|12/16/2012|1|1|997345|
2677|47877351|10|16:29:27|12/16/2012|1|1|792584|
 
2677|47876992|1|1|40|
2677|47877024|32|1|100|
2677|47877052|2|1|39|
2677|47877055|1|1|75|
2677|47877074|1|1|9|
2677|47877081|2|1|175|
2677|47877232|1|1|10|
2677|47877246|9|1|25|
2677|47877258|25|1|40|
2677|47877350|9|1|50|

2677|47876992|1|7000|603098|40|0|
2677|47877024|1|7000|603086|100|0|
2677|47877052|1|1700|200180|39|0|
2677|47877055|1|7000|603098|75|0|
2677|47877074|1|1700|003400|9|0|
2677|47877081|1|7000|603062|25|0|
2677|47877081|2|7000|603065|50|0|
2677|47877081|3|7000|603074|100|0|
2677|47877232|1|7000|601802|10|0|
2677|47877246|1|7000|252120|25|0|

The output should be the lines that are not in all the three files. that is like below.

Code:
2677|47877099
2677|47877258
2677|47877351
2677|47877350

It would be geat if the output has the filenames as well. Smilie Smilie like below

Code:
2677|47877099|1.txt|
2677|47877258|1.txt,2.txt|
2677|47877351|1.txt|
2677|47877350|2.txt|

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare two files based on values of fields.

Hi All, I have two files and data looks like this: File1 Contents #Field1,Field2 Dist_Center_file1.txt;21 Dist_Center_file3.txt;20 Dist_Center_file2.txt;20 File2 Contents (*** No Header ***) Dist_Center_file1.txt;23 Dist_Center_file2.txt;20 Dist_Center_file3.txt;20 I have... (4 Replies)
Discussion started by: Hangman2
4 Replies

2. Shell Programming and Scripting

Compare fields in 2 files using AWK

Hi unix gurus, I have a urgent requirement, I need to write a AWK script to compare each fields in 2 files using AWK. Basically my output should be like this. file1 row|num1|num2|num3 1|one|two|three 2|one|two|three file2 row|num1|num2|num3 1|one|two|three 2|one|two|four ... (5 Replies)
Discussion started by: rashmisb
5 Replies

3. Shell Programming and Scripting

AWK Compare files, different fields, output

Hi All, Looking for a quick AWK script to output some differences between two files. FILE1 device1 1.1.1.1 PINGS device1 2.2.2.2 PINGS FILE2 2862 SITE1 device1-prod 1.1.1.1 icmp - 0 ... (4 Replies)
Discussion started by: stacky69
4 Replies

4. Shell Programming and Scripting

Compare files with fields separated with semicolon

Dear experts I have files like ABD : 5869 events, relative ratio : 1.173800E-01 , sum of ratios : 1.173800E-01 VBD : 12147 events, relative ratio : 2.429400E-01 , sum of ratios : 3.603200E-01 SDF : 17000 events, relative ratio : 3.400000E-01 , sum of ratios : 7.003200E-01 OIP: 14984... (9 Replies)
Discussion started by: Alkass
9 Replies

5. Shell Programming and Scripting

compare fields in different files

HI I'm having some troubles to compare and permut diffrent fields indexed with another filed like the following example `: file1 1 1 2 2 3 3 file2 7 1 9 2 10 3 result------------------- (6 Replies)
Discussion started by: yassinegoth
6 Replies

6. Shell Programming and Scripting

Compare fields in files

Hi, I need the most efficient way of comparing the following and arriving at the result I have a file which has entries like, File1: 1|2|5|7|8|2|3|6|3|1 File2: 1|2|3|1|2|7|9|2 I need to compare the entries in these two file with those of a general file, 1|2|3|5|2|5|6|9|3|1... (7 Replies)
Discussion started by: pradebban
7 Replies

7. Shell Programming and Scripting

Compare two fields in text files?

Hi, I have two text files, compare column one in both the files and if it matches then the output should contain the id in column one, the number and the description. Both the files are sorted. Is there a one liner to get this done, kindly help. Thank you File 1: NC_000964 92.33 ... (2 Replies)
Discussion started by: pulikoti
2 Replies

8. UNIX for Dummies Questions & Answers

Compare 2 fields in 2 files

I am trying to compare two files (separted by a pipe) using 2 fields (field 1,3 from fileA and 1,2 from fileB) if the two files match i want the whole record of fileA adding the extra fields left from fileB. 1. A.txt cat|floffy|12|anything|anythings cat|kitty|15|lala|lalala... (6 Replies)
Discussion started by: sabercats
6 Replies

9. Shell Programming and Scripting

Compare fields in two files

Hi, I am trying to check two files based on certain string and field. cat f1 source=\GREP\" hi this \\ source=\SED\" skdmsmd dnksdns source=\PERL\" cat f2 source=\SED\" source=\GREP\" vlamskds amdksk m source=\AWK\" awk \here\" (3 Replies)
Discussion started by: greet_sed
3 Replies

10. UNIX for Beginners Questions & Answers

Is there a UNIX command that can compare fields of files with differing number of fields?

Hi, Below are the sample files. x.txt is from an Excel file that is a list of users from Windows and y.txt is a list of database account. $ head -500 x.txt y.txt ==> x.txt <== TEST01 APP_USER_PROFILE USER03 APP_USER_PROFILE TEST02 APP_USER_EXP_PROFILE TEST04 APP_USER_PROFILE USER01 ... (3 Replies)
Discussion started by: newbie_01
3 Replies
RPNTUTORIAL(1)							      rrdtool							    RPNTUTORIAL(1)

NAME
rpntutorial - Reading RRDtool RPN Expressions by Steve Rader DESCRIPTION
This tutorial should help you get to grips with RRDtool RPN expressions as seen in CDEF arguments of RRDtool graph. Reading Comparison Operators The LT, LE, GT, GE and EQ RPN logic operators are not as tricky as they appear. These operators act on the two values on the stack preceding them (to the left). Read these two values on the stack from left to right inserting the operator in the middle. If the resulting statement is true, then replace the three values from the stack with "1". If the statement if false, replace the three values with "0". For example, think about "2,1,GT". This RPN expression could be read as "is two greater than one?" The answer to that question is "true". So the three values should be replaced with "1". Thus the RPN expression 2,1,GT evaluates to 1. Now consider "2,1,LE". This RPN expression could be read as "is two less than or equal to one?". The natural response is "no" and thus the RPN expression 2,1,LE evaluates to 0. Reading the IF Operator The IF RPN logic operator can be straightforward also. The key to reading IF operators is to understand that the condition part of the traditional "if X than Y else Z" notation has *already* been evaluated. So the IF operator acts on only one value on the stack: the third value to the left of the IF value. The second value to the left of the IF corresponds to the true ("Y") branch. And the first value to the left of the IF corresponds to the false ("Z") branch. Read the RPN expression "X,Y,Z,IF" from left to right like so: "if X then Y else Z". For example, consider "1,10,100,IF". It looks bizarre to me. But when I read "if 1 then 10 else 100" it's crystal clear: 1 is true so the answer is 10. Note that only zero is false; all other values are true. "2,20,200,IF" ("if 2 then 20 else 200") evaluates to 20. And "0,1,2,IF" ("if 0 then 1 else 2) evaluates to 2. Notice that none of the above examples really simulate the whole "if X then Y else Z" statement. This is because computer programmers read this statement as "if Some Condition then Y else Z". So it's important to be able to read IF operators along with the LT, LE, GT, GE and EQ operators. Some Examples While compound expressions can look overly complex, they can be considered elegantly simple. To quickly comprehend RPN expressions, you must know the algorithm for evaluating RPN expressions: iterate searches from the left to the right looking for an operator. When it's found, apply that operator by popping the operator and some number of values (and by definition, not operators) off the stack. For example, the stack "1,2,3,+,+" gets "2,3,+" evaluated (as "2+3") during the first iteration and is replaced by 5. This results in the stack "1,5,+". Finally, "1,5,+" is evaluated resulting in the answer 6. For convenience, it's useful to write this set of operations as: 1) 1,2,3,+,+ eval is 2,3,+ = 5 result is 1,5,+ 2) 1,5,+ eval is 1,5,+ = 6 result is 6 3) 6 Let's use that notation to conveniently solve some complex RPN expressions with multiple logic operators: 1) 20,10,GT,10,20,IF eval is 20,10,GT = 1 result is 1,10,20,IF read the eval as pop "20 is greater than 10" so push 1 2) 1,10,20,IF eval is 1,10,20,IF = 10 result is 10 read pop "if 1 then 10 else 20" so push 10. Only 10 is left so 10 is the answer. Let's read a complex RPN expression that also has the traditional multiplication operator: 1) 128,8,*,7000,GT,7000,128,8,*,IF eval 128,8,* result is 1024 2) 1024 ,7000,GT,7000,128,8,*,IF eval 1024,7000,GT result is 0 3) 0, 7000,128,8,*,IF eval 128,8,* result is 1024 4) 0, 7000,1024, IF result is 1024 Now let's go back to the first example of multiple logic operators, but replace the value 20 with the variable "input": 1) input,10,GT,10,input,IF eval is input,10,GT ( lets call this A ) Read eval as "if input > 10 then true" and replace "input,10,GT" with "A": 2) A,10,input,IF eval is A,10,input,IF read "if A then 10 else input". Now replace A with it's verbose description again and--voila!--you have an easily readable description of the expression: if input > 10 then 10 else input Finally, let's go back to the first most complex example and replace the value 128 with "input": 1) input,8,*,7000,GT,7000,input,8,*,IF eval input,8,* result is A where A is "input * 8" 2) A,7000,GT,7000,input,8,*,IF eval is A,7000,GT result is B where B is "if ((input * 8) > 7000) then true" 3) B,7000,input,8,*,IF eval is input,8,* result is C where C is "input * 8" 4) B,7000,C,IF At last we have a readable decoding of the complex RPN expression with a variable: if ((input * 8) > 7000) then 7000 else (input * 8) Exercises Exercise 1: Compute "3,2,*,1,+ and "3,2,1,+,*" by hand. Rewrite them in traditional notation. Explain why they have different answers. Answer 1: 3*2+1 = 7 and 3*(2+1) = 9. These expressions have different answers because the altering of the plus and times operators alter the order of their evaluation. Exercise 2: One may be tempted to shorten the expression input,8,*,56000,GT,56000,input,*,8,IF by removing the redundant use of "input,8,*" like so: input,56000,GT,56000,input,IF,8,* Use traditional notation to show these expressions are not the same. Write an expression that's equivalent to the first expression, but uses the LE and DIV operators. Answer 2: if (input <= 56000/8 ) { input*8 } else { 56000 } input,56000,8,DIV,LE,input,8,*,56000,IF Exercise 3: Briefly explain why traditional mathematic notation requires the use of parentheses. Explain why RPN notation does not require the use of parentheses. Answer 3: Traditional mathematic expressions are evaluated by doing multiplication and division first, then addition and subtraction. Parentheses are used to force the evaluation of addition before multiplication (etc). RPN does not require parentheses because the ordering of objects on the stack can force the evaluation of addition before multiplication. Exercise 4: Explain why it was desirable for the RRDtool developers to implement RPN notation instead of traditional mathematical notation. Answer 4: The algorithm that implements traditional mathematical notation is more complex then algorithm used for RPN. So implementing RPN allowed Tobias Oetiker to write less code! (The code is also less complex and therefore less likely to have bugs.) AUTHOR
Steve Rader <rader@wiscnet.net> 1.4.8 2013-05-23 RPNTUTORIAL(1)
All times are GMT -4. The time now is 10:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy