Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

comparing 2 text files to get unique values??

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting comparing 2 text files to get unique values??
# 1  
Old 12-16-2008
comparing 2 text files to get unique values??
Hi all,

I have got a problem while comparing 2 text files and the result should contains the unique values(Non repeatable).

For eg:

file1.txt
1
2
3
4

file2.txt
2
3

So after comaping the above 2 files I should get only 1 and 4 as the output. Pls help me out.
# 2  
Old 12-16-2008
Code:
awk ' FILENAME=="file1" { if ($0 in arr) {continue} else {print $0}}
        FILENAME=="file2" {arr[$0]++ }
       ' file2 file1

# 3  
Old 12-16-2008
Code:
( cat file1.txt file2.txt ) | sort | uniq -c | awk '$1==1 {print $2}'

Last edited by vgersh99; 12-16-2008 at 01:51 PM..
# 4  
Old 12-16-2008
@vgersh99 @Jim


Thanks Frens, ur code works awesome...

It gave me the proper o/p. hey if u don mind can u pls explain me the things in that?

anyways thanks for ur help...
Last edited by smarty86; 12-16-2008 at 02:02 PM..
# 5  
Old 12-16-2008
Code:
( cat file1.txt file2.txt ) | sort | uniq -c | nawk -v OFS='\t' '$1==1 {$1=$1;print}' | cut -f2-

# 6  
Old 12-16-2008
Quote:
Originally Posted by jim mcnamara
Code:
awk ' FILENAME=="file1" { if ($0 in arr) {continue} else {print $0}}
        FILENAME=="file2" {arr[$0]++ }
       ' file2 file1

Jim,
if the content of file1 and file2 is reversed, this will not work.

here's another alternative (assuming a given record/line appears only once in a given file):
Code:
 nawk '{ if ($0 in a) delete a[$0]; else a[$0]} END { for (i in a) print i}' file1 file2

Last edited by vgersh99; 12-16-2008 at 02:27 PM..
# 7  
Old 12-16-2008
@vgersh99

after ur changes the first code is working well man... thank u.. pls explain me about awk statement in that if u dont mind...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge 4 bim files by keeping only the overlapping variants (unique rs values )

Dear community, I am facing a problem and I kindly ask your help: I have 4 different data sets consisted from 3 different types of array. On each file, column 1 is chromosome position, column 2 is SNP id etc... Lets say I have the following (bim) datasets: x2014: 1 rs3094315... (4 Replies)
Discussion started by: fondan
4 Replies

2. Shell Programming and Scripting

How to identify varying unique fields values from a text file in UNIX?

Hi, I have a huge unsorted text file. We wanted to identify the unique field values in a line and consider those fields as a primary key for a table in upstream system. Basically, the process or script should fetch the values from each line that are unique compared to the rest of the lines in... (13 Replies)
Discussion started by: manikandan23
13 Replies

3. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber... (3 Replies)
Discussion started by: Sharma331
3 Replies

4. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

5. Shell Programming and Scripting

Comparing the values of two files

Hi Am trying to compare the values of two files.. One is a big file that has many values and the other is a small file.. The big file has all values present in small file.. # cat SmallFile 4456602 22347881 7471282 15859891 8257690 21954701 7078068 18219229 2883826 6094959 100000 ... (3 Replies)
Discussion started by: Priya Amaresh
3 Replies

6. UNIX for Dummies Questions & Answers

Comparing two test files and printing out the values that do not match

Hi, I have two text files with matching first columns. Some of the values in the second column do not match. I want to write a script to print out the rows (only the first column) where the values in the second column do not match. Example: Input 1 A 1 B 2 C 3 D 4 Input 2 A 2 B 2... (6 Replies)
Discussion started by: evelibertine
6 Replies

7. UNIX for Dummies Questions & Answers

Comparing two text files by a column and printing values that do not match

I have two text files where the first three columns are exactly the same. I want to compare the fourth column of the text files and if the values are different, print that row into a new output file. How do I go about doing that? File 1: 100 rs3794811 0.01 0.3434 100 rs8066551 0.01... (8 Replies)
Discussion started by: evelibertine
8 Replies

8. UNIX for Dummies Questions & Answers

Comparing two files and dividing the values

Hi all, I am new to unix and I am trying hard to get this requirement, but no luck. I am trying to compare two cloumns in two files and if it matches, the last column in file1 must be divided by file2 and the output must be written in a new file. To elaborate the 2nd column in file1 (EUR) must be... (6 Replies)
Discussion started by: smadderla
6 Replies

9. Shell Programming and Scripting

Comparing 2 files and return the unique lines in first file

Hi, I have 2 files file1 ******** 01-05-09|java.xls| 02-05-08|c.txt| 08-01-09|perl.txt| 01-01-09|oracle.txt| ******** file2 ******** 01-02-09|windows.xls| 02-05-08|c.txt| 01-05-09|java.xls| 08-02-09|perl.txt| 01-01-09|oracle.txt| ******** (8 Replies)
Discussion started by: shekhar_v4
8 Replies

10. UNIX for Dummies Questions & Answers

Need to find only unique values for a given tag across the files

Need to find only unique values for a given tag across the files: For eg: Test1: <Tag1>aaa</Tag1> <Tag2>bbb</Tag2> <Tag3>ccc</Tag3> Test2: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test3: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test4: (8 Replies)
Discussion started by: sudheshnaiyer
8 Replies
Login or Register to Ask a Question