Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Compare two files in Linux

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare two files in Linux
# 1  
Old 04-02-2015
Compare two files in Linux
Hi,


I want to compare two files and put difference into another file.

file1 having 84M records and file2 having 85M records. I tried the below command to get the difference and it is not showing the output after sometime i killed the command.

Code:
awk -F"," 'FNR==NR{a[$1$2]++;next}!a[$1$2]' file1.txt file2.txt > difffile.txt

Is there anyway to find the difference fast. Please let me know.


Thx in advance

Moderator's Comments:
Mod Comment Use code tags, thanks.
Last edited by zaxxon; 04-02-2015 at 08:22 AM..
# 2  
Old 04-02-2015
Did you try to parse the output of diff?
# 3  
Old 04-02-2015
I also tried using diff and i got the out of memory error. Could you please let me know how can we compare large files and put it into a difference in another file.


Thx,
# 4  
Old 04-02-2015
Even with the "diff --speed-large-files" option?
# 5  
Old 04-02-2015
You can also try comm

for rows in file1 but not in file2, you can do

Code:
$ comm -23 <(sort file1) <(sort file2)

and similarly to get the rows in file2 but not in file1

Code:
$ comm -13 <(sort file1) <(sort file2)


If you run out of memory with the sort , try -T option.


Using awk

Code:
awk 'NR==FNR{a[$0];next} ! $0 in a{print $0}' file1 file2

will give you rows in file2 that are not in file1

Also a vim command should work, though I`ve never used it

Code:
vim -d file1 file2

Last edited by senhia83; 04-02-2015 at 11:18 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Compare two big files for differences using Linux

Hello everybody Looking for help in comparing two files in Linux(files are big 800MB each). Example:- File1 has below data $ cat file1 5,6,3 2.1.4 1,1,1 8,9,1 File2 has below data $ cat file2 5,6,3 8,9,8 1,2,1 2,1,4 (8 Replies)
Discussion started by: shanul karim
8 Replies

2. Shell Programming and Scripting

Compare two big files for differences using Linux

Hello everybody Looking for help in comparing two files in Linux(files are big 800MB each). Example:- File1 has below data $ cat file1 5,6,3 2.1.4 1,1,1 8,9,1 File2 has below data $ cat file2 5,6,3 8,9,8 1,2,1 2,1,4 (1 Reply)
Discussion started by: shanul karim
1 Replies

3. Shell Programming and Scripting

Compare multiple files, and extract items that are common to ALL files only

I have this code awk 'NR==FNR{a=$1;next} a' file1 file2 which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones... (7 Replies)
Discussion started by: castrojc
7 Replies

4. Shell Programming and Scripting

Linux Script to compare two folders and copy missing files

Hi, I need help in shell scripting. If someone can help me, that would be great! Problem. I want Linux Script to compare two folders and copy missing files. Description. I have two directories /dir1 /dir2 I need to copy all distinct/new/unique/missing files from /dir1 and that... (1 Reply)
Discussion started by: S.Praveen Kumar
1 Replies

5. Shell Programming and Scripting

LINUX - How to compare the values in 2 files & exit from the script

Hi All, I have a requirement where I need to compare 2 files & if the values in the files match, it should proceed, else exit the script without proceeding further. For e.g : Scenario 1 In this case, the script should exit without proceeding further. Scenario 2 In this case, the script... (7 Replies)
Discussion started by: dsfreddie
7 Replies

6. Shell Programming and Scripting

Require compare command to compare 4 files

I have four files, I need to compare these files together. As such i know "sdiff and comm" commands but these commands compare 2 files together. If I use sdiff command then i have to compare each file with other which will increase the codes. Please suggest if you know some commands whcih can... (6 Replies)
Discussion started by: nehashine
6 Replies

7. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

8. Shell Programming and Scripting

How to compare 2 files & get only few columns based on a condition related to both files?

Hiiiii friends I have 2 files which contains huge data & few lines of it are as shown below File1: b.dat(which has 21 columns) SSR 1976 8 12 13 10 44.00 39.0700 70.7800 7.0 0 0.00 0 2.78 0.00 0.00 0 0.00 2.78 0 NULL ISC 1976 8 12 22 32 37.39 36.2942 70.7338... (6 Replies)
Discussion started by: reva
6 Replies

9. Shell Programming and Scripting

compare files in two directories and output changed files to third directory

I have searched about 30 threads, a load of Google pages and cannot find what I am looking for. I have some of the parts but not the whole. I cannot seem to get the puzzle fit together. I have three folders, two of which contain different versions of multiple files, dist/file1.php dist/file2.php... (4 Replies)
Discussion started by: bkeep
4 Replies

10. Linux

compare files software on linux

Hi all, could you please recommend a good software to compare files? I prefer graphical one. Many thanks, Lan (1 Reply)
Discussion started by: lanchen
1 Replies
Login or Register to Ask a Question