Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Q: Howto compare 2 files
Top Forums UNIX for Dummies Questions & Answers Q: Howto compare 2 files Post 302695759 by Don Cragun on Monday 3rd of September 2012 07:42:08 PM
Old 09-03-2012
Do you only want to find lines that are in file1 that are not in file2, or do you also want to find lines that are in file2 that are not in file1?

---------- Post updated at 04:42 PM ---------- Previous update was at 04:00 PM ----------

If you can't perform a select as Scott suggested, there are at least a couple of fairly straight-forward ways of handling this. The easiest may well be the best:
Code:
sort -n File1 > sFile1
sort -n File2 > sFile2
diff sFile[12]

although with 15,000,000 lines you may have to read the man page for your sort utility to find out how to specify a file system with enough space for the temporary files that will be required.

Another way would be to create an associative array in awk of the entries found in File2 and then read the entries in File1 and report entries that aren't in the array you built while reading File2. (If you need the entries in File2 that aren't in File1 as well, you could mark or delete the entries you found on the first pass and make another pass to print the entries that weren't matched.)
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Howto Archive Including Hidden Files?

Hi I want to archive the following all the files and directory like listed below: $ ls -a . .. .bash_history .bash_logout .bash_profile .bashrc .emacs .mysql_history public_html .viminfo What I tried is to use the following command $ gtar cvzf allmyfiles.tar.gz * ... (1 Reply)
Discussion started by: monkfan
1 Replies

2. UNIX for Dummies Questions & Answers

Howto removing files with the same inode

Dear all, # ls -li total 16 2623392 drwxrwxrwx 2 root root 512 Apr 10 01:57 10HPA- 8447490 drwxr-xr-x 3 root root 512 Apr 14 05:29 118OQ- 8447490 drwxr-xr-x 3 root root 512 Apr 14 05:29 118OQ-.old 1925572 drwxrwxrwx 2 root root 512... (3 Replies)
Discussion started by: fu4d
3 Replies

3. Shell Programming and Scripting

compare files in two directories and output changed files to third directory

I have searched about 30 threads, a load of Google pages and cannot find what I am looking for. I have some of the parts but not the whole. I cannot seem to get the puzzle fit together. I have three folders, two of which contain different versions of multiple files, dist/file1.php dist/file2.php... (4 Replies)
Discussion started by: bkeep
4 Replies

4. Shell Programming and Scripting

How to compare 2 files & get only few columns based on a condition related to both files?

Hiiiii friends I have 2 files which contains huge data & few lines of it are as shown below File1: b.dat(which has 21 columns) SSR 1976 8 12 13 10 44.00 39.0700 70.7800 7.0 0 0.00 0 2.78 0.00 0.00 0 0.00 2.78 0 NULL ISC 1976 8 12 22 32 37.39 36.2942 70.7338... (6 Replies)
Discussion started by: reva
6 Replies

5. Shell Programming and Scripting

HowTo translate KSH Scripts to DOS Batch Files ?

Hi there, in near future I have to change my work surrounding from HP UNIX to Windows Vista (great to get rid of old hardware :), but bad to loose UNIX :( ). As I heavily use KSH scripts to do my job, I was wondering, if there is any HowTo available, supporting me in re-writing the scripts to... (4 Replies)
Discussion started by: Joe-K7
4 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Require compare command to compare 4 files

I have four files, I need to compare these files together. As such i know "sdiff and comm" commands but these commands compare 2 files together. If I use sdiff command then i have to compare each file with other which will increase the codes. Please suggest if you know some commands whcih can... (6 Replies)
Discussion started by: nehashine
6 Replies

8. Shell Programming and Scripting

howto monitor a directory for files then sftp them

Morning all I hope I have put this in the correct forum. I have a requirement to monitor a directory on a server for files being sftp'ed in and then to sftp them of to another server. The issues I have though of are making sure the files have completely transferred onto the server before they... (6 Replies)
Discussion started by: ltodd2
6 Replies

9. Shell Programming and Scripting

Compare multiple files, and extract items that are common to ALL files only

I have this code awk 'NR==FNR{a=$1;next} a' file1 file2 which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones... (7 Replies)
Discussion started by: castrojc
7 Replies

10. Shell Programming and Scripting

Howto compare the columns of 2 diff tables of 2 different schemas in UNIX shell script

HI All, I am new to Unix shell scripts.. Could you please post the unix shell script for for the below request., There are two different tables(sample1, sample2) in different schemas(s_schema1, s_schema2). Unix shell script to compare the columns of two different tables of two... (2 Replies)
Discussion started by: Rajkumar Gopal
2 Replies

LEARN ABOUT DEBIAN

btarcmp

BTARCMP(1)						      General Commands Manual							BTARCMP(1)

NAME

       btarcmp - a diff for Barry tar backup files

SYNOPSIS

       btarcmp [-b][-d db][-D db][-h][-I charset][-P][-S][-v] tar0 tar1

DESCRIPTION

       btarcmp	takes  two  Barry  backup  tar	files  on the command line, and displays the differences it finds between them.  If records can be
       parsed, individual field differences are also displayed.  Added and removed records are noted.  Differences in  database  availability  are
       noted as well.  If unable to parse the records, hex dumps are used.

OPTIONS

       -b     Use brief filename markers.  Useful when tar filenames are extremely long.

       -d db  Name  of	database  to compare.  Can be used multiple times to compare multiple databases.  If not specified, all databases are com-
	      pared.

       -D db  Name of database to skip.  Can be used multiple times.  Overrides all other database selection options.

       -h     Show summary of options.

       -I charset
	      Specifies the iconv charset to use for converting international strings.	The Blackberry uses the  WINDOWS-1252  charset,  which	is
	      incompatible with the more common code pages used in Linux.  The most useful charset to use with this option is UTF-8, and is highly
	      recommended.  Any other charset available via 'iconv --list' can be used here too, but may not be successful for some character con-
	      versions.

       -S     Displays list of known database records, which can be parsed.

       -v     Verbose output, which includes record data of added and deleted records in the output.  If used twice, hex data is printed as well.

AUTHOR

       btarcmp is part of the Barry project.

SEE ALSO

       http://www.netdirect.ca/barry

								 January 20, 2010							BTARCMP(1)
All times are GMT -4. The time now is 03:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy