Sponsored Content
Full Discussion: Huge File Comparison
Top Forums Shell Programming and Scripting Huge File Comparison Post 302396157 by naveenn08 on Wednesday 17th of February 2010 11:52:46 PM
Old 02-18-2010
Bug Huge File Comparison

Hi i need to compare two fixed length files and produce the differences if any to a seperate file. I have to capture each and every differneces line by line. Ideally my files should not have any differences but if there are any then it should be captured without any miss. Also my files sizes are very huge say more than 2 GB.

Please help me with a code in either awk or shell script which does this huge file comparison with increase in some performance.

Regards,
Naveen
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

spliting up a huge file

I have a file {filename} which contains 65000 records I need to split into 6 smaller files roughly 11000 records each. Can someone advise me of the Unix command to do so ? Many thanks (2 Replies)
Discussion started by: grinder182533
2 Replies

2. Solaris

compare huge file

Hi, I have files with records of 40,00,000& 39,00,000 and i want to find out the content 1.which is existing in file1 and not in file2. 2.Which is exisitng in file2 and not in file1. The format of the file will be like 404ABCDEFGHIJK|CDEFGHIJK|1234567890|1 If its a smaller one i... (1 Reply)
Discussion started by: salaathi
1 Replies

3. Shell Programming and Scripting

sorting huge file

Hi All I am sorting a huge file -rw-r--r-- 1 rama users 448156978 May 13 18:48 102384.temp $ sort -k 1,40n 102384.temp > 102384.temp1 msgcnt 1468 vxfs: mesg 001: vx_nospace - /dev/vg00/var file system full (1 block extent) sort: A write error occurred while sorting. I thought... (3 Replies)
Discussion started by: dhanamurthy
3 Replies

4. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

5. Shell Programming and Scripting

Help on splitting this huge file

Hi , i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character. I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies

6. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

7. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

8. UNIX for Dummies Questions & Answers

File comparison of huge files

Hi all, I hope you are well. I am very happy to see your contribution. I am eager to become part of it. I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies

9. UNIX for Dummies Questions & Answers

My file system is 100%, can't find the huge file

Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now. I did to this to find the big file but it's taking so much time. Is there any other way? du -ah / | more find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies
CMP(1)							    BSD General Commands Manual 						    CMP(1)

NAME
cmp -- compare two files SYNOPSIS
cmp [-l | -s | -x] [-hz] file1 file2 [skip1 [skip2]] DESCRIPTION
The cmp utility compares two files of any type and writes the results to the standard output. By default, cmp is silent if the files are the same; if they differ, the byte and line number at which the first difference occurred is reported. Bytes and lines are numbered beginning with one. The following options are available: -h Do not follow symbolic links. -l Print the byte number (decimal) and the differing byte values (octal) for each difference. -s Print nothing for differing files; return exit status only. -x Like -l but prints in hexadecimal and using zero as index for the first byte in the files. -z For regular files compare file sizes first, and fail the comparison if they are not equal. The optional arguments skip1 and skip2 are the byte offsets from the beginning of file1 and file2, respectively, where the comparison will begin. The offset is decimal by default, but may be expressed as a hexadecimal or octal value by preceding it with a leading ``0x'' or ``0''. EXIT STATUS
The cmp utility exits with one of the following values: 0 The files are identical. 1 The files are different; this includes the case where one file is identical to the first part of the other. In the latter case, if the -s option has not been specified, cmp writes to standard error that EOF was reached in the shorter file (before any differences were found). >1 An error occurred. SEE ALSO
diff(1), diff3(1) STANDARDS
The cmp utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. The -h, -x, and -z options are extensions to the standard. HISTORY
A cmp command appeared in Version 1 AT&T UNIX. BSD
November 18, 2013 BSD
All times are GMT -4. The time now is 11:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy