Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Huge File Comparison
Top Forums Shell Programming and Scripting Huge File Comparison Post 302396750 by methyl on Friday 19th of February 2010 07:40:15 AM
Old 02-19-2010
If the files are normally the same I'd run the checksum program "cksum" on each file first and compare the results. This is the quickest way to prove whether two files are identical.
If they are not identical then actually run a command to compare them.

Whether you can use any standard unix commands to compare files which are larger than 2Gb depends on your Operating System and version.

We seem to be assuming that these are unix standard text files. Are they text files? If not, what software was used to create the files?
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

spliting up a huge file

I have a file {filename} which contains 65000 records I need to split into 6 smaller files roughly 11000 records each. Can someone advise me of the Unix command to do so ? Many thanks (2 Replies)
Discussion started by: grinder182533
2 Replies

2. Solaris

compare huge file

Hi, I have files with records of 40,00,000& 39,00,000 and i want to find out the content 1.which is existing in file1 and not in file2. 2.Which is exisitng in file2 and not in file1. The format of the file will be like 404ABCDEFGHIJK|CDEFGHIJK|1234567890|1 If its a smaller one i... (1 Reply)
Discussion started by: salaathi
1 Replies

3. Shell Programming and Scripting

sorting huge file

Hi All I am sorting a huge file -rw-r--r-- 1 rama users 448156978 May 13 18:48 102384.temp $ sort -k 1,40n 102384.temp > 102384.temp1 msgcnt 1468 vxfs: mesg 001: vx_nospace - /dev/vg00/var file system full (1 block extent) sort: A write error occurred while sorting. I thought... (3 Replies)
Discussion started by: dhanamurthy
3 Replies

4. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

5. Shell Programming and Scripting

Help on splitting this huge file

Hi , i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character. I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies

6. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

7. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

8. UNIX for Dummies Questions & Answers

File comparison of huge files

Hi all, I hope you are well. I am very happy to see your contribution. I am eager to become part of it. I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies

9. UNIX for Dummies Questions & Answers

My file system is 100%, can't find the huge file

Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now. I did to this to find the big file but it's taking so much time. Is there any other way? du -ah / | more find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies

LEARN ABOUT REDHAT

tiffcmp

TIFFCMP(1)						      General Commands Manual							TIFFCMP(1)

NAME

       tiffcmp - compare two TIFF files

SYNOPSIS

       tiffcmp [ options ] file1.tif file2.tif

DESCRIPTION

       Tiffcmp compares the tags and data in two files created according to the Tagged Image File Format, Revision 6.0.  The schemes used for com-
       pressing data in each file are immaterial when data are compared-data are compared on a	scanline-by-scanline  basis  after  decompression.
       Most  directory	tags are checked; notable exceptions are: GrayResponseCurve, ColorResponseCurve, and ColorMap tags.  Data will not be com-
       pared if any of the BitsPerSample, SamplesPerPixel, or ImageWidth values are not equal.	By default, tiffcmp will terminate if  it  encoun-
       ters any difference.

OPTIONS

       -l     List each byte of image data that differs between the files.

       -t     Ignore any differences in directory tags.

BUGS

       Tags that are not recognized by the library are not compared; they may also generate spurious diagnostics.

       The image data of tiled files is not compared, since the TIFFReadScanline() function is used.  A error will be reported for tiled files.

       The pixel and/or sample number reported in differences may be off in some exotic cases.

SEE ALSO

       pal2rgb(1), tiffinfo(1), tiffcp(1), tiffmedian(1), libtiff(3)

								September 26, 1994							TIFFCMP(1)
All times are GMT -4. The time now is 02:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy