02-19-2010
If the files are normally the same I'd run the checksum program "cksum" on each file first and compare the results. This is the quickest way to prove whether two files are identical.
If they are not identical then actually run a command to compare them.
Whether you can use any standard unix commands to compare files which are larger than 2Gb depends on your Operating System and version.
We seem to be assuming that these are unix standard text files. Are they text files? If not, what software was used to create the files?
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have a file {filename} which contains 65000 records
I need to split into 6 smaller files roughly 11000 records each.
Can someone advise me of the Unix command to do so ?
Many thanks (2 Replies)
Discussion started by: grinder182533
2 Replies
2. Solaris
Hi,
I have files with records of 40,00,000& 39,00,000 and i want to find out the
content
1.which is existing in file1 and not in file2.
2.Which is exisitng in file2 and not in file1.
The format of the file will be like
404ABCDEFGHIJK|CDEFGHIJK|1234567890|1
If its a smaller one i... (1 Reply)
Discussion started by: salaathi
1 Replies
3. Shell Programming and Scripting
Hi All
I am sorting a huge file
-rw-r--r-- 1 rama users 448156978 May 13 18:48 102384.temp
$ sort -k 1,40n 102384.temp > 102384.temp1
msgcnt 1468 vxfs: mesg 001: vx_nospace - /dev/vg00/var file system full (1 block extent)
sort: A write error occurred while sorting.
I thought... (3 Replies)
Discussion started by: dhanamurthy
3 Replies
4. Shell Programming and Scripting
I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract
if i use sed, i have to redirect the output to a seperate file like
sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat
the same is true for awk
and... (10 Replies)
Discussion started by: deepaktanna
10 Replies
5. Shell Programming and Scripting
Hi ,
i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character.
I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies
6. Shell Programming and Scripting
I got three different file:
Part of File 1
ARTPHDFGAA
.
.
Part of File 2
ARTGHHYESA
.
.
Part of File 3
ARTPOLYWEA
.
. (4 Replies)
Discussion started by: patrick87
4 Replies
7. Shell Programming and Scripting
Hi Experts,
I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies
8. UNIX for Dummies Questions & Answers
Hi all,
I hope you are well. I am very happy to see your contribution. I am eager to become part of it.
I have the following question. I have two huge files to compare (almost 3GB each). The files are simulation outputs. The format of the files are as below
For clear picture, please see... (9 Replies)
Discussion started by: kaaliakahn
9 Replies
9. UNIX for Dummies Questions & Answers
Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now.
I did to this to find the big file but it's taking so much time. Is there any other way?
du -ah / | more
find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies
10. Solaris
Gents
I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server.
How to can I do that without changing anything in the source.
Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies
LEARN ABOUT OPENSOLARIS
cksum
cksum(1) User Commands cksum(1)
NAME
cksum - write file checksums and sizes
SYNOPSIS
cksum [file]...
DESCRIPTION
The cksum command calculates and writes to standard output a cyclic redundancy check (CRC) for each input file, and also writes to standard
output the number of octets in each file.
For each file processed successfully, cksum will write in the following format:
"%u %d %s
" <checksum>, <# of octets>, <path name>
If no file operand was specified, the path name and its leading space will be omitted.
The CRC used is based on the polynomial used for CRC error checking in the referenced Ethernet standard.
The encoding for the CRC checksum is defined by the generating polynomial:
G(x) = x^32 + x^26 + x^23 + x^22 + x^16 + x^12 + x^11 + x^10 + x^8 + x^7 + x^5 + x^4 + x^2 + x + 1
Mathematically, the CRC value corresponding to a given file is defined by the following procedure:
1. The n bits to be evaluated are considered to be the coefficients of a mod 2 polynomial M(x) of degree n-1. These n bits are the
bits from the file, with the most significant bit being the most significant bit of the first octet of the file and the last bit
being the least significant bit of the last octet, padded with zero bits (if necessary) to achieve an integral number of octets,
followed by one or more octets representing the length of the file as a binary value, least significant octet first. The small-
est number of octets capable of representing this integer is used.
2. M(x) is multiplied by x ^32 (that is, shifted left 32 bits) and divided by G(x) using mod 2 division, producing a remainder R(x)
of degree <= 31.
3. The coefficients of R(x) are considered to be a 32-bit sequence.
4. The bit sequence is complemented and the result is the CRC.
OPERANDS
The following operand is supported:
file A path name of a file to be checked. If no file operands are specified, the standard input is used.
USAGE
The cksum command is typically used to quickly compare a suspect file against a trusted version of the same, such as to ensure that files
transmitted over noisy media arrive intact. However, this comparison cannot be considered cryptographically secure. The chances of a dam-
aged file producing the same CRC as the original are astronomically small; deliberate deception is difficult, but probably not impossible.
Although input files to cksum can be any type, the results need not be what would be expected on character special device files. Since this
document does not specify the block size used when doing input, checksums of character special files need not process all of the data in
those files.
The algorithm is expressed in terms of a bitstream divided into octets. If a file is transmitted between two systems and undergoes any data
transformation (such as moving 8-bit characters into 9-bit bytes or changing "Little Endian" byte ordering to "Big Endian"), identical CRC
values cannot be expected. Implementations performing such transformations may extend cksum to handle such situations.
See largefile(5) for the description of the behavior of cksum when encountering files greater than or equal to 2 Gbyte ( 2^31 bytes).
ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of cksum: LANG, LC_ALL, LC_CTYPE, LC_MES-
SAGES, and NLSPATH.
EXIT STATUS
The following exit values are returned:
0 All files were processed successfully.
>0 An error occurred.
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
|Interface Stability |Standard |
+-----------------------------+-----------------------------+
SEE ALSO
digest(1), sum(1), bart(1M), attributes(5), environ(5), largefile(5), standards(5)
SunOS 5.11 1 Feb 1995 cksum(1)