11-20-2008
Compare 2 Large files
Hi,
i want to compare 2 large files both around 300 mb. They are text files having only one column of numbers. I wish to get the unique values in file2. I tried using diff but it gave an error of memory exhausted. Both files are sorted and i am running on a 1gb ram core 2 duo 2ghz. Help!!
Thanks in advance
AC
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I am trying to understand the webserver log file for an error which has occured on my live web site.
The webserver access file is very big in size so it's not possible to open this file using vi editor. I know the approximate time the error occured, so i am interested in looking for the log file... (4 Replies)
Discussion started by: sehgalniraj
4 Replies
2. UNIX for Dummies Questions & Answers
The windows version of gzip supports pretty much unlimited file sizes while the one we have in solaris only goes up to a set size, one or two gigs I think.
Is there a new version of gzip I can put on our systems that supports massive file sizes? (2 Replies)
Discussion started by: LordJezo
2 Replies
3. UNIX for Dummies Questions & Answers
How do we check 'large files' is enabled on a Unix box -- HP-UX B11.11 (2 Replies)
Discussion started by: ranj@chn
2 Replies
4. Shell Programming and Scripting
I have a large Filesystem on an AIX server and another one on a Red Hat box. I have syncd the two filesystems using rsysnc.
What Im looking for is a script that would compare to the two filesystems to make sure the bits match up and the number of files match up.
its around 2.8 million... (5 Replies)
Discussion started by: zippdawg2001
5 Replies
5. UNIX for Dummies Questions & Answers
I have two very large datasets (>100MB) in a simple vertical list format. They are of different size and with different order and formatting (e.g. whitespace and some other minor cruft that would thwart easy regex).
Let's call them set1 and set2.
I want to check set2 to see if it contains... (2 Replies)
Discussion started by: uiop44
2 Replies
6. Shell Programming and Scripting
Hello everyone!
I have 2 types of files in the following format:
1) *.fa
>1234
...some text...
>2345
...some text...
>3456
...some text...
.
.
.
.
2) *.info
>1234 (7 Replies)
Discussion started by: ad23
7 Replies
7. Shell Programming and Scripting
I have four files, I need to compare these files together.
As such i know "sdiff and comm" commands but these commands compare 2 files together. If I use sdiff command then i have to compare each file with other which will increase the codes.
Please suggest if you know some commands whcih can... (6 Replies)
Discussion started by: nehashine
6 Replies
8. Solaris
Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS.
I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies
9. Shell Programming and Scripting
I have a very large system generated file containing around 500K rows size 100MB like following
HOME|ALICE STREET|3||NEW LISTING
HOME|NEWPORT STREET|1||NEW LISTING
HOME|KING STREET|5||NEW LISTING
HOME|WINSOME AVENUE|4||MODIFICATION
CAR|TOYOTA|4||NEW LISTING
CAR|FORD|4||NEW... (9 Replies)
Discussion started by: jubaier
9 Replies
10. Shell Programming and Scripting
Hello, my first thread here.
I've been searching and fiddling around for about a week and I cannot find a solution.:confused:
I have been converting all of my home videos to HEVC and sometimes the files end up smaller and sometimes they don't. I am currently comparing all the video files... (5 Replies)
Discussion started by: Josh52180
5 Replies
BZDIFF(1) General Commands Manual BZDIFF(1)
NAME
bzcmp, bzdiff - compare bzip2 compressed files
SYNOPSIS
bzcmp [ cmp_options ] file1 [ file2 ]
bzdiff [ diff_options ] file1 [ file2 ]
DESCRIPTION
Bzcmp and bzdiff are used to invoke the cmp or the diff program on bzip2 compressed files. All options specified are passed directly to
cmp or diff. If only 1 file is specified, then the files compared are file1 and an uncompressed file1.bz2. If two files are specified,
then they are uncompressed if necessary and fed to cmp or diff. The exit status from cmp or diff is preserved.
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+--------------------+-----------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+--------------------+-----------------+
|Availability | SUNWbzip |
+--------------------+-----------------+
|Interface Stability | External |
+--------------------+-----------------+
SEE ALSO
cmp(1), diff(1), bzmore(1), bzless(1), bzgrep(1), bzip2(1)
BUGS
Messages from the cmp or diff programs refer to temporary filenames instead of those specified.
BZDIFF(1)