10-22-2012
Diff on 1gb files
Hey Guys,
I have a scenario to compare two different files which are of size 1gb each. I need to get the uncommon lines. I planned to use sdiff command, which generally works perfect for me. But in this case am facing a error saying
"diff: memory exhausted"
Can anyone please explain this.
Also, can anyone suggest a unix command to compare huge files, to get the common and the uncommon lines between the two files.
Thanks,
Abhishek S.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello,
I want to compare two files. All records in file 2 that are not in file 1 should be output to file 3.
For example:
file 1
123
1234
123456
file 2
123
2345
23456
file 3 should have
2345
23456
I have looked at diff, bdiff, cmp, comm, diff3 without any luck! (2 Replies)
Discussion started by: blt123
2 Replies
2. UNIX for Advanced & Expert Users
Hi,
I have to do a search on a zip files whose sizes vary from 1GB to 1.5GB.
I dont want to unzip it since if it goes beyond 2GB.......
also will unzip -p filename | grep create any problems, will it unzip the whole file or will it unzip it piece by piece??
I appreciate your inputs...
... (1 Reply)
Discussion started by: baanprog
1 Replies
3. Shell Programming and Scripting
I need to compare 2 diff type of files and find out the duplicate after comparing each types of files:
Type 1 file name is like: file1.abc
(the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations).
The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies
4. Shell Programming and Scripting
Hi Masters,
I have two files named file1 and file2.
Both the files contains the same contents with some difference in comments,space.But no content change.
I tried to find the diff between the two files to make sure that contents are same.
For that i tried
diff -ibw file1 file2
But... (1 Reply)
Discussion started by: ecearund
1 Replies
5. Red Hat
Hi, I have a Linux distribution ( Oralce Enterprise Linux 5.3 i.e. Redhat ) that I have installed. It works fine when I used 2*512Mb dimms or replace them with a single 1Gb dimm. However when I try to go above 1 Gb the bootup and general performance deteriorates badly. The BIOS picks up the memory... (3 Replies)
Discussion started by: jimthompson
3 Replies
6. Shell Programming and Scripting
Hi,
I have 2 files.I want to check if file1 is contained in file2.
A.txt:
-----
AAA
BBB
B.txt:
------
CCC
AAA
BBB
DDD
I want to check if A.txt is contained in B.txt. Can it be done using SED ? (12 Replies)
Discussion started by: giri_luck
12 Replies
7. Shell Programming and Scripting
Hi All,
I have two files which look as below
File1
serial="1" name="abc" type="employee" field="IT"
serial="2" name="cde" type="intern" field="Marketing"
serial="3" name="pqr" type="contractor" field="IT"
serial="4" name="xyz" type="employee" field="Sales"
File2
serial="1"... (3 Replies)
Discussion started by: grajp002
3 Replies
8. Shell Programming and Scripting
Moderator, please, delete this topic (1 Reply)
Discussion started by: optik77
1 Replies
9. UNIX for Dummies Questions & Answers
Hi All,
I know the separate commands for finding files greater than 30 days and finding files greater than 1GB.
How do I combine these two commands?
Meaning how do I find files which are > 1GB and older than 30 days?
;) (4 Replies)
Discussion started by: Hangman2
4 Replies
10. Shell Programming and Scripting
Guys i have 3 files,
but i want to compare and diff only the 2nd column
path=`/home/whois/doms`
for i in `cat domain.tx`
do
whois $i| sed -n '/Registry Registrant ID:/,/Registrant Email:/p' > $path/$i.registrant
whois $i| sed -n '/Registry Admin ID:/,/Admin Email:/p' > $path/$i.admin... (10 Replies)
Discussion started by: kenshinhimura
10 Replies
sdiff(1) General Commands Manual sdiff(1)
NAME
sdiff - Compares two files and displays the differences in a side-by-side format
SYNOPSIS
sdiff [-l | -s] [-w number] [-o output_file] file1 file2
The sdiff command reads file1 and file2, uses diff to compare them, and writes the results to standard output in a side-by-side format.
OPTIONS
Displays only the left side when lines are identical. Creates a third file, output_file, by a controlled interactive line-by-line merging
of file1 and file2. The following subcommands govern the creation of this file: Adds the left side to output_file. Adds the right side to
output_file. Stops displaying identical lines. Begins displaying identical lines. Enters ed with the left side, the right side, both
sides, or an empty file, respectively.
Each time you exit from ed, sdiff writes the resulting edited file to the end of output_file. If you fail to save the changes
before exiting, sdiff writes the initial input to output_file. Exits the interactive session. Suppresses display of identical
lines. Sets the width of the output line to number (130 characters by default).
DESCRIPTION
The sdiff command displays each line of the two files with a series of spaces between them if the lines are identical, a < (left angle
bracket) in the field of spaces if the line only exists in file1, a > (right angle bracket) if the line only exists in file2, and a | (ver-
tical bar) for lines that are different.
When you specify the -o option, sdiff produces a third file by merging file1 and file2 according to your instructions.
Note that the sdiff command invokes the diff -b command to compare two input files. The -b option causes the diff command to ignore trail-
ing spaces, tab characters, and consider other strings of spaces as equal.
EXAMPLES
To print a comparison of two files, enter: sdiff chap1.bak chap1
This displays a side-by-side listing that compares each line of chap1.bak and chap1. To display only the lines that differ, enter:
sdiff -s -w 80 chap1.bak chap1
This displays the differences at the tty. The -w 80 sets page width to 80 columns. The -s option tells sdiff not to display lines
that are identical in both files. To selectively combine parts of two files, enter: sdiff -s -w 80 -o chap1.combo chap1.bak
chap1
This combines chap1.bak and chap1 into a new file called chap1.combo. For each group of differing lines, sdiff asks you which group
to keep or whether you want to edit them using ed.
SEE ALSO
Commands: diff(1), ed(1)
sdiff(1)