File Comparison - Shell Programming and Scripting

net_shree · #1 01-02-2008

I have to compare two text files, very few of the lines in these files will have some difference in some column.
The files size is in GB.
Sample lines are as below:
11111122222222333333aaaaaaaaaabbbbbbbbbccccccccdddddd
11111122222222333333aaaaaaaaaabbbbbbbbbccccccccddeddd

So assuming these two lines are from file1 and file2 respectively, I should get the second file line in a new output file which is the difference file.

What I would like to do is read line1 from file1 and loop through all the lines in file2 and stop when a match is found, else print it that line to output file. And repeat the same steps for all the lines from file1.

Appreciate any help in this regard.

jim mcnamara · #2 01-02-2008

What do you mean by "stop when a match is found", then you read more from file1....
Do you want the line number? Stop usually means to exit the read loop.

net_shree · #3 01-02-2008

Yes, I want to exit the read loop when a match is found, I do not want to check any more for that line.
No I do not need the line number.

dislusive · #4 01-02-2008

If I understand what you're trying to do correctly, here's a quick bash script.

Code:

#!/bin/bash

compareFile = "/path/to/file/to/compare.txt"
outputFile = "/path/to/outputFile.txt"

for filename in /some/dir/of/text/files/*.txt; do 
        
        numlines=`cat $filename | wc -l`
                
        for i in `seq 1 $numlines`; do 
                current=`cat $filename | head -$i | tail -1` 
 
                grep -q "${current}" ${compareFile} 
 
                if [ $? != 0 ]; then
                         #doesn't exist, append to $outputFile
                        echo "${filename}:${current}" >> ${outputFile} 
                fi
        done 
done

net_shree · #5 01-02-2008

Hi, Thank you for the quick solution and looks pretty much what I want.
But I am unable to run this script, I use ksh.
One of the errors is "seq: command not found"

dislusive · #6 01-02-2008

which seq (usually resides in /usr/bin/)

It's an individual executable command; should be part of the coreutils package if you're using linux.

if it exists on your system, modify the script
seq="/path/to/seq"

then modify the for statement to use the variable: for i in `${seq}...

ghostdog74 · #7 01-02-2008

Quote:

Originally Posted by dislusive

If I understand what you're trying to do correctly, here's a quick bash script.

Code:

#!/bin/bash

compareFile = "/path/to/file/to/compare.txt"
outputFile = "/path/to/outputFile.txt"

for filename in /some/dir/of/text/files/*.txt; do 
        
        numlines=`cat $filename | wc -l`
                
        for i in `seq 1 $numlines`; do 
                current=`cat $filename | head -$i | tail -1` 
 
                grep -q "${current}" ${compareFile} 
 
                if [ $? != 0 ]; then
                         #doesn't exist, append to $outputFile
                        echo "${filename}:${current}" >> ${outputFile} 
                fi
        done 
done

As mentioned by OP, the files are in GB. I think there will be some performance lag. just a guess.
Also seq is not a standard command in some *nix OS. Therefore if you want to use loops that loop over a counter, a while loop can be used instead. eg while [ $num -le $numlines ]

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
file comparison...help needed.	er_ashu	UNIX for Dummies Questions & Answers	4	05-15-2008 06:37 PM
Comparison Unix and Windows file sysytem	localp	UNIX for Dummies Questions & Answers	1	04-11-2008 01:02 AM
Output format - comparison with I/p file	velappangs	Shell Programming and Scripting	1	04-03-2008 03:31 AM
file comparison script	tiger99	Shell Programming and Scripting	1	01-30-2008 07:47 AM
File Time Comparison Question	pc9456	UNIX for Advanced & Expert Users	2	07-23-2003 12:05 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode