04-02-2007
Help comparing 2 files to find deleted records
Hi,
I need to compare todays file to yesterdays file to find deletes.
I cannot use comm -23 file.old file.new.
Because each record may have a small change in it but is not really a delete.
I have two delimited files. the first field in each file is static. All other fields may change. I want to know if the static data has disappeared from the file. If I use comm -23 I will get the lines that data in field 2 or 3 etc may have changed but the static data in filed 1 is the same.
I did search this site and found a reply to another post that I used to solve my problem. I wrote a smal script that does work. But the problem is it is very slow.
My old file has 7000 lines my new file has 4000 lines. I ran my script and 30 minutes later it still was not done. I ran it on a UNIX box that has a lot of memory and processors. It is not a hardware issue.
Any sugestions? Below is my code:
while read static
do
found="no"
data=`echo "$static" | cut -d'|' -f1`
while read line
do
echo "$line" | grep "$data" >/dev/null
if [ $? -eq 0 ]
then
found="yes"
break
fi
done < file.new
if [ $found = "no" ]
then
echo "$static"
fi
done < file.old
I am trying to keep all data on the deleted line. I could cut both files down filed 1 into smaller files and then run a comm -23 on the smaller files. But then I loose all data in the other fields.
Last edited by eja; 04-02-2007 at 09:36 PM..
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I need a perl script which will create an output file after comparing two diff file in a directory path:
/export/home/abc/file1
/export/home/abc/file2
File Format: <IP>TAB<DeviceName><TAB>DESCRIPTIONS
file1:
10.1.2.1.3<tab>abc123def<tab>xyz.mm1.ppp.... (2 Replies)
Discussion started by: ricky007
2 Replies
2. UNIX for Dummies Questions & Answers
Hi,
I want to use the sed command to delete some lines in a file and I was wondering whether there is a possibility of knowing which lines are deleted, or at least which line numbers.
Thanks (4 Replies)
Discussion started by: vanagreg
4 Replies
3. Shell Programming and Scripting
hi i have two files and i want to compare both the files and find out mismatch in 3rd file
file1
00354|1|0|1|1|0|0|0|1|2
52424|1|0|1|1|0|0|0|1|2
43236|1|0|1|1|0|0|0|1|2
41404|1|0|1|1|0|0|0|1|2
79968|1|0|1|1|0|0|0|1|2
file2
00354|1|0|1|1|0|0|0|1|2
52424|1|0|1|1|0|0|0|0|2... (9 Replies)
Discussion started by: dodasajan
9 Replies
4. Shell Programming and Scripting
I have a directory where lot of "gzip" files are dropped in every 5 minutes. There is an application which will process these gzip and move it to another directory but will leave a gzip.out file with lot of output data.
I need to remove all the outfiles except for the one which is being... (1 Reply)
Discussion started by: gubbu
1 Replies
5. Shell Programming and Scripting
Hello all,
Would like to know how to find records between two files which are not exists in one another in one. For example: I've two files "fileA" and "fileB" and want to find record from "fileB" which does not exists in "fileA".
fileA
--------
ABCD
DEFG
GHIJ
KLMN
NOPQ
RSTU
VUWX... (5 Replies)
Discussion started by: nvkuriseti
5 Replies
6. UNIX for Dummies Questions & Answers
Hi,
There were a few files deleted from a server by user xyz.
The file names are:-
/oraextME4/oradata/ME11G22/TEST_IMPORT_01.dbf
/oraextME4/oradata/ME11G22/RKVITR1_03.dbf
/oraextME4/oradata/ME11G22/TEST_IMPORT_02.dbf
need to know the ip address of the terminal from which that... (10 Replies)
Discussion started by: Abhinav Jaiswal
10 Replies
7. Shell Programming and Scripting
Hi All,
I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need.
File1.csv:
RAJAK,ACTIVE,1
VIJAY,ACTIVE,2
TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies
8. UNIX for Advanced & Expert Users
Hi Guys ,
we have one directory ...in that directory all files will be set on each day..
files must have header ,contents ,footer..
i wants to compare the header,contents,footer ..if its same means display an error message as 'files contents same' (7 Replies)
Discussion started by: Venkatesh1
7 Replies
9. Shell Programming and Scripting
Hello all,
Please help me for a script that compares two files and reads only those records that are to be inserted and updated.
File1:
c_id name place contact_no
1 abc xyz 34567
10 efg uvw 82725
6 hjk wth 01823
2 iuy ... (4 Replies)
Discussion started by: T@ni@
4 Replies
10. UNIX for Beginners Questions & Answers
I have two files:
File_1:
@M04961:22:000000000-B5VGJ:1:1101:9280:7106 1:N:0:86
GGCATGAAAACATACAAACCGTCTTTCCAGAAATTGTTCCAAGTATCGGCAACAGCTTTATCAATACCATGAAAAATATCAACCACACCAGAAGCAGCAT
+
GGGGGGGGGGGGGGGGGCCGGGGGF,EDFFGEDFG,@DGGCGGEGGG7DCGGGF68CGFFFGGGG@CGDGFFDFEFEFF:30CGAFFDFEFF8CAF;;8F
... (3 Replies)
Discussion started by: Xterra
3 Replies
diff3(1) General Commands Manual diff3(1)
Name
diff3 - 3-way differential file comparison
Syntax
diff3 [-ex3] file1 file2 file3
Description
The command compares three versions of a file, and publishes the ranges of text that disagree, flagged with the following codes:
==== all three files differ
====1 file1 is different
====2 file2 is different
====3 file3 is different
The type of change needed to convert a given range of a given file to some other is indicated in one of these ways:
f : n1 a Text is to be appended after line number n1 in file f, where f = 1, 2, or 3.
f : n1 , n2 c
Text is to be changed in the range line n1 to line n2. If n1 = n2, the range may be abbreviated to n1.
The original contents of the range follows immediately after a c indication. When the contents of two files are identical, the contents of
the lower-numbered file is suppressed.
Options
-3 Produces an editor script containing the changes between file1 and file2 that are to be incorporated into file3.
-e Produces an editor script containing the changes between file2 and file3 that are to be incorporated into file1.
-x Produces an editor script containing the changes among all three files.
Examples
Under the -e option, publishes a script for the editor that incorporates into file1 all changes between file2 and file3 - that is, the
changes that would normally be flagged ==== and ====3. Option -x (-3) produces a script to incorporate only changes flagged ==== (====3).
The following command applies the resulting script to `file1':
(cat script; echo '1,$p') | ed - file1
Restrictions
Text lines that consist of a single `.' defeat -e.
Files
/tmp/d3?????
/usr/lib/diff3
See Also
cmp(1), comm(1), diff(1), dffmk(1), join(1), sccsdiff(1), uniq(1)
diff3(1)