Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Comparing two files to get only records to be inserted and updated

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Comparing two files to get only records to be inserted and updated
# 1  
Old 12-02-2013
Comparing two files to get only records to be inserted and updated
Hello all,

Please help me for a script that compares two files and reads only those records that are to be inserted and updated.
File1:
Code:
c_id    name   place   contact_no
1         abc       xyz      34567
10       efg        uvw     82725
6         hjk        wth      01823
2         iuy       wqa      9876

File2:
Code:
c_id    name   place   contact_no
1         abc       xyz      34567
10       efg        uvw     77777
6         zzz        wth      01823
2         iuy       wqa      9876
3        axy         cde        3837

I want to create a file3 that would contain
Code:
c_id    name   place   contact_no
10       efg        uvw     77777
6         zzz        wth      01823
3        axy         cde        3837


I have used the command:
Code:
comm -13 <(sort file1) <(sort file2) >file3

But somehow null records are coming to file3 which is preventing its loading in database.And the header is appended in the last line of file3.

Can anyone please help me fixing the problem.
Last edited by Don Cragun; 12-02-2013 at 05:30 AM.. Reason: Add CODE tags.
# 2  
Old 12-02-2013
you can try the below

Code:
for i in `cat file2`
do
grep "$i" file1 >> file3
done

# 3  
Old 12-02-2013
When I tried zozoo's script, it didn't come close to producing the requested output. (It is grepping for matching words instead of matching lines.) I think the following comes closer to what was requested:
Code:
awk '
FNR == NR {
        if(NR > 1) f1[$0]
        next
}
!($0 in f1) {
        print
}' file1 file2 > file3

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.
# 4  
Old 12-02-2013
Quote:
Originally Posted by Don Cragun
When I tried zozoo's script, it didn't come close to producing the requested output. (It is grepping for matching words instead of matching lines.) I think the following comes closer to what was requested:
Code:
awk '
FNR == NR {
        if(NR > 1) f1[$0]
        next
}
!($0 in f1) {
        print
}' file1 file2 > file3

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.

thanks Don Cragun for trying and poiting out the error. if you use
Code:
while

loop and
Code:
grep

combination then we can achieve the desired result.
# 5  
Old 12-02-2013
Quote:
Originally Posted by zozoo
thanks Don Cragun for trying and poiting out the error. if you use
Code:
while

loop and
Code:
grep

combination then we can achieve the desired result.
Hi zozoo,
If we make some assumptions about the input that the original poster did not specify, then it might be possible to write a while read loop using grep to get the output specified in the 1st message in this thread, but I don't think it is as easy as you're implying.

Can you show us a while loop using grep that produces the output requested in the 1st message in this thread from the two input files shown in that message?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Comparing fastq files and outputting common records

I have two files: File_1: @M04961:22:000000000-B5VGJ:1:1101:9280:7106 1:N:0:86 GGCATGAAAACATACAAACCGTCTTTCCAGAAATTGTTCCAAGTATCGGCAACAGCTTTATCAATACCATGAAAAATATCAACCACACCAGAAGCAGCAT + GGGGGGGGGGGGGGGGGCCGGGGGF,EDFFGEDFG,@DGGCGGEGGG7DCGGGF68CGFFFGGGG@CGDGFFDFEFEFF:30CGAFFDFEFF8CAF;;8F ... (3 Replies)
Discussion started by: Xterra
3 Replies

2. UNIX for Beginners Questions & Answers

Date from filename inserted into records

Hi Folks, I have below files in one directory: Spiross-MBP:AIRTEMP spirospap$ ls -1 CPK2004001 CPK2004002 CPK2004003 etc... JFK2003001 JFK2003002 JFK2003003 etc... TEB1999001 TEB1999002 TEB1999003 etc... Month/year is in Filename and also in the file Header, first line. (18 Replies)
Discussion started by: spirospap
18 Replies

3. Shell Programming and Scripting

Problem in shell script db2 query for fetching and comparing records

hi.. my requirement is following: i want to have a shell script that queries a table i.e. fcm_auth in database fcmcore (I use db2) to get no of rows present for a given combination eg org and logo for past 5-10 mins interval. You can take some values of org(numerical values) and corresponding... (2 Replies)
Discussion started by: nishantrefound
2 Replies

4. Shell Programming and Scripting

Updated files using awk

If third column contains null value then it should delete the row and i want to know the files which contains null value in 3rd column.Below code is not showing the updated files correctly..not sure what the issue is.plz help me #!/usr/bin/sh Scripts=/ushhquest/data001/Scripts cd... (5 Replies)
Discussion started by: katakamvivek
5 Replies

5. Shell Programming and Scripting

Compare two files with different number of records and output only the Extra records from file1

Hi Freinds , I have 2 files . File 1 |nag|HYd|1|Che |esw|Gun|2|hyd |pra|bhe|3|hyd |omu|hei|4|bnsj |uer|oeri|5|uery File 2 |nag|HYd|1|Che |esw|Gun|2|hyd |uer|oi|3|uery output : (9 Replies)
Discussion started by: i150371485
9 Replies

6. Shell Programming and Scripting

Show only new and removed records by comparing to MySQL tables

Hello all; I have been really frustrated with finding the correct perl code (and MySql statements) to accomplish what I thought was straight forward...I have tested I don't know how many different codes\suggestions I found on the net without any success...but anyhow let me explain my plight and... (0 Replies)
Discussion started by: gvolpini
0 Replies

7. Shell Programming and Scripting

removing duplicate records comparing 2 csv files

Hi All, I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need. File1.csv: RAJAK,ACTIVE,1 VIJAY,ACTIVE,2 TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies

8. Shell Programming and Scripting

comparing part of header with part of detailed records.

Hi there, I am lil confused with the following issue. I have a File, which has the following header: IMSHRATE_043008_101016 a sample detailed record is :9820101 A982005000CAVG030108000000000000010169000MAR 2008 9820102 MAR 2008 D030108 ... (1 Reply)
Discussion started by: cmaroju
1 Replies

9. UNIX for Dummies Questions & Answers

Output number of rows inserted, updated...

Hi all, I am new to Unix. I have written pl/sql script to be run in Unix. I have used Merge statement and subsequently would like to know the number of rows updated or inserted. Any suggestions in this regard would be great Thanks in advance Kushal (0 Replies)
Discussion started by: kushal_cog
0 Replies

10. UNIX for Dummies Questions & Answers

Help comparing 2 files to find deleted records

Hi, I need to compare todays file to yesterdays file to find deletes. I cannot use comm -23 file.old file.new. Because each record may have a small change in it but is not really a delete. I have two delimited files. the first field in each file is static. All other fields may change. I... (2 Replies)
Discussion started by: eja
2 Replies
Login or Register to Ask a Question