Sponsored Content
Top Forums Shell Programming and Scripting Find duplicate rows between files Post 302878763 by Selva_2507 on Sunday 8th of December 2013 09:52:17 PM
Old 12-08-2013
Find duplicate rows between files

Hi champs,

I have one of the requirement, where I need to compare two files line by line and ignore duplicates. Note, I hav files in sorted order.

I have tried using the comm command, but its not working for my scenario.

Input file1

Code:
srv1..development..employee..empname,empid,empdesg
srv1..enhancement..Team_Level..level1,level2,level3
srv0..testing..functional..test1,test2,test3
srv2..analog..analysis_tb..step1,step2,step3
srv2..Zoo..India..chennai,delhi,Agra

Input file2

Code:
srv1..development..employee..empname,empid,empdesg
srv1..enhancement..Team_Level..level1,level2,level3
srv2..analog..analysis_tb..step1,step2,step3
srv0..testing..functional..test1,test2,test3
srv2..Zoo..India..chennai,delhi,Agra

Using the command:

comm -13 Inputfile1 Inputfile2

Final output file which I am getting

Code:
srv1..development..employee..empname,empid,empdesg,empsalary
srv2..analog..analysis_tb..step1,step2,step3
srv0..testing..functional..test1,test2,test3
srv2..Zoo..India..chennai,delhi,Agra

But I need output as below:

Code:
srv1..development..employee..empname,empid,empdesg,empsalary

Note: need to ignore duplicate lines by checking these delimiters (.. & ,) irrespective of checking line by line.

Please help me here..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to find duplicate files with find ?

hello all I like to make search on files , and the result need to be the files that are duplicated? (8 Replies)
Discussion started by: umen
8 Replies

2. Shell Programming and Scripting

Find Duplicate files, not by name

I have a directory with images: -rw-r--r-- 1 root root 26216 Mar 19 21:00 020109.210001.jpg -rw-r--r-- 1 root root 21760 Mar 19 21:15 020109.211502.jpg -rw-r--r-- 1 root root 23144 Mar 19 21:30 020109.213002.jpg -rw-r--r-- 1 root root 31350 Mar 20 00:45 020109.004501.jpg -rw-r--r-- 1 root... (2 Replies)
Discussion started by: Ikon
2 Replies

3. Shell Programming and Scripting

Duplicate rows in CSV files based on values

I want to duplicate a row if found two or more values in a particular column for corresponding row which is delimitted by comma. Input abc,line one,value1 abc,line two, value1, value2 abc,line three,value1 needs to converted to abc,line one,value1 abc,line two, value1 abc,line... (8 Replies)
Discussion started by: Incrediblian
8 Replies

4. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

5. Ubuntu

delete duplicate rows with awk files

Hi every body I have some text file with a lots of duplicate rows like this: 165.179.568.197 154.893.836.174 242.473.396.153 165.179.568.197 165.179.568.197 165.179.568.197 154.893.836.174 how can I delete the repeated rows? Thanks Saeideh (2 Replies)
Discussion started by: sashtari
2 Replies

6. Shell Programming and Scripting

Duplicate rows in CSV files based on values

I am new to this forum and this is my first post. I am looking at an old post with exactly the same name. Can not paste URL because I do not have 5 posts My requirement is exactly opposite. I want to get rid of duplicate rows and try to append the values of columns in those rows ... (10 Replies)
Discussion started by: vbhonde11
10 Replies

7. Shell Programming and Scripting

find duplicate string in many different files

I have more than 100 files like this: SVEAVLTGPYGYT 2 SVEGNFEETQY 10 SVELGQGYEQY 28 SVERTGTGYT 6 SVGLADYNEQF 21 SVGQGYEQY 32 SVKTVLGYEQF 2 SVNNEQF 12 SVRDGLTNSPLH 3 SVRRDREGLEQF 11 SVRTSGSYEQY 17 SVSVSGSPLQETQY 78 SVVHSTSPEAF 59 SVVPGNGYT 75 (4 Replies)
Discussion started by: xshang
4 Replies

8. Shell Programming and Scripting

Find duplicate files but with different extensions

Hi ! I wonder if anyone can help on this : I have a directory: /xyz that has the following files: chsLog.107.20130603.gz chsLog.115.20130603 chsLog.111.20130603.gz chsLog.107.20130603 chsLog.115.20130603.gz As you ca see there are two files that are the same but only with a minor... (10 Replies)
Discussion started by: fretagi
10 Replies

9. Shell Programming and Scripting

Find help in shell - that clears away duplicate files

I am so frustrated!!! I want a nice command that clears away duplicate files: find . -type f -regex '.*{1,3}\..*' | xargs -I## rm -v '##' should work in my opinion. But it finds nothing even though I have files that have the file name: Scooby-Doo-1.txt Himalaya-2.jpg Camping... (8 Replies)
Discussion started by: Mr.Glaurung
8 Replies

10. UNIX for Advanced & Expert Users

AIX find duplicate backup files

I would like find and delete old backup files in aix. How would I go about doing this? For example: server1_1-20-2020 server1_1-21-2020 server1_1-22-2020 server1_1-23-2020 server2_1-20-2020 server2_1-21-2020 server2_1-22-2020 server2_1-23-2020 How would I go about finding and... (3 Replies)
Discussion started by: cokedude
3 Replies
WDMD(8) 						      System Manager's Manual							   WDMD(8)

NAME
wdmd - watchdog multiplexing daemon SYNOPSIS
wdmd [OPTIONS] DESCRIPTION
This daemon opens /dev/watchdog and allows multiple independent sources to detmermine whether each KEEPALIVE is done. Every test interval (10 seconds), the daemon tests each source. If any test fails, the KEEPALIVE is not done. In a standard configuration, the watchdog timer will reset the system if no KEEPALIVE is done for 60 seconds ("fire timeout"). This means that if single test fails 5-6 times in row, the watchdog will fire and reset the system. With multiple test sources, fewer separate failures back to back can also cause a reset, e.g. T seconds, P pass, F fail T00: test1 P, test2 P, test3 P: KEEPALIVE done T10: test1 F, test2 F, test3 P: KEEPALIVE skipped T20: test1 F, test2 P, test3 P: KEEPALIVE skipped T30: test1 P, test2 F, test3 P: KEEPALIVE skipped T40: test1 P, test2 P, test3 F: KEEPALIVE skipped T50: test1 F, test2 F, test3 P: KEEPALIVE skipped T60: test1 P, test2 F, test3 P: KEEPALIVE skipped T60: watchdog fires, system resets (Depending on timings, the system may be reset sometime shortly before T60, and the tests at T60 would not be run.) A crucial aspect to the design and function of wdmd is that if any single source does not pass tests for the fire timeout, the watchdog is guaranteed to fire, regardless of whether other sources on the system have passed or failed. A spurious reset due to the combined effects of multiple failing tests as shown above, is an accepted side effect. wdmd will exit if a watchdog driver is not loaded. wdmd cannot be used on the system with any other program that needs to open /dev/watchdog, e.g. watchdog(8). Test Source: clients Using libwdmd, programs connect to wdmd via a unix socket, and send regular messages to wdmd to update an expiry time for their connection. Every test interval, wdmd will check if the expiry time for a connection has been reached. If so, the test for that client fails. (Other test sources such as scripts executed each test interval may be added in the future.) OPTIONS
-D Enable debugging to stderr and don't fork. -H num Enable (1) or disable (0) high priority features such as realtime scheduling priority and mlockall. Default 1. 2011-08-01 WDMD(8)
All times are GMT -4. The time now is 11:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy