Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Removing duplicates except the last occurrence
Top Forums Shell Programming and Scripting Removing duplicates except the last occurrence Post 302923902 by mechvijays on Wednesday 5th of November 2014 06:01:08 AM
Old 11-05-2014
Removing duplicates except the last occurrence
Hi All,

i have a file like below,
Code:
@DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql
@DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql
@DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql
@DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql
@DB_FCTS\src\Data\Scripts\Insert_CU_OM_BT_STMT_TYP.sql
@DB_FCTS\src\Data\Scripts\Insert_OM_BIL_T_ADDR.sql
@DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql
@DB_FCTS\src\Scripts\MC400_PreDb_Script.sql
@DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql
@DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql

==========================================
in the file,
line no 1 is repeated in the line 7 and
line no 3 is repeated in the line 9 and 10

my requirement is to remove the duplicate lines and keep only the last occurrence of it.

the output should be like below,
Code:
@DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql
@DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql
@DB_FCTS\src\Data\Scripts\Insert_CU_OM_BT_STMT_TYP.sql
@DB_FCTS\src\Data\Scripts\Insert_OM_BIL_T_ADDR.sql
@DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql
@DB_FCTS\src\Scripts\MC400_PreDb_Script.sql
@DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql

My env details,

SunOS sasbsd27c1 5.10 Generic_150400-10 sun4u sparc SUNW,SPARC-Enterprise

Please script a script to achieve this, i have been trying from morning, but nothing works out.

Thanks in advance
Last edited by vbe; 11-05-2014 at 11:27 AM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates

Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working. I've search the forum but nothing comes close. I have a sample file: 27147140631203RA CCD * 27147140631203RA PPN * 37147140631207RD AAA 47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies

2. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

3. Shell Programming and Scripting

removing duplicates

Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here ) I'm trying to write a scrip that will remove duplicate entries the script can... (5 Replies)
Discussion started by: stevie_velvet
5 Replies

4. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

5. UNIX for Advanced & Expert Users

removing duplicates.

Hi All In unix ,we have a file ,there we have to remove the duplicates by using one specific column. Can any body tell me the command. ex: file1 id,name 1,ww 2,qwq 2,asas 3,asa 4,asas 4,asas o/p: 1,ww 2,qwq 3,asa (7 Replies)
Discussion started by: raju4u
7 Replies

6. Shell Programming and Scripting

Removing duplicates

I have a test file with the following 2 columns: Col 1 | Col 2 T1 | 1 <= remove T5 | 1 T4 | 2 T1 | 3 T3 | 3 T4 | 1 <= remove T1 | 2 <= remove T3 ... (7 Replies)
Discussion started by: gctex
7 Replies

7. Emergency UNIX and Linux Support

Removing all the duplicates

i want to remove all the duplictaes in a file.I dont want even a single entry. For the input data: 12345|12|34 12345|13|23 3456|12|90 15670|12|13 12345|10|14 3456|12|13 i need the below data in one file 15670|12|13 and the below data in another file (9 Replies)
Discussion started by: pandeesh
9 Replies

8. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies

9. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies

LEARN ABOUT NETBSD

zfgrep

ZGREP(1)						    BSD General Commands Manual 						  ZGREP(1)

NAME

     zgrep, zegrep, zfgrep -- print lines matching a pattern in gzip-compressed files

SYNOPSIS

     zgrep [grep-flags] [--] pattern [files ...]

     zegrep [grep-flags] [--] pattern [file ...]

     zfgrep [grep-flags] [--] pattern [file ...]

DESCRIPTION

     zgrep runs grep(1) on files or stdin, if no files argument is given, after decompressing them with zcat(1).

     The grep-flags and pattern arguments are passed on to grep(1).  If an -e flag is found in the grep-flags, zgrep will not look for a pattern
     argument.

     zegrep calls egrep(1), while zfgrep calls fgrep(1).

EXIT STATUS

     In case of missing arguments or missing pattern, 1 will be returned, otherwise 0.

SEE ALSO

     egrep(1), fgrep(1), grep(1), gzip(1), zcat(1)

AUTHORS

     Thomas Klausner <wiz@NetBSD.org>

BSD
								 December 28, 2003							       BSD
All times are GMT -4. The time now is 08:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy