Sponsored Content
Top Forums Shell Programming and Scripting Remove subsequent duplicate only Post 302772067 by Don Cragun on Friday 22nd of February 2013 04:13:27 PM
Old 02-22-2013
Quote:
Originally Posted by jamie_123
Hi,

Thanks for responding, here is a simplified case. Say I have this as input
Code:
1-num1
2-num2
3-num2
4-num3
5-num3
2-num2

Now what I want to do is not just find repetitions and remove them, but to find repetitions that are only in the next line and remove the first occurrence of that value. Repetitions are checked on $2 with FS as "-". So the output should be

Code:
1-num1
3-num2
4-num3
5-num3
2-num2

.
I think I have a simpler awk script that does what you said you want, but I don't understand why 4-num3 appears in what you say the output should be. That is the 1st line in the input file that has num3 after the hyphen and there is another line later that contains num3 so I thought you wanted that line to be dropped from the output.

Try:
Code:
awk 'BEGIN{FS = OFS = "-"}
{       f1[NR] = $1
        c[f2[NR] = $2]++
}
END {   for(i = 1; i <= NR; i++)
                if(c[f2[i]] > 1)
                        c[f2[i]] = 1
                else    print f1[i], f2[i]
}' input

produces the output:
Code:
1-num1
3-num2
5-num3
2-num2

when given the input:
Code:
1-num1
2-num2
3-num2
4-num3
5-num3
2-num2

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies

2. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

3. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

4. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

5. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

6. Shell Programming and Scripting

Remove duplicate records

I want to remove the records based on duplicate. I want to remove if two or more records exists with combination fields. Those records should not come once also file abc.txt ABC;123;XYB;HELLO; ABC;123;HKL;HELLO; CDE;123;LLKJ;HELLO; ABC;123;LSDK;HELLO; CDF;344;SLK;TEST key fields are... (7 Replies)
Discussion started by: svenkatareddy
7 Replies

7. Shell Programming and Scripting

Remove Duplicate Records

Hi frinds, Need your help. item , color ,desc ==== ======= ==== 1,red ,abc 1,red , a b c 2,blue,x 3,black,y 4,brown,xv 4,brown,x v 4,brown, x v I have to elemnet the duplicate rows on the basis of item. the final out put will be 1,red ,abc (6 Replies)
Discussion started by: imipsita.rath
6 Replies

8. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532 (9 Replies)
Discussion started by: buzzme
9 Replies

9. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

10. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies
elfdump(1)						      General Commands Manual							elfdump(1)

NAME
elfdump - dump information contained in object files. SYNOPSIS
num] num2] name] section] num] num2] files... DESCRIPTION
takes one or more object files or libraries and dumps information about them. The following options are supported: Dumps archive headers from an archive library. Dumps the string table(s). (Modifier) Demangles C++ symbol names before printing them. This modifier is valid with and If specified with this modifier is ignored. If specified with name, the symbol whose unmangled name matches name will be printed, and its symbol name will be printed as a demangled name. Prints the .note section which contains the compilation unit dictionary and linker footprint. This option has the same effect as Prints the compilation unit dictionary of the .notes section. Prints the linker footprint of the .notes section. The linker footprint has information on the linker used to generate the file as well as the link time. (Modifier) Prints the section whose index is num. (Modifier) Prints the sections in the range 1 to num2. If used with the sections in the range num to num2 are printed. Valid with If used with only the relocations which apply to the section(s) in the range are printed. Dumps the file header (ELF header). Dumps global symbols from an archive. Dumps the section headers. (Modifier) Dumps output information in hexadecimal, octal, or decimal format, with all options. Displays the run-time interpreter path name for (usually the location of the dynamic loader and microloader). Only shared bound executables have this string. To change the setting, use the command. Prints the object dictionary for one or more executable files, if the source file was compiled with the option or linked with (PA-RISC only) option. The object dictionary entry contains the name of the object file that contributed to a particular section, the relative offset within the section, size of the object file's contribution, and attributes of the entry. Prints the CTTI section headers according to the directory member relationship. Dumps the section in shared libraries and dynamically linked program files. Prints the section, which is only created when the incremental linker is used (with the command), or when the linker option is used (which is the default), along with the compiler option, (which is NOT the default). Prints the section, which is only created when the incremental linker is used (with the command), or when the linker option is used (which is the default), along with the compiler option, (which is NOT the default). Prints the section, which is only created when the incremental linker is used (with the command), or when the linker option is used (which is the default), along with the compiler option, (which is NOT the default). (Modifier) Dumps information about the specified section or symbol name. This option is valid with and If used with name pertains to a symbol name and will only dump the symbol entry whose name matches name. If used with the other options, name pertains to a section name and will only dump the section whose name matches it. Dumps the optional headers (program headers). (Modifier) Do not print titles, with all options. (Modifier) Suppresses printing CTTI section headers. Valid with option. Dumps the relocations. Dumps the section contents. Dumps any section beginning with as a string table. (Modifier) Dumps the section specified by name. Valid with and only. (Modifier) Dumps output information in short format. Valid with the and options. Dumps the symbol table entries. (Itanium-based systems) Dumps the value of in symbol table, in addition to information dump from option. This option is useful to verify the data stored in the symbol table. Prints the symbol whose index is num. (Modifier) Prints the symbols in the range 0 to num2. If used with print the symbols in the range num to num2. Valid with Prints versioned symbols. Prints the usage menu. Prints the unwind table. (Modifier) Verifies the CTTI section headers before printing. Valid with the option. Prints the version number for EXAMPLES
To see the functions exported from a shared library: To see the global data items exported from a shared library: To display string table information To list the shared libraries linked with a program or shared library (dependent libraries): To list the embedded path for shared libraries opened by a program: SEE ALSO
System Tools ld(1) Invoke the link editor Miscellaneous a.out(4) Assembler, compiler, and linker output elf(3E) Executable and Linking Format Texts and Tutorials (See the option) (See manuals(5) for ordering information)
All times are GMT -4. The time now is 01:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy