Sponsored Content
Full Discussion: Remove duplicate in colomn
Top Forums UNIX for Beginners Questions & Answers Remove duplicate in colomn Post 303016668 by RavinderSingh13 on Wednesday 2nd of May 2018 01:54:58 AM
Old 05-02-2018
Hello mirwasim,

You are using correct logic but wrong field number, try following and let me know if this helps you.
Code:
awk '!a[$2]++'   Input_file

Where $2 denotes 2nd field of the current line in Input_file.

Thanks,
R. Singh
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies

2. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

3. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

4. Shell Programming and Scripting

changing colomn to the line

I have a file which has only one colomn of numbers,ex: 122 173 292 400 979 2152 2339 2376 2387 2446 2450 What ksh / unix command should I use to create a file in which those numbers will be in one line,like this 122 173 292 400 979 .... etc Thanks a lot for help (9 Replies)
Discussion started by: aoussenko
9 Replies

5. Shell Programming and Scripting

switch line to colomn

Hi It's possible to switch the line to colon from a file using Perl or AWK? for example my file have somthing like this: 10 11 12 13 14 20 21 22 23 24 30 31 32 33 34 I want to have a file with the line switched like : 10 20 30 11 21 31 12 22 32 13 23 33 14 24 34 Thanks a lot (4 Replies)
Discussion started by: rauchy
4 Replies

6. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

7. Shell Programming and Scripting

How to exclude certain colomn from the file?

Hi I have a file in the following format: 4135 f4135acc: 39798 rmtdb: 0 /t1/data/f4135acc.dta 4135 f4135pdb: 39795 rmtdb: 0 /bb/data/f4135pdb.dta 4135 p4135eng: 0 rmtdb: 0 /bb/bin/p4135eng 4135 r4135eng: 14142 rmtdb: 0 ... (6 Replies)
Discussion started by: aoussenko
6 Replies

8. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532 (9 Replies)
Discussion started by: buzzme
9 Replies

9. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

10. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies
COLRM(1)						    BSD General Commands Manual 						  COLRM(1)

NAME
colrm -- remove columns from a file SYNOPSIS
colrm [start [stop]] DESCRIPTION
The colrm utility removes selected columns from the lines of a file. A column is defined as a single character in a line. Input is read from the standard input. Output is written to the standard output. If only the start column is specified, columns numbered less than the start column will be written. If both start and stop columns are spec- ified, columns numbered less than the start column or greater than the stop column will be written. Column numbering starts with one, not zero. Tab characters increment the column count to the next multiple of eight. Backspace characters decrement the column count by one. ENVIRONMENT
The LANG, LC_ALL and LC_CTYPE environment variables affect the execution of colrm as described in environ(7). EXIT STATUS
The colrm utility exits 0 on success, and >0 if an error occurs. SEE ALSO
awk(1), column(1), cut(1), paste(1) HISTORY
The colrm command appeared in 3.0BSD. BSD
August 4, 2004 BSD
All times are GMT -4. The time now is 12:00 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy