11-04-2008
Print Only second Duplicate entry in the file
I have file where it contains 2 columns. In two columns the first column is repeated more than once. I wanted to take the unique record in first column
and the corresponding second column value .
The below is the example of the file:
8244100320012955|000b063471a4
8244100320012955|00080e5a6624
8244100330007664|0004bde2ac84
8244100330007664|00080e429228
8244100330011369|0004bd0ec236
8244100330011369|0012c9a1c44a
8244100330016046|00080e0b46c4
8244100330016046|000b06cb18fe
In the above example i need the output as
8244100320012955|00080e5a6624
8244100330007664|00080e429228
8244100330011369|0012c9a1c44a
8244100330016046|000b06cb18fe
I am using the following command
cat f1.txt | cut -d"|" -f 1,2 |sort |uniq -d|wc -l
but the above command is failed
Please help me.
Thank you very much in advance.
10 More Discussions You Might Find Interesting
1. HP-UX
Hi All,
How to prevent starting of processes that have duplicate entries in cron file, i have written a shell script to validate with "ps |grep" command before starting the process, but still when same process started at same time, it may not be able to detect the existing process.
Sample... (3 Replies)
Discussion started by: nag_sundaram
3 Replies
2. UNIX for Dummies Questions & Answers
Can anyone help me how can i print only the unique entry in a line?
MI_AP MI_AP MI_CM MI_MF
RC_NAP MBS_AP SF_RAN MBS_AP NT_CAR
so that it will on output the one unique entry per line.
MI_AP MI_CM MI_MF
RC_NAP MBS_AP SF_RAN NT_CAR
I can't find the same situation on the knowledge... (5 Replies)
Discussion started by: kharen11
5 Replies
3. Shell Programming and Scripting
Hi,
I just stuckup to echo a variable that is assigned in .profile file from a crontab entry.
I have a entry in .profile as,
CONF_FILE_PATH=/home/test/conf
export CONF_FILE_PATH
I am able to echo it in command line but I am not able to print $CONF_FILE_PATH when i scheduled it in... (4 Replies)
Discussion started by: naren_0101bits
4 Replies
4. UNIX for Dummies Questions & Answers
OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script.
So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup
... (2 Replies)
Discussion started by: bfurlong
2 Replies
5. Shell Programming and Scripting
Hi *,
I need to delete duplicate lease entries in file according to MAC/IP.
I'm having tempfile which contains many lease info and need to have one entry for each IP(not more than that), if it contains more than one entry for same set, need to be deleted that entry...
EX:
lease... (4 Replies)
Discussion started by: SMNK
4 Replies
6. UNIX for Dummies Questions & Answers
input
chr1 10 100
chr1 10 100
chr1 20 200
output
chr1 10 100 2
chr1 20 200 1 (1 Reply)
Discussion started by: repinementer
1 Replies
7. Shell Programming and Scripting
I have a file where some of the lines are duplicates.
How do I use bash to print all the lines that have duplicates? (2 Replies)
Discussion started by: locoroco
2 Replies
8. Shell Programming and Scripting
Hi i have a file like
110.10
120.10
-1120
110.10
and the lines are having more than 10k.
do we have anycommand to check the duplicate entries in the file.
I applied the while loop by greping each line with whole file,
but it is taking huge amount of time as the file size is large.
... (5 Replies)
Discussion started by: saluja.deepak
5 Replies
9. Shell Programming and Scripting
AWK Gurus,
data:
srvhcm01 AZSCI
srvhcm01 AZSDB
srvhcm01 BZSDB
srvhcm01 E2QDI31
srvhcm01 YPDCI
srvhcm01 YPDDB
srvhcm01 UV2FSCR
srvhcm01 UV2FSBI
srvhcm01 UV2FSXI
srvhcm01 UV2FSUC
srvhcm01 UV2FSEP
srvhcm01 UV2FSRE
srvhcm01 NASCI
srvhcm01 NASDB
srvhcm01 UV2FSSL
srvhcm01 UV2FSDI (7 Replies)
Discussion started by: rveri
7 Replies
10. Shell Programming and Scripting
I am working on an Urdu to Hindi dictionary and I have created the following file structure:
Headword=Gloss1,Gloss2,Gloss3
i.e. glosses delimited by a comma.
It so happens that in some cases (around 6000+ in a file of over 200,000+ the glosses are duplicated.
Since this may be a... (3 Replies)
Discussion started by: gimley
3 Replies
LEARN ABOUT CENTOS
column
COLUMN(1) User Commands COLUMN(1)
NAME
column - columnate lists
SYNOPSIS
column [options] file...
DESCRIPTION
The column utility formats its input into multiple columns. Rows are filled before columns. Input is taken from file or, by default, from
standard input. Empty lines are ignored.
OPTIONS
-c, --columns width
Output is formatted to a width specified as number of characters.
-t, --table
Determine the number of columns the input contains and create a table. Columns are delimited with whitespace, by default, or with
the characters supplied using the separator. Table output is useful for pretty-printing.
-s, --separator separators
Specify possible table delimiters (default is whitespace).
-o, --output-separator separators
Specify table output delimiter (default is two whitespaces).
-x, --fillrows
Fill columns before filling rows.
-h, --help
Print help and exit.
ENVIRONMENT
The environment variable COLUMNS is used to determine the size of the screen if no other information is available.
EXAMPLES
sed 's/#.*//' /etc/fstab | column -t
BUGS
The util-linux version 2.23 changed -s option to be non-greedy, for example:
$ printf "a:b:c
1::3
" | column -t -s ':'
old output:
a b c
1 3
new output (since util-linux 2.23)
a b c
1 3
SEE ALSO
colrm(1), ls(1), paste(1), sort(1)
HISTORY
The column command appeared in 4.3BSD-Reno.
AVAILABILITY
The column command is part of the util-linux package and is available from ftp://ftp.kernel.org/pub/linux/utils/util-linux/.
util-linux October 2010 COLUMN(1)