07-05-2008
removing duplicates
Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here )
I'm trying to write a scrip that will remove duplicate entries
the script can typically made up of the following :
--------------
Ms AA
Unique to A
More of A
Mr BB
Mr CC
Ms AA
Unique to A
More of A
Mr DD
Mr EE
Mr BB
------------
Some of my technqiues of just are't working quite right especially with ignoring white spaces (maybe sed here)
(e.g. awk -F, '! mail[$3]++' inputfile )
any tips ?
ts
ms s
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working.
I've search the forum but nothing comes close.
I have a sample file:
27147140631203RA CCD *
27147140631203RA PPN *
37147140631207RD AAA
47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies
2. UNIX for Dummies Questions & Answers
Hello experts,
I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter
sort -u -k 2,2 File.csv > Output.csv
File.csv
File Name|Document Name|Document Title|Organization
Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies
3. Shell Programming and Scripting
Hi,
I have a file in the below format.,
test test (10)
to to (25)
see see (45)
and i need the output in the format of
test 10
to 25
see 45
Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies
4. UNIX for Advanced & Expert Users
Hi All
In unix ,we have a file ,there we have to remove the duplicates by using one specific column.
Can any body tell me the command.
ex:
file1
id,name
1,ww
2,qwq
2,asas
3,asa
4,asas
4,asas
o/p:
1,ww
2,qwq
3,asa (7 Replies)
Discussion started by: raju4u
7 Replies
5. Shell Programming and Scripting
I have a test file with the following 2 columns:
Col 1 | Col 2
T1 | 1 <= remove
T5 | 1
T4 | 2
T1 | 3
T3 | 3
T4 | 1 <= remove
T1 | 2 <= remove
T3 ... (7 Replies)
Discussion started by: gctex
7 Replies
6. Emergency UNIX and Linux Support
i want to remove all the duplictaes in a file.I dont want even a single entry.
For the input data:
12345|12|34
12345|13|23
3456|12|90
15670|12|13
12345|10|14
3456|12|13
i need the below data in one file
15670|12|13
and the below data in another file (9 Replies)
Discussion started by: pandeesh
9 Replies
7. Shell Programming and Scripting
I have an input file abc.txt with info like:
abcd
rateuse
inklite
robet
rateuse
abcd
I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file.
can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies
8. UNIX for Dummies Questions & Answers
Hi All,
I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file
I,01,000131,764,2,4.00
I,01,000131,765,2,4.00
I,01,000131,772,2,4.00
I,01,000131,773,2,4.00
I,01,000168,762,2,2.00
I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies
9. Shell Programming and Scripting
Hi All,
i have a file like below,
@DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql
@DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql
@DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql
@DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql... (11 Replies)
Discussion started by: mechvijays
11 Replies
10. Shell Programming and Scripting
i hav two files like
i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies
LEARN ABOUT SUSE
mlmmj-recieve
mlmmj-recieve(1) General Commands Manual mlmmj-recieve(1)
NAME
mlmmj-recieve - recieve mails for an mlmmj managed mailinglist
SYNOPSIS
mlmmj-recieve -L /path/to/listdir [-h] [-V] [-P] [-F]
-h: This help
-F: Don't fork in the background (debugging only)
-L: Full path to list directory
-P: Don't execute mlmmj-process (debugging only)
-V: Print version
DESCRIPTION
The mlmmj-recieve binary is the one specified in the mailserver configuration file (aliases file), which writes the mail to the <list-
dir>/incoming directory and invokes mlmmj-process unless the -P option is specified. On systems using mailservers supporting the
/etc/aliases file, a line to activate an mlmmj managed mailinglist would look like this:
list: "|/usr/bin/mlmmj-recieve -L /var/spool/mlmmj/list/"
It's very important to specify the full path to the binary, or the mailinglist will not function.
When the -F option is used, it will not fork in the background. The reason it forks is that if delivery of a mail takes longer time than
the mail server will allow a command to be idle before presumed dead, the mail server would kill it.
SEE ALSO
mlmmj-process(1)
AUTHORS
This manual page was written by the following persons:
Soren Boll Overgaard <boll@debian.org> (based on html2man output)
Mads Martin Jorgensen <mmj@mmj.dk>
mlmmj-recieve September 2004 mlmmj-recieve(1)