Do Not Output Duplicates Post: 302898758

Sponsored Content

Top Forums Shell Programming and Scripting Do Not Output Duplicates Post 302898758 by sudo on Wednesday 23rd of April 2014 05:27:02 PM

04-23-2014

Registered User

Do Not Output Duplicates

Mac OS 10.9

Let me preface this by saying this is not for marketing or spamming purposes.

I have a script that scans all the email messages in a directory (~/Library/Mail/Mailboxes) and outputs a single column list of email addresses. This will run multiple times a day and append the output file with new entries.

If an email is duplicated in the email folder- it is duplicated in the output file. How do I remove these duplications from the output file? Its just a single column of data separated by a new line. Not sure if I should have it check and exclude the output of duplicates or simply run a scan for duplicates after the output file is appended.

This list is being used as input for LDAP queries.

For reference, the scanning/output portion of my script is below:

Code:

find $SRC -type f -name *.emlx |
	while read FILE
	do
	   awk '/^From:/ && gsub(/.*<|>.*/,x)' $FILE
	done > ~/Desktop/output.txt
echo "complete"

sudo

View Public Profile for sudo

Find all posts by sudo

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading Input from File and Duplicates Output

Greetings to all, I would like to read input from a file and make duplications from it with Linux shell. For e.g. Input file ----------- ABC ABB ABA ------------------------------- Output file ------------ ABC ABC ABC ABB ABB

2. HP-UX

getting duplicates

how to get duplicates in a file containing data in columns using command or scripting?

3. Shell Programming and Scripting

Non Duplicates

I have input file like below. I00789524 0213 5212 D00789524 0213 5212 I00778787 2154 5412 The first two records are same(Duplicates) except I & D in the first character. I want non duplicates(ie. 3rd line) to be output. How can we get this . Can you help. Is there any single AWK or SED...

4. UNIX for Dummies Questions & Answers

Duplicates

Hi, How to eliminate the duplicate values in unix? I have a excel file which contains duplicate values. Need to use this in a script. Thanks in advance.

5. AIX

Duplicates in bootlist

Hello, I'm moving some disks from the rootvg on AIX 5.3. # replacepv hdiskOLD hdiskNEW I have for example hdisk12 and hdisk13 with hd5 (boot) LV and want to move hdisk13 So 1st I'm excluding it from the bootlist: # bootlist -om normal hdisk12 then # replacepv hdisk13...

6. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :(

7. Shell Programming and Scripting

Remove duplicates based on query and subject fields from blast output file

Hi all I have a blast outfile file like this : NZ_1540841_1561981 ICMP_1687819_1695946 92.59 27 2 0 12826 12852 3136 3162 0.28 38.2 NZ_1540841_1561981 ICMP_1687819_1695946 95.65 23 1 0 12268 12290 5815 5837 0.28 38.2 NZ_1540841_1561981 ICMP_3674888_3676546 82.70 185 32 0 9454 9638 11 195 6e-24 ...

8. UNIX for Dummies Questions & Answers

Filtering the duplicates

Hello, I want to filter all the duplicates of a record to one place. Sample input and output will give you better idea. I am new to unix. Can some one help me on this? Input: 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr1.fa chr1.fa...

9. Shell Programming and Scripting

sed & remove duplicates on output

sed -e '1d' -e 's/^\(]\{2\}\)-\(]\{3\}\)-\(]\{4\}\).*/"0000020\1\200\3"\,/g' abc.txt This script returns many duplicates due to the duplciates in the .txt file. i.e. ... "000002012149000060", "000002012149000064", "000002012149000064", "000002012149000064", "000002012149000064",...

10. Shell Programming and Scripting

Remove duplicates

LEARN ABOUT DEBIAN

hmine

HMINE(1)																  HMINE(1)

NAME

       hmine - a mail message header analyzer.

SYNOPSIS

       hmine [-vDa] [FILE]

       hmine -V

DESCRIPTION

       hmine reads a mail message from FILE or STDIN and outputs a variety of information found in the message headers. The message is expected in
       Internet mail format (RFC 821,822,2821,2822 or variations thereof). The body is not inspected.

EXIT STATUS

       On success, hmine returns 1. In case of a problem, hmine returns zero.

OPTIONS

       -a     Print mailboxes and groups found in various header fields, one per line, preceded by the field  name.  Actual  email  addresses  are
	      always  enclosed	in  '<'  and '>' for easy parsing, ie anything not within these delimiters is not part of an email address. Beware
	      that not every line need contain an email address.

       -D     Debug output.

       -V     Print the program version number and exit.

USAGE

       An invocation looks like this:

       % hmine email.txt

SOURCE

       The source code for the latest version of this program is available at the following locations:

       http://www.lbreyer.com/gpl.html
       http://dbacl.sourceforge.net

BUGS

       At present, hmine parses messages but doesn't output anything useful.

AUTHOR

       Laird A. Breyer <laird@lbreyer.com>

SEE ALSO

       dbacl(1), mailcross(1), mailfoot(1), mailinspect(1), mailtoe(1), regex(7)

Version 1.12						   Bayesian Classification Tools						  HMINE(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading Input from File and Duplicates Output

Discussion started by: noelcantona

2. HP-UX

getting duplicates

Discussion started by: megh

3. Shell Programming and Scripting

Non Duplicates

Discussion started by: awk_beginner

4. UNIX for Dummies Questions & Answers

Duplicates

Discussion started by: venkatesht