hi all,
i have a file contain multicolumns, this file is sorted by col2 and col3.
i want to remove the duplicated columns if the col2 and col3 are the same in another line.
example
fileA
AA BB CC DD
CC XX CC DD
BB CC ZZ FF
DD FF HH HH
the output is
AA BB CC DD
BB CC ZZ FF... (6 Replies)
Hello all,
I've got a strange behaviour of sort and uniq commands: they do not recognise apparently duplicated lines in a file (already sorted). The lines are identical by eye, but they must differ in smth, because when they are put in two files, those have slightly different size.
What can make... (8 Replies)
I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
hello, I have a large file (about 1gb) that is in a file similar to the following:
I want to make it so that I can put all the duplicates where column 3 (delimited by the commas) are shown on top. Meaning all people with the same age are listed at the top.
The command I used was ... (3 Replies)
In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows.
The file name in the folder will look like e.q:
MissingData_2014-08-25_09-30-18.txt
MissingData_2014-08-25_09-30-14.txt
MissingData_2014-08-26_09-30-12.txt
The content... (9 Replies)
Hi, all
I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Hello.
I have a server with 2 boot disk but in the bootlist there are 5 paths of one disk but no path of the other.
How can I remove paths from one disk to insert paths from the other disk?
Thanks in advance.
server074:root:/# bootlist -om normal
hdisk0 blv=hd5 pathid=0
hdisk0... (7 Replies)
Discussion started by: Gabriander
7 Replies
LEARN ABOUT CENTOS
doveadm-deduplicate
DOVEADM-DEDUPLICATE(1) Dovecot DOVEADM-DEDUPLICATE(1)NAME
doveadm-deduplicate - expunge duplicate messages
SYNOPSIS
doveadm [-Dv] deduplicate [-u user|-A] [-S socket_path] [-m] search_query
DESCRIPTION
This command is used to expunge duplicated messages in mailboxes. doveadm deduplicate is mainly useful to revert some (more or less) acci-
dental duplication of messages, e.g. after doveadm copy or doveadm import. doveadm(1) will delete the newest duplicated messages from the
mailbox and keep the oldest.
Deduplication across multiple mailboxes is not supported.
OPTIONS
Global doveadm(1) options:
-D Enables verbosity and debug messages.
-v Enables verbosity, including progress counter.
Command specific options:
-A If the -A option is present, the command will be performed for all users. Using this option in combination with system users from
userdb { driver = passwd } is not recommended, because it contains also users with a lower UID than the one configured with the
first_valid_uid setting.
When the SQL userdb module is used make sure that the iterate_query setting in /etc/dovecot/dovecot-sql.conf.ext matches your data-
base layout. When using the LDAP userdb module, make sure that the iterate_attrs and iterate_filter settings in /etc/dovecot/dove-
cot-ldap.conf.ext match your LDAP schema. Otherwise doveadm(1) will be unable to iterate over all users.
-S socket_path
The option's argument is either an absolute path to a local UNIX domain socket, or a hostname and port (hostname:port), in order to
connect a remote host via a TCP socket.
This allows an administrator to execute doveadm(1) mail commands through the given socket.
-m if the -m option is given, doveadm(1) will deduplicate by Message-Id header. By default deduplication will be done by message
GUIDs.
-u user/mask
Run the command only for the given user. It's also possible to use '*' and '?' wildcards (e.g. -u *@example.org).
When neither the -A option nor -u user was specified, the command will be executed with the environment of the currently logged in
user.
ARGUMENTS
search_query
expunge duplicates found from messages matching the given search query. Typically a search query like 'mailbox mailbox_name OR
mailbox other_box' will be sufficient. See doveadm-search-query(7) for details.
EXAMPLE
This example shows how to list and expunge duplicate messages from a mailbox.
doveadm -f table fetch -u jane 'guid uid' mailbox a_Box | sort
guid uid
8aad0f0a30169f4bea620000ca356bad 18751
8aad0f0a30169f4bea620000ca356bad 18756
923e301ab9219b4b4f440000ca356bad 18748
923e301ab9219b4b4f440000ca356bad 18753
...
doveadm deduplicate -u jane mailbox a_Box
doveadm -f table fetch -u jane 'guid uid' mailbox a_Box | sort
guid uid
8aad0f0a30169f4bea620000ca356bad 18751
923e301ab9219b4b4f440000ca356bad 18748
a7999e1530739c4bd26d0000ca356bad 18749
...
REPORTING BUGS
Report bugs, including doveconf -n output, to the Dovecot Mailing List <dovecot@dovecot.org>. Information about reporting bugs is avail-
able at: http://dovecot.org/bugreport.html
SEE ALSO doveadm(1), doveadm-copy(1), doveadm-fetch(1), doveadm-import(1), doveadm-search-query(7)Dovecot v2.2 2013-08-03 DOVEADM-DEDUPLICATE(1)