How can i remove the duplicate lines from a file, for example
sample123456Sample
testing123456testing
XXXXX131323XXXXX
YYYYY423432YYYYY
fsdfdsf123456gsdfdsd
all the duplicates from column 6-12 , must be deleted. I want to consider the first row, if same comes in the given range i want to... (1 Reply)
hi.. i have a file in the following format :-
name-a
age -12
address-123
age-12
phone-22222
============
name-ab
age -11
address-123
age-11
phone-222223
=============
name-abc
age -12
address-1234
age-12
phone-2222223
============= (2 Replies)
Hi,
I am writing a shell script that needs to remove duplicate lines within a file by category.
example:
section a
a
c
b
a
section b
a
b
a
c
I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Hi,
I need to remove duplicates from a file. The file will be like this
0003 10101 20100120 abcdefghi
0003 10101 20100121 abcdefghi
0003 10101 20100122 abcdefghi
0003 10102 20100120 abcdefghi
0003 10103 20100120 abcdefghi
0003 10103 20100121 abcdefghi
Here if the first colum and... (6 Replies)
Hi,
I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file.
Source filename: Filename.csv
"1","ccc","information","5000","temp","concept","new"
"1","ddd","information","6000","temp","concept","new"... (2 Replies)
All,
I have a file 1181CUSTOMER-L061411_003500.dat.Z having duplicate records in it.
bash-2.05$ zcat 1181CUSTOMER-L061411_003500.dat.Z|grep "90876251S"
90876251S|ABG, AN ADAYANA COMPANY|3550 DEPAUW BLVD|||US|IN|INDIANAPOLIS||DAL|46268||||||GEN|||||||USD|||ABG, AN ADAYANA... (3 Replies)
Hi,
I have a tablular separated file and I want to remove all the rows that have duplicates. The diuplicates I need to check are in column 13.
I have tried to use awk but I have no Idea how to keep the duplicate file.
awk 'FNR==NR{a++;next}(a> 1)' tomodify.txt tomodify.txt > new.txt
... (4 Replies)
Hi some one please help me to remove duplicates from a pipe delimited file based on first two columns.
123|asdf|sfsd|qwrer
431|yui|qwer|opws
123|asdf|pol|njio
Here My first record and last record are duplicates.As per my requirement I want all the latest records into one file.
I want the... (12 Replies)
Hi all,
I have a issues while loading a flat file to the DB. It is taking much time.
When analyzed i found out that there are duplicates entry in the flat file.
There are 2 type of Duplicate entry.
1) is entire row is duplicate. ( i can use sort | uniq) to remove the duplicated entry.
2) the... (4 Replies)
Discussion started by: samjoshuab
4 Replies
LEARN ABOUT DEBIAN
pilot-dedupe
pilot-dedupe(1) PILOT-LINK pilot-dedupe(1)NAME
pilot-dedupe - Removes duplicate records from any Palm database.
SECTION
pilot-link: Userland conduits
SYNOPSIS
pilot-dedupe [-p|--port <port>] [-q|--quiet] [--version] [-?|--help] [--usage] database ...
DESCRIPTION
pilot-dedupe is used to remove duplicate records from any Palm database.
OPTIONS
Conduit Options
-p, --port lt;port>
Use device file <port> to communicate with the Palm handheld. If this is not specified, pilot-dedupe will look for the $PILOTPORT envi-
ronment variable. If neither are found or supplied, pilot-dedupe will print the usage information.
-q,
--quiet
Suppress 'Hit HotSync button' message
-v, --version
Display version of pilot-dedupe and exit without connecting.
Help Options
-h, --help
Display the help synopsis for pilot-dedupe and exit without connecting.
--usage
Display a brief usage message and exit without connecting.
EXAMPLE
To remove duplicates from the Address and Todo databases:
pilot-dedupe -p /dev/pilot AddressDB ToDoDB
AUTHOR
This manual page was written by Neil Williams <linux@codehelp.co.uk>
REPORTING BUGS
We have an online bug tracker. Using this is the only way to ensure that your bugs are recorded and that we can track them until they are
resolved or closed. Reporting bugs via email, while easy, is not very useful in terms of accountability. Please point your browser to
http://bugs.pilot-link.org: http://bugs.pilot-link.org and report your bugs and issues there.
COPYRIGHT
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER-
CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.
SEE ALSO pilot-link(7).
0.12.5 Copyright 1996-2007 FSF pilot-dedupe(1)