04-28-2010
Script to remove duplicates
Hi
I need a script that removes the duplicate records and write it to a new file
for example I have a file named test.txt and it looks like
abcd.23
abcd.24
abcd.25
qwer.25
qwer.26
qwer.98
I want to pick only $1 and compare with the next record and the output should be
abcd.23
qwer.25
can some one help?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello Experts,
I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
2. Shell Programming and Scripting
Hi,
I am writing a shell script that needs to remove duplicate lines within a file by category.
example:
section a
a
c
b
a
section b
a
b
a
c
I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies
3. Shell Programming and Scripting
Hi,
I need to remove duplicates from a file. The file will be like this
0003 10101 20100120 abcdefghi
0003 10101 20100121 abcdefghi
0003 10101 20100122 abcdefghi
0003 10102 20100120 abcdefghi
0003 10103 20100120 abcdefghi
0003 10103 20100121 abcdefghi
Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies
4. Shell Programming and Scripting
Hi,
I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another.
Thanks (6 Replies)
Discussion started by: dvah
6 Replies
5. Shell Programming and Scripting
I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different.
I need to go from this:
http://***/fae78fe/file1.wmv
http://***/39du7si/file1.wmv
http://***/d8el2hd/file2.wmv
http://***/h893js3/file2.wmv
to this:
... (2 Replies)
Discussion started by: locoroco
2 Replies
6. UNIX for Dummies Questions & Answers
Hello experts!
I'd like a way to remove duplicates per line. Strings are enclosed in brackets, and I would prefer to maintain the order of the file:
example input
(56)(63)
(56)(70)(56)(70)(24)
(25)(78)
(12)(33)(12)
(10)
(10)
desired output
(56)(63)
(56)(70)(24)
(25)(78)... (5 Replies)
Discussion started by: torchij
5 Replies
7. Shell Programming and Scripting
I have a file with the following format:
fields seperated by "|"
title1|something class|long...content1|keys
title2|somhing class|log...content1|kes
title1|sothing class|lon...content1|kes
title3|shing cls|log...content1|ks
I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies
8. Shell Programming and Scripting
hello ,
I have a requirement with input in below format
abc 123 xyz
bcd 365 kii
abc 987 876
cdf 987 uii
abc 456 yuu
bcd 654 rrr
Expecting Output
abc 456 yuu
bcd 654 rrr
cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies
9. Shell Programming and Scripting
Hi I have a below file structure.
200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,,
200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,,
300,20140223,0.001,0.001,0.001,0.001,0.001
300,20140224,0.001,0.001,0.001,0.001,0.001
300,20140225,0.001,0.001,0.001,0.001,0.001
300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
10. Shell Programming and Scripting
values=(1 2 3 5 4 2 3 1 6 8 3 5 )
#i need the output like this by removing the duplicates
1
2
3
5
4
6
8
#i dont need sorting in my program
#plz explain me as simple using for loop
#os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies
LEARN ABOUT OSX
test::script
Test::Script(3) User Contributed Perl Documentation Test::Script(3)
NAME
Test::Script - Basic cross-platform tests for scripts
DESCRIPTION
The intent of this module is to provide a series of basic tests for 80% of the testing you will need to do for scripts in the script (or
bin as is also commonly used) paths of your Perl distribution.
Further, it aims to provide this functionality with perfect platform-compatibility, and in a way that is as unobtrusive as possible.
That is, if the program works on a platform, then Test::Script should always work on that platform as well. Anything less than 100% is
considered unacceptable.
In doing so, it is hoped that Test::Script can become a module that you can safely make a dependency of all your modules, without risking
that your module won't on some platform because of the dependency.
Where a clash exists between wanting more functionality and maintaining platform safety, this module will err on the side of platform
safety.
FUNCTIONS
script_compiles
script_compiles( 'script/foo.pl', 'Main script compiles' );
The "script_compiles" test calls the script with "perl -c script.pl", and checks that it returns without error.
The path it should be passed is a relative unix-format script name. This will be localised when running "perl -c" and if the test fails the
local name used will be shown in the diagnostic output.
Note also that the test will be run with the same perl interpreter that is running the test script (and not with the default system perl).
This will also be shown in the diagnostic output on failure.
script_runs
script_runs( 'script/foo.pl', 'Main script runs' );
The "script_runs" test executes the script with "perl script.pl" and checks that it returns success.
The path it should be passed is a relative unix-format script name. This will be localised when running "perl -c" and if the test fails the
local name used will be shown in the diagnostic output.
The test will be run with the same perl interpreter that is running the test script (and not with the default system perl). This will also
be shown in the diagnostic output on failure.
SUPPORT
All bugs should be filed via the bug tracker at
http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Script <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Script>
For other issues, or commercial enhancement and support, contact the author.
AUTHOR
Adam Kennedy <adamk@cpan.org>
SEE ALSO
prove, <http://ali.as/>
COPYRIGHT
Copyright 2006 - 2009 Adam Kennedy.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
The full text of the license can be found in the LICENSE file included with this module.
perl v5.16.2 2009-11-23 Test::Script(3)