Sponsored Content
Full Discussion: Unique words in each line
Top Forums Shell Programming and Scripting Unique words in each line Post 302761935 by Scrutinizer on Sunday 27th of January 2013 01:17:53 PM
Old 01-27-2013
HI. that would work, yes but it would introduce excess whitespace. An extra recalculation of the fields ( $0=$0 ) first reduces the number of fields (if duplicates were removed), so that after that, by recalculating the record ( $1=$1 ) the excess whitespace gets removed...

Last edited by Scrutinizer; 01-28-2013 at 01:59 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to read all the unique words in a text file

How can i read all the unique words in a file, i used - cat comment_file.txt | /usr/xpg6/bin/tr -sc 'A-Za-z' '/012' and cat comment_file.txt | /usr/xpg6/bin/tr -sdc 'A-Za-z' '/012' but they didnt worked..... (5 Replies)
Discussion started by: aditya.ece1985
5 Replies

2. Shell Programming and Scripting

How to print the words in the same line with space or to the predefined line?

HI, cat test abc echo "def" >> test output is cat test abc def the needed output is cat test abc def and so on (5 Replies)
Discussion started by: jobycxa
5 Replies

3. Shell Programming and Scripting

Copying x words from end of line to specific location in same line

Hello all i know it is pretty hard one but you will manage it all after noticing and calculating i find a rhythm for the file i want to edit to copy the last 12 characters in line but the problem is to add after first 25 characters in same line in other way too copy the last 12 characters... (10 Replies)
Discussion started by: princesasa
10 Replies

4. Shell Programming and Scripting

Finding the number of unique words in a file

find the number of unique words in a file using sort com- mand. (7 Replies)
Discussion started by: abhikamune
7 Replies

5. Homework & Coursework Questions

unique words in files of folder and its subfolders

Hello, I tried to count all unique words of all files in one folder and its subfolders. Can anybody say me, why this doesnt work: ls| find -d | cat | tr "\ " "\n"| uniq -u | wc -l ??? Cat writes only the names of those files, but not the wors, which should be in them. Thanks for any advice. ... (9 Replies)
Discussion started by: Dworza
9 Replies

6. Shell Programming and Scripting

display unique words.

I am having a file with duplicate words how can I eliminate them ant,bat bat,cat cat a.txt | grep -bat | awk '{print $1}' expecting o/p as ant,bat,cat How can I display the output as ant,bat,cat in a single line and no duplicates exists. (2 Replies)
Discussion started by: shikshavarma
2 Replies

7. Shell Programming and Scripting

Search words in multiple file line by line

Hi All I have to search servers name say like 1000+ "unique names" line by line in child.txt files in another file that is a master file where all server present say "master.txt",if child.txt's server name matches with master files then it print yes else no with server name. (4 Replies)
Discussion started by: netdbaind
4 Replies

8. Shell Programming and Scripting

Regex to identify unique words in a dictionary database

Hello, I have a dictionary which I am building for the Open Source Community. The data structure is as under HEADWORD=PARTOFSPEECH=ENGLISH MEANING as shown in the example below अ=m=Prefix signifying negation. अँहँ=ind=Interjection expressing disapprobation. अं=int=An interjection... (2 Replies)
Discussion started by: gimley
2 Replies

9. UNIX for Beginners Questions & Answers

Count unique words

Dear all, I would like to know how to list and count unique words in thousands number of text files. Please help me out thanks in advance (9 Replies)
Discussion started by: imranrasheedamu
9 Replies

10. Shell Programming and Scripting

Replace particular words in file based on if finds another words in that line

Hi All, I need one help to replace particular words in file based on if finds another words in that file . i.e. my self is peter@king. i am staying at north sydney. we all are peter@king. How to replace peter to sham if it finds @king in any line of that file. Please help me... (8 Replies)
Discussion started by: Rajib Podder
8 Replies
Array::Unique(3pm)					User Contributed Perl Documentation					Array::Unique(3pm)

NAME
Array::Unique - Tie-able array that allows only unique values SYNOPSIS
use Array::Unique; tie @a, 'Array::Unique'; Now use @a as a regular array. DESCRIPTION
This package lets you create an array which will allow only one occurrence of any value. In other words no matter how many times you put in 42 it will keep only the first occurrence and the rest will be dropped. You use the module via tie and once you tied your array to this module it will behave correctly. Uniqueness is checked with the 'eq' operator so among other things it is case sensitive. As a side effect the module does not allow undef as a value in the array. EXAMPLES
use Array::Unique; tie @a, 'Array::Unique'; @a = qw(a b c a d e f); push @a, qw(x b z); print "@a "; # a b c d e f x z DISCUSSION
When you are collecting a list of items and you want to make sure there is only one occurrence of each item, you have several option: 1) using an array and extracting the unique elements later You might use a regular array to hold this unique set of values and either remove duplicates on each update by that keeping the array always unique or remove duplicates just before you want to use the uniqueness feature of the array. In either case you might run a function you call @a = unique_value(@a); The problem with this approach is that you have to implement the unique_value function (see later) AND you have to make sure you don't forget to call it. I would say don't rely on remembering this. There is good discussion about it in the 1st edition of the Perl Cookbook of O'Reilly. I have copied the solutions here, you can see further discussion in the book. Extracting Unique Elements from a List (Section 4.6 in the Perl Cookbook 1st ed.) # Straightforward %seen = (); @uniq = (); foreach $item (@list) [ unless ($seen{$item}) { # if we get here we have not seen it before $seen{$item} = 1; push (@uniq, $item); } } # Faster %seen = (); foreach $item (@list) { push(@uniq, $item) unless $seen{$item}++; } # Faster but different %seen; foreach $item (@list) { $seen{$item}++; } @uniq = keys %seen; # Faster and even more different %seen; @uniq = grep {! $seen{$_}++} @list; 2) using a hash Some people use the keys of a hash to keep the items and put an arbitrary value as the values of the hash: To build such a list: %unique = map { $_ => 1 } qw( one two one two three four! ); To print it: print join ", ", sort keys %unique; To add values to it: $unique{$_}=1 foreach qw( one after the nine oh nine ); To remove values: delete @unique{ qw(oh nine) }; To check if a value is there: $unique{ $value }; # which is why I like to use "1" as my value (thanks to Gaal Yahas for the above examples) There are three drawbacks I see: 1) You type more. 2) Your reader might not understand at first why did you use hash and what will be the values. 3) You lose the order. Usually non of them is critical but when I saw this the 10th time in a code I had to understand with 0 documentation I got frustrated. 3) using Array::Unique So I decided to write this module because I got frustrated by my lack of understanding what's going on in that code I mentioned. In addition I thought it might be interesting to write this and then benchmark it. Additionally it is nice to have your name displayed in bright lights all over CPAN ... or at least in a module. Array::Unique lets you tie an array to hmmm, itself (?) and makes sure the values of the array are always unique. Since writing this I am not sure if I really recommend its usage. I would say stick with the hash version and document that the variable is aggregating a unique list of values. 4) Using real SET There are modules on CPAN that let you create and maintain SETs. I have not checked any of those but I guess they just as much of an overkill for this functionality as Unique::Array. BUGS
use Array::Unique; tie @a, 'Array::Unique'; @c = @a = qw(a b c a d e f b); @c will contain the same as @a AND two undefs at the end because @c you get the same length as the right most list. TODO
Test: Change size of the array Elements with false values ('', '0', 0) splice: splice @a; splice @a, 3; splice @a, -3; splice @a, 3, 5; splice @a, 3, -5; splice @a, -3, 5; splice @a, -3, -5; splice @a, ?, ?, @b; Benchmark speed Add faster functions that don't check uniqueness so if I know part of the data that comes from a unique source then I can speed up the process, In short shoot myself in the leg. Enable optional compare with other functions Write even better implementations. AUTHOR
Gabor Szabo <gabor@pti.co.il> LICENSE
Copyright (C) 2002-2008 Gabor Szabo <gabor@pti.co.il> All rights reserved. http://www.pti.co.il/ You may distribute under the terms of either the GNU General Public License or the Artistic License, as specified in the Perl README file. No WARRANTY whatsoever. CREDITS
Thanks for suggestions and bug reports to Szabo Balazs (dLux) Shlomo Yona Gaal Yahas Jeff 'japhy' Pinyan Werner Weichselberger VERSION
Version: 0.08 Date: 2008 June 04 perl v5.10.0 2009-03-06 Array::Unique(3pm)
All times are GMT -4. The time now is 06:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy