The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Advanced & Expert Users
.
google unix.com



UNIX for Advanced & Expert Users Expert-to-Expert. Learn advanced UNIX, UNIX commands, Linux, Operating Systems, System Administration, Programming, Shell, Shell Scripts, Solaris, Linux, HP-UX, AIX, OS X, BSD.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Removing tokens from cmd line bashuser2 Shell Programming and Scripting 1 06-01-2009 06:55 AM
selecting tokens from a string... c_d Shell Programming and Scripting 1 01-15-2009 05:04 AM
: + : more tokens expected Nomaad Shell Programming and Scripting 3 04-17-2008 03:49 PM
reverse tokens with sed markc Shell Programming and Scripting 1 02-22-2008 01:55 AM
tokens in unix ? seaten UNIX for Dummies Questions & Answers 6 05-09-2005 05:57 AM

Reply
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 09-22-2009
MWPita MWPita is offline
Registered User
  
 

Join Date: Sep 2009
Location: Costa Rica
Posts: 3
Find and eliminate duplcate tokens

I have a file like this:

Code:
[token1]=value1
[token2]=value2
.
.
.
[token n]=valuen

The issue is that if we get to have i.e. the [token17] line duplicated it may incurr into errors in our application.
I tried to find those repeated lines with something like

Code:
uniq -cd prueba1.txt

But it only found the repeated lines that are inmediately after the other i.e.

Code:
$ cat prueba1.txt
uno
dos
tres
tres
cuatro
cinco
seis
cuatro
siete
ocho
$ uniq -cd prueba1.txt
      2 tres

Only finding "tres" when it should also find "cuatro"
Any idea on how to fix that?

Last edited by vgersh99; 09-22-2009 at 06:08 PM.. Reason: code tags, PLEASE!
  #2 (permalink)  
Old 09-22-2009
vgersh99's Avatar
vgersh99 vgersh99 is offline Forum Staff  
Moderator
  
 

Join Date: Feb 2005
Location: Boston, MA
Posts: 5,131
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

---------- Post updated at 05:09 PM ---------- Previous update was at 05:09 PM ----------

Sort your file first - 'man sort'
  #3 (permalink)  
Old 09-22-2009
migurus migurus is offline
Registered User
  
 

Join Date: Sep 2008
Location: US
Posts: 49
Why don't you sort before running uniq?

Also, just a starting point, look at arrays usage in awk:


Code:
 
awk '
{
        key = $0; # use your key field here
        if(key in regarr) {
                duparr[key] = key
        }
        else {
                regarr[key] = key
        }
}
END {
        for(idx in duparr) {
                print idx;
        }
}
'

  #4 (permalink)  
Old 09-22-2009
MWPita MWPita is offline
Registered User
  
 

Join Date: Sep 2009
Location: Costa Rica
Posts: 3
Thanks guys
I appreciate your flash responses, this worked better than I expected
My final version is this command:

Code:
tr ' ] ' ' ' < prueba.txt | sort | awk '{print $1} ' | uniq -d | wc -l

That way, it should return 0, if it returns any higher, we have problems!
Let me know what you think about it, perhaps there is a way to make it shorter, I think it's really long...
  #5 (permalink)  
Old 09-22-2009
Scrutinizer Scrutinizer is online now
Registered User
  
 

Join Date: Nov 2008
Posts: 757
A little bit smaller:

Code:
 cut -d']' -f1 prueba2.txt|sort|uniq -d|wc -l

  #6 (permalink)  
Old 09-22-2009
MWPita MWPita is offline
Registered User
  
 

Join Date: Sep 2009
Location: Costa Rica
Posts: 3
Quote:
Originally Posted by Scrutinizer View Post
A little bit smaller:

Code:
 cut -d']' -f1 prueba2.txt|sort|uniq -d|wc -l
Nice one, at least is smaller that mine, hehe
Reply

Bookmarks

Tags
find duplicate uniq sequence script

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 06:53 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0