Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Detect duplicated words in file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Detect duplicated words in file
# 8  
Old 08-10-2010
Gadba,
Let me know which unix flavor you are working on?
Do you get anything when you do...

Code:
tr -s "[ \t]" "\n" <  file

# 9  
Old 08-10-2010
Quote:
Originally Posted by rajamadhavan
Gadba,
Let me know which unix flavor you are working on?
Do you get anything when you do...

Code:
tr -s "[ \t]" "\n" <  file

$ tr -s "[ \t]" "\n" < try1.txt
etst test1 fdsfsdfds
test1
# 10  
Old 08-10-2010
Guess you are using solaris or similar which isn't honouring the square brackets.

You can either use
Code:
tr -s " " "\n" < try1.txt

or

tr -s "\t" "\n" < try1.txt

based on your column separator on your file...(space/tab)
# 11  
Old 08-11-2010
Quote:
Originally Posted by rajamadhavan
Guess you are using solaris or similar which isn't honouring the square brackets.

You can either use
Code:
tr -s " " "\n" < try1.txt
 
or
 
tr -s "\t" "\n" < try1.txt

based on your column separator on your file...(space/tab)
$ tr -s " " "\n" < try1.txt
etst
test1
fdsfsdfds
test1
$ tr -s "\t" "\n" < try1.txt
etst test1 fdsfsdfds
test1
$

I did not get the duplicates picked up.
# 12  
Old 08-11-2010
Try...
Code:
awk '{for(i=1;i<=NF;i++)a[$i]++}END{for(i in a)if(a[i]>1)print i}' try1.txt

# 13  
Old 08-11-2010
For picking the duplicates you must run the entire command..

Code:
tr -s " " "\n" < try1.txt | sort | uniq -d

# 14  
Old 08-12-2010
Thanks Rajamahavan, Ygor, both work in Solaris. and thanks all of the rest.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replace particular words in file based on if finds another words in that line

Hi All, I need one help to replace particular words in file based on if finds another words in that file . i.e. my self is peter@king. i am staying at north sydney. we all are peter@king. How to replace peter to sham if it finds @king in any line of that file. Please help me... (8 Replies)
Discussion started by: Rajib Podder
8 Replies

2. Shell Programming and Scripting

Join files, omit duplicated records from one file

Hello I have 2 files, eg more file1 file2 :::::::::::::: file1 :::::::::::::: 1 fromfile1 2 fromfile1 3 fromfile1 4 fromfile1 5 fromfile1 6 fromfile1 7 fromfile1 :::::::::::::: file2 :::::::::::::: 3 fromfile2 5 fromfile2 (4 Replies)
Discussion started by: CHoggarth
4 Replies

3. Shell Programming and Scripting

Deleting duplicated chunks in a file using awk/sed

Hi all, I'd always appreciate all helps from this site. I would like to delete duplicated chunks of strings on the same row(?). One chunk is comprised of four lines such as: path name starting point ending point voltage number I would like to delete duplicated chunks on the same... (5 Replies)
Discussion started by: jypark22
5 Replies

4. Shell Programming and Scripting

How to remove duplicated column in a text file?

Dear all, How can I remove duplicated column in a text file? Input: LG10_PM_map_19_LEnd 1000560 G AA AA AA AA AA GG LG10_PM_map_19_LEnd 1005621 G GG GG GG AA AA GG LG10_PM_map_19_LEnd 1011214 A AA AA AA AA GG GG LG10_PM_map_19_LEnd 1011673 T TT TT TT TT CC CC... (1 Reply)
Discussion started by: huiyee1
1 Replies

5. UNIX for Dummies Questions & Answers

Replace the words in the file to the words that user type?

Hello, I would like to change my setting in a file to the setting that user input. For example, by default it is ONBOOT=ON When user key in "YES", it would be ONBOOT=YES -------------- This code only adds in the entire user input, but didn't replace it. How do i go about... (5 Replies)
Discussion started by: malfolozy
5 Replies

6. Shell Programming and Scripting

How count the number of two words associated with the two words occurring in the file?

Hi , I need to count the number of errors associated with the two words occurring in the file. It's about counting the occurrences of the word "error" for where is the word "index.js". As such the command should look like. Please kindly help. I was trying: grep "error" log.txt | wc -l (1 Reply)
Discussion started by: jmarx
1 Replies

7. UNIX for Dummies Questions & Answers

Sort csv file by duplicated column value

hello, I have a large file (about 1gb) that is in a file similar to the following: I want to make it so that I can put all the duplicates where column 3 (delimited by the commas) are shown on top. Meaning all people with the same age are listed at the top. The command I used was ... (3 Replies)
Discussion started by: jl487
3 Replies

8. Shell Programming and Scripting

Splitting concatenated words in input file with words from the same file

Dear all, I am working with names and I have a large file of names in which some words are written together (upto 4 or 5) and their corresponding single forms are also present in the word-list. An example would make this clear annamarie mariechristine johnsmith johnjoseph smith john smith... (8 Replies)
Discussion started by: gimley
8 Replies

9. Shell Programming and Scripting

Splitting Concatenated Words in Input File with Words from a Master File

Hello, I have a complex problem. I have a file in which words have been joined together: Theboy ranslowly I want to be able to correctly split the words using a lookup file in which all the words occur: the boy ran slowly slow put child ly The lookup file which is meant for look up... (21 Replies)
Discussion started by: gimley
21 Replies

10. Shell Programming and Scripting

Pattern matching in Duplicated file and print once

Dear Experts, I have many alarms appeared in a file twice, i want to grep them with this info EVTTIME & DOMAIN, and print them in second file with 1 occurance. I have tried uniq -d test.txt > newfile and awk '!arr++' test.txt > newfile both are not working Please help me with this!!! ... (1 Reply)
Discussion started by: Danish Shakil
1 Replies
Login or Register to Ask a Question