Count unique words

unique words

Login to Reply

Thread Tools Search this Thread
# 1  
Old 02-15-2017
Count unique words

Dear all,

I would like to know how to list and count unique words in thousands number of text files.

Please help me out
thanks in advance
# 2  
Old 02-15-2017
What have you tried?

Also, due to the vague nature of this request, it appears that this may be homework/classwork. If it is, there are specific rules relative to schoolwork.
These 2 Users Gave Thanks to joeyg For This Post:
imranrasheedamu (02-16-2017), rbatte1 (02-15-2017)
# 3  
Old 02-16-2017
Dear Joeyg

I have a list of thousands of text files like


So, I would like to count the of total number and unique words for all files based on fourth field of the filename.


count total and unique words

count total and unique words

count total and unique words

and so on...

Please help me

Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 02-16-2017 at 04:14 AM.. Reason: Added CODE tags.
# 4  
Old 02-16-2017
How about sth along this line?
for FN in *.txt
  do    TMP=${FN##*_}
        echo "if(filename==$TMP)"
        echo count total and unique words
count total and unique words

count total and unique words


Please note that your pseudo code is not somewhere near any real code doing what you seem to describe.
This User Gave Thanks to RudiC For This Post:
imranrasheedamu (02-16-2017)
# 5  
Old 02-16-2017
Actually Sir, I like to count words all those files whose filename contains National, Page, International, Health & Entertainment etc.
# 6  
Old 02-16-2017
So - how would you do that? And how would you handle the results?
# 7  
Old 02-16-2017
(Apologies for any typos.)
RudiC has already given you a starter with this, assume 'FN' is pointing to an Entertainment text file:-

This would give you a result inside the TMP variable, Entertainment .

So your logic would require a count for each file containing 'Entertainment'.
Similarly for the others.

So what would your logic be to obtain your count(s) per category?

You are here to learn how to do it for yourself and the best way is to attempt something no matter how bad your code looks. We are not here to ridicule your attempts but to correct your logic so that you understand what is going on and become capable of doing it again if need be.
If it is JUST the filenames you want then this will _perhaps_ help:-
ls *.txt > /your/path/to/filenames which will create a single text file with your thousands of filenames ONLY inside it.
grep is your friend here.

However if you intend to read EACH individual file to count these words also, then this is a totally different _animal_.

Last edited by wisecracker; 02-16-2017 at 07:12 AM.. Reason: Added the 'grep' line.
Login to Reply

Thread Tools Search this Thread
Search this Thread:
Advanced Search

Similar Threads More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Count unique column nans UNIX for Beginners Questions & Answers 6 05-17-2017 06:40 AM
Regex to identify unique words in a dictionary database gimley Shell Programming and Scripting 2 02-21-2017 03:09 AM
Print count of unique values H squared Shell Programming and Scripting 3 08-26-2016 07:13 AM
Count occurrence of column one unique value having unique second column value angshuman Shell Programming and Scripting 4 08-02-2016 12:27 PM
How count the number of two words associated with the two words occurring in the file? jmarx Shell Programming and Scripting 1 05-06-2014 06:12 PM
awk to count using each unique value ncwxpanther Shell Programming and Scripting 4 05-06-2014 01:08 PM
Search and count a unique string a_bahreini UNIX for Dummies Questions & Answers 6 01-29-2014 05:18 PM
Unique words in each line Viernes Shell Programming and Scripting 6 01-27-2013 02:17 PM
display unique words. shikshavarma Shell Programming and Scripting 2 04-16-2012 05:03 AM
Unique count from flat file Pratik4891 Shell Programming and Scripting 11 01-06-2012 05:35 AM
unique words in files of folder and its subfolders Dworza Homework & Coursework Questions 9 03-14-2011 04:24 AM
Shell script to find out words, replace them and count words alex83 Shell Programming and Scripting 3 12-05-2010 05:18 PM
Finding the number of unique words in a file abhikamune Shell Programming and Scripting 7 09-01-2010 08:03 AM
How to count unique strings my_Perl Shell Programming and Scripting 6 07-31-2009 05:28 AM
how to read all the unique words in a text file aditya.ece1985 Shell Programming and Scripting 5 11-30-2007 02:26 AM
All times are GMT -4. The time now is 11:40 AM.

Unix & Linux Forums Content Copyright 1993-2018. All Rights Reserved.