Counting number of files that contain words stored in another file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Counting number of files that contain words stored in another file
# 1  
Old 01-28-2011
Counting number of files that contain words stored in another file

Hi All,

I have written a script on this but it does not do the requisite job. My requirement is this:

1. I have two kinds of files each with different extensions. One set of files are *.dat (6000 unique DAT files all in one directory) and another set *.dic files (6000 unique DIC files in all in the same directory where DAT files are located)

2. The files only contain words all in new lines. For example:
1.dat contains something like this:
Code:
computer
red
apple
orange

1.dic looks like this:
Code:
computer
apple
red
blue

3. For every corresponding DAT file there is a DIC file. For 1.dat, I have 1.dic, 2.dat and 2.dic .......6000.dat and 6000.dic

4. What I want to do is to read every word from DIC files and search in all DAT files and find the number of DAT files that contain that word from the DIC file and store the result in FIL files. This means I have to only count once in the DAT files even if that word appears several times in that DAT file. For example:
1.dic contains 10 words, I read every word from 1.dic line by line and search in all DAT files as to how many DAT files contain that word from 1.dic. Then I write the result (i.e. count values) in every line in 1.fil. Similarly, I read every word in 2.dic line by line, search words in all DAT files and write the count values in 2.fil. My 2.fil should look something like this:
Code:
2
3
1
3

i.e word in the first line (of 2.dic) appears 2 times in all the DAT files (counting that word only once in all DAT files even if one DAT file contains that word several times). Same thing has to be done with all the 6000 DIC files.

What I have done so far:

Code:
for DAT in *.dat
do
for DIC in *.dic
do
while read word
CNT=$(basename "$DAT" .dat).fil
DIC=$(basename "$DAT" .dat).dic
grep -il "$word" | find . | wc -l $DIC $DAT > $FIL
done
done

# 2  
Old 01-28-2011
Bumping up posts or double posting is not permitted in these forums.

Please read the rules, which you agreed to when you registered, if you have not already done so.

You may receive an infraction for this. If so, don't worry, just try to follow the rules more carefully. The infraction will expire in the near future

Thank You.

The UNIX and Linux Forums.

Please continue here:

https://www.unix.com/shell-programmin...ther-file.html
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How count the number of two words associated with the two words occurring in the file?

Hi , I need to count the number of errors associated with the two words occurring in the file. It's about counting the occurrences of the word "error" for where is the word "index.js". As such the command should look like. Please kindly help. I was trying: grep "error" log.txt | wc -l (1 Reply)
Discussion started by: jmarx
1 Replies

2. Shell Programming and Scripting

Eliminating words from a file through ngrams stored in another file

Hello, I have a large data file which contains a huge amount of garbage i.e. words which do not exist in the language. An example will make this clear: kpaware nlupset rrrbring In other words these words are invalid in English and constitute garbage in the data. I have identified such... (2 Replies)
Discussion started by: gimley
2 Replies

3. Shell Programming and Scripting

Counting occurrences of all words in multiple files

Hey Unix gurus, I would like to count the number occurrences of all the words (regardless of case) across multiple files, preferably outputting them in descending order of occurrence. This is well beyond my paltry shell scripting ability. Researching, I can find many scripts/commands that... (4 Replies)
Discussion started by: twjolson
4 Replies

4. Shell Programming and Scripting

Counting occurrence of all words in a file

Hi, Given below is the input file: http://i53.tinypic.com/2vmvzb8.png Given below is what the output file should look like: http://i53.tinypic.com/1e6lfq.png I know how to count the occurrence of 1 word from a file, but not all of them. Can someone help please? An explanation on the... (1 Reply)
Discussion started by: r4v3n
1 Replies

5. Shell Programming and Scripting

Help in counting the no of repeated words with count in a file

Hi Pls help in solving my doubt.Iam having file like below file1.txt priya jenny jenny priya raj radhika priya bharti bharti Output required: I need a output like count of repeated words with name for ex: priya 3 jenny 2 (4 Replies)
Discussion started by: bha148
4 Replies

6. Programming

Counting the words in a file

Please find the below program. It contains the purpose of the program itself. /* Program : Write a program to count the number of words in a given text file */ /* Date : 12-June-2010 */ # include <stdio.h> # include <stdlib.h> # include <string.h> int main( int argc, char *argv ) {... (6 Replies)
Discussion started by: ramkrix
6 Replies

7. UNIX for Dummies Questions & Answers

counting words then amending to a file

i want to count the number of words in a file and then redirect this to a file echo 'total number of words=' wc -users>file THis isnt working, anyone any ideas. (1 Reply)
Discussion started by: iago
1 Replies

8. UNIX for Dummies Questions & Answers

count the number of files which have a search string, but counting the file only once

I need to count the number of files which have a search string, but counting the file only once if search string is found. eg: File1: Please note that there are 2 occurances of "aaa" aaa bbb ccc aaa File2: Please note that there are 3 occurances of "aaa" aaa bbb ccc... (1 Reply)
Discussion started by: sudheshnaiyer
1 Replies

9. UNIX for Dummies Questions & Answers

Counting number of files in a directory

Some simple questions from a simple man. If i wanted to count the number of files contained within a directory, say /tmp would ls -l /tmp ¦ wc -l suffice and will it be accurate? second one: How would i check the number of files with a certain string in the filename, in the same directory. ... (2 Replies)
Discussion started by: iamalex
2 Replies

10. Shell Programming and Scripting

Counting words in a file

I'm trying to figure out a way to count the number of words in the follwing file: cal 2002 > file1 Is there anyway to do this without using wc but instead using the cut command? (1 Reply)
Discussion started by: r0mulus
1 Replies
Login or Register to Ask a Question