search of common words in set of files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting search of common words in set of files
# 1  
Old 05-31-2009
search of common words in set of files

Hi,

I have a set of simple, one columned text files (in thousands).
file1:
a
b
c
d
file 2:
b
c
d
e
and so on. There is a collection of words in another file:
b d
b c d e
I have to find out the set of words (in each row) is present or absent in the given set of files. So, the output would be in matrix form (file*set) like:
1 0
1 1
I have the following code in bash, which is working well, but it involves very high computational cost with the increase of the file and set size. Any suggestion for better checking for the words is much appreciated.

My code segment:
Code:
#!/bin/sh
rm -f feat.txt 
touch feat.txt 
rm -f tem.txt
touch tem.txt
#read the rows of set file s.txt and put into seperate files 
lables=1
while read myline
  do
   echo $myline > temp.txt
   cat temp.txt|awk '{for (i=1;i<=NF;i++) print $i}'|sort|uniq > l$lables.txt
   lables=`expr $lables + 1`
  done < s.txt
p=`wc -l s.txt| awk '{print $1}'`
q=`expr $p - 1`
a=1
c=`expr $a + $q`
while [ $a -le $c ]
  do
  rm -f a.txt
  touch a.txt
  fileno=1
  while [ $fileno -le 1000 ]
   do
     cat l$a.txt|fgrep -xvf $fileno.txt| awk '{printf ($1 " ")}' >> a.txt
     echo >>a.txt
     fileno=`expr $fileno + 1`
   done 
  cat a.txt |awk '{if ($0 != NULL) print "0"; if ($0 == null) print "1"}'>c.txt
  paste c.txt feat.txt > feat1.txt
  cat feat1.txt >feat.txt
  a=`expr $a + 1`
 done

Thank you in advance

Last edited by vidyadhar85; 05-31-2009 at 10:45 AM.. Reason: code tag added
# 2  
Old 05-31-2009
Quote:
Originally Posted by mala
I have the following code in bash, which is working well, but it involves very high computational cost with the increase of the file and set size.

You use far too many external commands where they are not encessary. They slow down the script by a large factor.
Quote:
Any suggestion for better checking for the words is much appreciated.

My code segment:
Code:
#!/bin/sh
rm -f feat.txt 
touch feat.txt 
rm -f tem.txt
touch tem.txt


You are calling four external commands (rm and touch twice each); you don't need any:

Code:
> feat.txt
> tem.txt

Quote:
Code:
#read the rows of set file s.txt and put into seperate files 
lables=1
while read myline
  do
   echo $myline > temp.txt
   cat temp.txt|awk '{for (i=1;i<=NF;i++) print $i}'|sort|uniq > l$lables.txt


You don't need cat and uniq. And tr will be faster than awk.

Code:
tr -s ' ' '\n' < temp.txt | sort -u

Quote:
Code:
   lables=`expr $lables + 1`


The shell can do integer arithmetic; you don't need expr:

Code:
lables=$(( $lables + 1 ))

Quote:
Code:
  done < s.txt
p=`wc -l s.txt| awk '{print $1}'`


You don't need awk:

Code:
p=$( wc -l < s.txt )

Quote:
Code:
q=`expr $p - 1`


Code:
q=$(( $p - 1 ))

Quote:
Code:
a=1
c=`expr $a + $q`


Code:
c=$(( $a + $q ))

Quote:
Code:
while [ $a -le $c ]
  do
  rm -f a.txt
  touch a.txt


You don't need rm and touch.

Code:
> a.txt

Get rid of all instances of cat and expr from the rest of the script.
Quote:
Code:
  fileno=1
  while [ $fileno -le 1000 ]
   do
     cat l$a.txt|fgrep -xvf $fileno.txt| awk '{printf ($1 " ")}' >> a.txt
     echo >>a.txt
     fileno=`expr $fileno + 1`
   done 
  cat a.txt |awk '{if ($0 != NULL) print "0"; if ($0 == null) print "1"}'>c.txt
  paste c.txt feat.txt > feat1.txt
  cat feat1.txt >feat.txt
  a=`expr $a + 1`
 done

Thank you in advance
# 3  
Old 05-31-2009
The script has many unnecessary calls to external commands like sort, uniq, wc, expr...
Some blocks could be rewritten as a single call to awk. For example:

Code:
while read myline
  do
   echo $myline > temp.txt
   cat temp.txt|awk '{for (i=1;i<=NF;i++) print $i}'|sort|uniq > l$lables.txt
   lables=`expr $lables + 1`
  done < s.txt

Code:
awk '{
f = "l"NR".txt"
for (i=1;i<=NF;i++) if (!a[$i]++) print $i > f
close(f)
}' s.txt

I even believe that the entire work can be done in one single awk program, without the need to call external programs, but the requirements are not very clear.
# 4  
Old 05-31-2009
Code:
awk 'FNR==NR{for(i=1;i<=NF;i++)a[$i]++;next}{if(a[$0]) print FILENAME,1
else print FILENAME,"0" }' coll file*

o/p:

HTML Code:
file1 0
file1 1
file1 1
file1 1
file2 1
file2 1
file2 1
file2 1
-Devaraj Takhellambam
# 5  
Old 05-31-2009
Thanks everybody. Your suggestions are excellent. I could improve the code in great scale. Thanks again
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search a column a return a set of words

Hi I have two files. One is a text file consisting of sentences i.e. INPUT.txt and the second file is SEARCH.txt consisting of two or three columns. I need help to write a script to search the second column of SEARCH.txt for each set of five words (blue color as set one and green color as set... (6 Replies)
Discussion started by: my_Perl
6 Replies

2. Shell Programming and Scripting

Help needed with shell script to search and replace a set of strings among the set of files

Hi, I am looking for a shell script which serves the below purpose. Please find below the algorithm for the same and any help on this would be highly appreciated. 1)set of strings need to be replaced among set of files(directory may contain different types of files) 2)It should search for... (10 Replies)
Discussion started by: Amulya
10 Replies

3. Shell Programming and Scripting

Search string within a file and list common words from the line having the search string

Hi, Need your help for this scripting issue I have. I am not really good at this, so seeking your help. I have a file looking similar to this: Hello, i am human and name=ABCD. How are you? Hello, i am human and name=PQRS. I am good. Hello, i am human and name=ABCD. Good bye. Hello, i... (12 Replies)
Discussion started by: royzlife
12 Replies

4. Shell Programming and Scripting

Extract common words from two/more csv files

I have two (or more, to make it generic) csv files. Each line contains words separated by comma. None of words have any space. The number of words per line is not fixed. Some may have one, and some may have 12... The number of lines per file is also not fixed. What I need is to find common words... (1 Reply)
Discussion started by: nick2011
1 Replies

5. Shell Programming and Scripting

Finding compound words from a set of files from another set of files

Hi All, I am completely stuck here. I have a set of files (with names A.txt, B.txt until L.txt) which contain words like these: computer random access memory computer networking mouse terminal windows All the files from A.txt to L.txt have the same format i.e. complete words in... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies

6. Shell Programming and Scripting

want to search for the words in the files

Hi Friends, I have been trying to write the script since morning and reached some where now. but i think i am stuck in the final step. please help I want to search the strings below in red in the be be searched in the directories below. How can i do that in my shell script. Thanks Adi ... (8 Replies)
Discussion started by: asirohi
8 Replies

7. Shell Programming and Scripting

Drop common lines at head/tail of a large set of files

Hi! I have a large set of pairs of text files (each pair in their own subdirectory) and each pair shares head/tail (a couple of first and last lines) but differs in the middle part. I need to delete the heads/tails and keep only the middle portions in which they differ. The lengths of heads/tails... (1 Reply)
Discussion started by: dobryden
1 Replies

8. UNIX for Dummies Questions & Answers

how to find common words and take them out from two files

Hi, everyone, Let's say, we have xxx.txt A 1 2 3 4 5 C 1 2 3 4 5 E 1 2 3 4 5 yyy.txt A 1 2 3 4 5 B 1 2 3 4 5 C 1 2 3 4 5 D 1 2 3 4 5 E 1 2 3 4 5 First I match the first column I find intersection (A,C, E), then I want to take those lines with ACE out from yyy.txt, like A 1... (11 Replies)
Discussion started by: kaixinsjtu
11 Replies

9. Shell Programming and Scripting

Search files that all contain 4 specific words

Hello, I want an one line command that brings me back all the files in a folder that contain 4 specific words anywhere inside them. I want to use find,xargs and grep. for example i know for one word the command would be: find . | xargs grep 'Word1' But i don't know for 4 specific words... (13 Replies)
Discussion started by: WoodenSword
13 Replies
Login or Register to Ask a Question