Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Counting number of times content in columns occurs in other files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Counting number of times content in columns occurs in other files
# 1  
Old 09-13-2016
Counting number of times content in columns occurs in other files
I need to figure out how many times a location (columns 1 and 2) is present within a group of files. I figured using a combination of 'while read' and 'grep' I could count the number of instances but its not working for me.

Code:
cat file.txt | while read line
do 
grep $line *08-new.txt | wc -l
done

The contents of column 3 are arbitrary in this case. Note that all locations in file.txt (source input) should have at least 1 instance in *08-new.txt, but there could be locations in *08-new.txt that are not in file.txt (source input). The files are titled {1981..2016}08-new.txt



file.txt (source input)
Code:
   30.68   -88.24   34
   32.30   -86.41   0.26
   35.14   -111.67   1.27
   33.43   -112.00   -0.08
   34.65   -112.43   0.63
   32.13   -110.96   0.60

198108-new.txt
Code:
   30.68   -88.24   1.14
   32.30   -86.41   0.26
   35.14   -111.67   1.27
   33.43   -112.00   -0.08
   34.65   -112.43   0.63
   32.13   -110.96   0.60

198208-new.txt
Code:
   30.68   -88.24   1.14
   32.30   -86.41   0.26
   35.14   -111.67   1.27
   30.48   -87.19   0.51
   34.65   -112.43   0.63
   32.13   -110.96   0.60

201608-new.txt
Code:
   30.68   -88.24   1.14
   32.30   -86.41   0.26
   35.14   -111.67   1.27
   33.43   -112.00   -0.08
   34.65   -112.43   0.63
   39.94   -91.19   1.80

Output
Code:
grep: invalid option -- .
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
0

Expected output
Code:
   30.68   -88.24   3
   32.30   -86.41   3
   35.14   -111.67   3
   33.43   -112.00   2
   34.65   -112.43   3
   32.13   -110.96   2

In the end I need to be able to have each location and its count (column 3) stored in a variable to be used further down the script.
# 2  
Old 09-13-2016
Here is one way to do it. If you need the values put in to a variable for later use, follow the example below of i=`command` to have it for later use. Note that this script follows your logic of counting the number of files that the string appears in, not the actual count of the number of times that it occurs.

Code:
while read line; do 
  i=`echo $line |awk '{ print $1, " ",  $2 }'`  # need to preserve the 3 spaces
  echo "$i `grep  "$i" *new.txt| wc -l`"
  done < file.txt

This User Gave Thanks to Padow1 For This Post:
ncwxpanther
# 3  
Old 09-13-2016
You can use Sort mechanism to get your desired output. Sort by keys of 1st and 2nd column, get the count and print.

Code:
sort -k1,1nr -k2,2nr file.txt *08-new.txt | uniq -c | awk '{print $2,$3,$1}

HTH
# 4  
Old 09-13-2016
Try also
Code:
awk 'NR==FNR {T[$1,$2]; next} ($1,$2) in T {T[$1,$2]++} END {for (t in T) print t, T[t]}' SUBSEP=" " file[1-4]
33.43 -112.00 2
35.14 -111.67 3
32.30 -86.41 3
34.65 -112.43 3
30.68 -88.24 3
32.13 -110.96 2

# 5  
Old 09-13-2016
Quote:
Originally Posted by RudiC
Try also
Code:
awk 'NR==FNR {T[$1,$2]; next} ($1,$2) in T {T[$1,$2]++} END {for (t in T) print t, T[t]}' SUBSEP=" " file[1-4]
33.43 -112.00 2
35.14 -111.67 3
32.30 -86.41 3
34.65 -112.43 3
30.68 -88.24 3
32.13 -110.96 2

Hello RudiC,

Thank you for nice code. I think OP has requested if there are at least one occurrences of fields 1st and 2nd from source file/Input_file(file.txt).
So if have to put a condition at last to check that else it will print those lines too which are coming only in very first/source file too.
Code:
awk 'NR==FNR {T[$1,$2]; next} ($1,$2) in T {T[$1,$2]++} END {for (t in T){if(T[t]>1){print t, T[t]}}}' SUBSEP=" "  file.txt  198108-new.txt  198208-new.txt  201608-new.txt

Output will be as follows.
Code:
35.14 -111.67 3
32.30 -86.41 3
30.68 -88.24 3
34.65 -112.43 3
33.43 -112.00 2
32.13 -110.96 2

Thanks,
R. Singh
# 6  
Old 09-14-2016
Quote:
Originally Posted by ncwxpanther
.
.
.
Note that all locations in file.txt (source input) should have at least 1 instance in *08-new.txt
.
.
.
# 7  
Old 09-15-2016
Quote:
Originally Posted by Padow1

Code:
while read line; do 
  i=`echo $line |awk '{ print $1, " ",  $2 }'`  # need to preserve the 3 spaces
  echo "$i `grep  "$i" *new.txt| wc -l`"
  done < file.txt

Thanks Padow. This worked as needed. I did not get a chance to try the other suggestions. But hope to soon.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed command to replace one value which occurs multiple times

Hi, My Input File : "MN.1.2.1.2.14.1.1" := "MN_13_TM_4" ( 000000110110100100110001111110110110101110101001100111110100011010110111001 ) "MOS.1.2.1.2.13.6.2" := "MOS_13_TM_4" ( 000000110110100100110001111110110110101110101001100111110100011010110111001 ) Like above template,I have... (4 Replies)
Discussion started by: Preeti Chandra
4 Replies

2. Homework & Coursework Questions

Accepting a phrase and counting the number of times that it is repeated in a specific website

1. The problem statement, all variables and given/known data: Develop a shell script that accepts a phrase and counts the number of times that it is repeated in a specific website. Note: Im not sure if it's the whole website, or just a specific page but im guessing its thewhole website. ... (2 Replies)
Discussion started by: Zakerii
2 Replies

3. UNIX for Dummies Questions & Answers

how to read how many times same result occurs?

Is there a way to read how many times each result occurs in a list, say the list.txt has the results: a c d a f c a d e and I need to know how many times each of them occurs such as : a=3 c=2 ..etc. (5 Replies)
Discussion started by: Iifa
5 Replies

4. Shell Programming and Scripting

Removal of a tag in the xml which occurs mutiple times

Hi, The tag can occur multiple times. I want to remove entire <SeqNum>..</SeqNum> from each line, regardless the values with in this tag. for each line value inside these tags could be different. So please suggest how to do this. Note: Each profile information is in one line. ... (2 Replies)
Discussion started by: Anusha_Reddy
2 Replies

5. Shell Programming and Scripting

Looking for a single line to count how many times one character occurs in a word...

I've been looking on the internet, and haven't found anything simple enough to use in my code. All I want to do is count how many times "-" occurs in a string of characters (as a package name). It seems it should be very simple, and shouldn't require more than one line to accomplish. And this is... (2 Replies)
Discussion started by: Shingoshi
2 Replies

6. UNIX for Dummies Questions & Answers

Counting the number of times a character appears

I am looking for a bash command that counts the number of times a character appears in a file. For example "I am a newbie, trying to learn shell script". Then the command counts the number of e and gives them as 4. Also I want one that counts the number of times a character in a string is replaced.... (2 Replies)
Discussion started by: #moveon
2 Replies

7. Shell Programming and Scripting

need help with counting of files then pass number to variable

hi all, i'm trying to pass a count of files to a variable thru these set of codes: sh_count=$(ls -1 fnd_upload_LV*.* |wc -l) problem is if no files matches that, it will give an error "ls: fnd_upload_LV*.*: No such file or directory". how do i avoid having the shell script show that... (2 Replies)
Discussion started by: adshocker
2 Replies

8. Shell Programming and Scripting

TO find the word which occurs maximum number of times

Hi Folks !!!!!!!!!!!!!!!!!!! My Requirement is............. i have a input file: 501,501.chan 502,502.anand 503,503.biji 504,504.raja 505,505.chan 506,506.anand 507,507.chan and my o/p should be chan->3 i.e. the word which occurs maximum number of times in a file should be... (5 Replies)
Discussion started by: aajan
5 Replies

9. UNIX for Dummies Questions & Answers

Counting number of files in a directory

Some simple questions from a simple man. If i wanted to count the number of files contained within a directory, say /tmp would ls -l /tmp ¦ wc -l suffice and will it be accurate? second one: How would i check the number of files with a certain string in the filename, in the same directory. ... (2 Replies)
Discussion started by: iamalex
2 Replies

10. Shell Programming and Scripting

Counting Number of times a File is accessed

Hi, I need to count the number of times a script is accessed from within the script. Is it possible ? Example: I have a script called lo.sh and i execute the script for the first time, then the counter variable declared inside the lo.sh should increment by 1. For every execution the... (1 Reply)
Discussion started by: pathanjalireddy
1 Replies
Login or Register to Ask a Question