grep running total/ final total across multiple files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers grep running total/ final total across multiple files
# 1  
Old 05-07-2007
grep running total/ final total across multiple files

Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to :

adventure.hits:6 adverts.hits:16 blog.hits:19 books.hits:8 cds.hits:7 contact.hits:2 crime.hits:11 dvds.hits:8 fantasy.hits:9 fiction.hits:12 food.hits:10 gadgets.hits:14 games.hits:21 gardening.hits:6 health.hits:13 hotlist.hits:20 index.hits:83 jobs.hits:4 mags.hits:14 music.hits:9 news.hits:5 nonfiction.hits:16 novels.hits:9 people.hits:10 recent.hits:17 sales.hits:4 sf.hits:16 software.hits:18 sport.hits:13 test.hits:6

is there a simple way to get grep to skip all the above information and just give the final total of occurences across all of the files?

I know it's probably staring me in the face but I just cant get my head around a simple and effective way to do it.

Regards.
# 2  
Old 05-07-2007
I would use awks to maintain the totals, and print out the results in the END paragraph - need more information on the file format, and what you are grepping for to duplicate in awk.
# 3  
Old 05-07-2007
basically I have narrowed the file down into a list of IP address. then I search through each of the hit files using awk and grep

a .hit file contains an entry per line:

192.168.2.1 Tue nov 22 20:30:45 GMT 2005

The code I have so far is:

IPList is a list of IP addresses

cat IPList | while read LINE
do
TOTALHITS=`grep -c "$LINE" *.hits`
echo $TOTALHITS $LINE >> totalIPHits
done

but the output of the grep command is as shown in the original post. how would I go about awking such a statement to keep total?
# 4  
Old 05-07-2007
something like this maybe

awk '{ Array[$1]++ }
END{ for (I in Array)
{ print I, Array[I]}
}' *.hit

This will scan multiple files that would be shown with an ls *.hit listing, then, using the first field as its key, maintain a total of the hits in the Array.

then, at the End, for each value in the array, print the Array Key followed by the count.
# 5  
Old 05-07-2007
ah I'll give that a try, thanks Smilie
# 6  
Old 05-08-2007
or simply ,

cat file1 file2 file3 file4 ............... | grep -c <ip_address>
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding total distinct count from multiple csv files through UNIX script

Hi All , I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct count on all files. We can't merge all the files here as file size are huge in millions.I have tried in below way for each... (9 Replies)
Discussion started by: STCET22
9 Replies

2. Shell Programming and Scripting

Performance of calculating total number of matching records in multiple files

Hello Friends, I've been trying to calculate total number of a certain match in multiple data records files (DRs). Let say I have a daily created folders for each day since the beginning of july like the following drwxrwxrwx 2 mmsuper med 65536 Jul 1 23:59 20150701 drwxrwxrwx 2 mmsuper... (1 Reply)
Discussion started by: EAGL€
1 Replies

3. Solaris

Find the total size of multiple files

If I have a number of files in a directory, for example, test.1 test.2 test.3 abc.1 abc.2 abc.3 and I need to find the total file size of all of the test.* files, I can use du -bc test.* in Linux. However, in Solaris, du does not have the -c option. What can I do in Solaris to get... (11 Replies)
Discussion started by: learnix
11 Replies

4. UNIX for Dummies Questions & Answers

Write the total number of rows in multiple files into another file

Hello Friends, I know you all are busy and inteligent too... I am stuck with one small issue if you can help me then it will be really great. My problem is I am having some files i.e. Input.txt1 Input.txt2 Input.txt3 Now my task is I need to check the total number of rows in... (4 Replies)
Discussion started by: malaya kumar
4 Replies

5. Shell Programming and Scripting

Print available running instance out of total

Hello. I have a status command in AIX box, which provides output as below: $ status You are running the application on pegasus2 ----Program Name------|--Avail / Total---------| MQ | 1/2 | ORACLE | 10/10 | TMADMIN ... (3 Replies)
Discussion started by: panchpan
3 Replies

6. Shell Programming and Scripting

Search and find total count from multiple files

Please advice how can we search for a string say (abc) in multiple files and to get total occurrence of that searched string. (Need number of records that exits in period of time). File look like this (read as filename.yyyymmdd) a.20100101 b.20100108 c.20100115 d.20100122 e.20100129... (2 Replies)
Discussion started by: zooby
2 Replies

7. Shell Programming and Scripting

Running Total Running Wild

Hi. A shell scripting newbie here. I am trying to write a script that will create a running total of Sales, and increment a counter for each Sales entry, but when I executed the program it never stopped. counter=0 Sales=0 echo "enter sales price" read sales while do let counter=counter+1... (6 Replies)
Discussion started by: Ccccc
6 Replies

8. Shell Programming and Scripting

Calculate total space, total used space and total free space in filesystem names matching keyword

Good afternoon! Im new at scripting and Im trying to write a script to calculate total space, total used space and total free space in filesystem names matching a keyword (in this one we will use keyword virginia). Please dont be mean or harsh, like I said Im new and trying my best. Scripting... (4 Replies)
Discussion started by: bigben1220
4 Replies

9. Shell Programming and Scripting

Running Total

HPUX 11i v2 #!/bin/sh Hi all. I have a space delimited flat file of about 9000 lines. I would like to get a running total of field 3 to the variable $TOTAL. Field 3 can be formatted as listed.... $ 0.00 $2804.15 <$ 4.14> (negative) Any ideas are most appreciated!!!! TIA!! (9 Replies)
Discussion started by: lyoncc
9 Replies
Login or Register to Ask a Question

Featured Tech Videos