Grep bunch of gzip files to count based on category


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Grep bunch of gzip files to count based on category
# 1  
Old 09-12-2011
Grep bunch of gzip files to count based on category

Started using unix commands recently.
I have 50 gzip files. I want to grep each of these files for a line count based particular category in column 3. How can I do that?

For example
Code:
Sr.No Date    City Description  Code  Address
1       06/09   NY  living here   0909  10st st nyc
2       07/09   NY    not here     1312   11 st st nyc
3       03.09    DC     gone          1321   232 st DC

I want to count all the cities for each of the 50 files. how do I do that?
Also want to output the city count for each of the file.

Last edited by Franklin52; 09-14-2011 at 04:05 AM.. Reason: Please use code tags for code and data samples, thank you
# 2  
Old 09-12-2011
See if this will help you:
Code:
#!/usr/bin/ksh
cut -d' ' -f3 Inp_File | sort -u |
while read mCity; do
  mTot=$(grep -c ${mCity} Inp_File)
  echo "City ${mCity} count ${mTot}"
done

# 3  
Old 09-12-2011
Code:
cd /path/to/gzfiles
for fname in *.gz
do
  echo "$fname ---------"
  gzcat $fname | awk ' {arr[$]++} ; END{for (i in arr) {print i, arr[i]   } }
  echo " "
done  > outputfile.txt

# 4  
Old 09-12-2011
Shell_life, I just forgot to mention there is no header in the file. That was just an example.Can this be represented in the form of column?

Last edited by jinxx; 09-12-2011 at 05:47 PM.. Reason: incorrect info
# 5  
Old 09-13-2011
Code:
$ for i in *.gz;do echo $i;gzip -dc $i | awk '{print $3|"sort"}'|uniq -c; done

This User Gave Thanks to jayan_jay For This Post:
# 6  
Old 09-13-2011
Thanks people!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split files based on row delimiter count

I have a huge file (around 4-5 GB containing 20 million rows) which has text like: <EOFD>11<EOFD>22<EORD>2<EOFD>2222<EOFD>3333<EORD>3<EOFD>44<EOFD>55<EORD>66<EOFD>888<EOFD>9999<EORD> Actually above is an extracted file from a Sql Server with each field delimited by <EOFD> and each row ends... (8 Replies)
Discussion started by: amvip
8 Replies

2. Shell Programming and Scripting

Category and count with awk

I want to categorize and count the as below: Input file: A1 G1 C1 F1 A2 G1 C1 F1 A3 G1 C1 F2 A4 G1 C2 F2 A7 G1 C2 F2 A8 G1 C2 F3 A11 G1 C2 F3 A23 G1 C2 F3 B4 G1 C2 F3 AC4 G2 C3 F4 B6 G2 C4 F4 BB5 G2 C4 F4 A25 G2 C5 F4 B13 G2 C5 F5 D12 G2 C5 F5 D2 G2 C5 F5 (3 Replies)
Discussion started by: aydj
3 Replies

3. Shell Programming and Scripting

Inserting column data based on category assignment

please help with the following. I have 4 col data .. instrument , category, variable and value. the instruments belong to particular categories and they all measure some variables (var1 and var2 in this example), the last column is the value an instrument outputs for a variable. I have used... (0 Replies)
Discussion started by: ritakadm
0 Replies

4. Shell Programming and Scripting

Total count in each category for given file list

I have list of file names in filename.txt below is file format >>File1 _________________________ 01~12345~Y~YES~aaaaa~can 02~23456~N~NO~bbbbb~can . . . 99~23__________________________ Need to find total count from each file depending on specific string and add them to have total count... (17 Replies)
Discussion started by: santoshdrkr
17 Replies

5. Shell Programming and Scripting

Moving the files based on count and time.

Hi, I have a requirement ,let us say 1000 files needs to be transferred in an hour from one path to another path and if the files (1000 files) are transferred within an hour ( say 40 mins), then the process should remain idle for the remaining time ( 20 mins). (3 Replies)
Discussion started by: Asaikarthik
3 Replies

6. UNIX for Advanced & Expert Users

How to perform Grep on many Gzip files, Searching for Specific information

Hello, I am wondering if you can assist with my question and ask kindly for this. I have a number of files that are listed as file1.gz through file100.gz. I am trying to perform a grep on the files and find a specific date that only resides within within one of the files. There are... (3 Replies)
Discussion started by: legharb
3 Replies

7. UNIX for Advanced & Expert Users

grep count across multiple files

I have a number of simulation log files and I want to get a total count of the "PASSED" expression in them. If I use grep -c <files>, grep would give a tally for each file. I just want one number, the total count. How do I do that? (4 Replies)
Discussion started by: CrunchMunch
4 Replies

8. Shell Programming and Scripting

Count of files based on date?

Hi Friends, Can anyone help me with this: To get the count of files that are existing in a directory created on a perticular date like in the example (01/08) .(having same pattern for the filename) ex: FileName Creted Date FILE001 01/08/2007 FILE005 ... (6 Replies)
Discussion started by: sbasetty
6 Replies

9. Shell Programming and Scripting

Grep, count and match two files

I am writing the below script to do a grep and count number of occurances between two tab delimited files. I am trying to achieve.. 1) Extract column 2 and column 3 from the S.txt file. Put it in a temp pattern file 2) Grep and count column 2 in D.txt file 3) Compare the counts between... (19 Replies)
Discussion started by: madhunk
19 Replies

10. UNIX for Dummies Questions & Answers

grep'ing for text within a bunch of files...?

I have, say, a dozen files, and I want to grep for a string of text within them. I don't remember the exact syntax, but let me give it a shot and show you an idea here... find . -type f -exec grep thisword {} \; ...and there's a way to put more than one grep into the statement, so it will tell... (1 Reply)
Discussion started by: kitykity
1 Replies
Login or Register to Ask a Question