Optimizing search using grep


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Optimizing search using grep
# 1  
Old 01-30-2016
Optimizing search using grep

I have a huge log file close to 3GB in size.

My task is to generate some reporting based on # of times something is being logged.

I need to find the number of time StringA , StringB , StringC is being called separately.

What I am doing right now is:

Code:
grep "StringA" server.log | wc -l
grep "StringB" server.log | wc -l
grep "StringC" server.log | wc -l

This is a long process and my script takes close to 10 minutes to complete. What I want to know is that whether this can be optimized or not ? Is is possible to run one grep command and find out the number of time StringA, StringB and StringC has been called individually ?
# 2  
Old 01-30-2016
This much I understand, you do not want to read the log file three times and that's what you got so far.

Are you tallying the result as StringA_count + StringB_count + StringC ?
If StringA and StringB or any combination is in the same line, does it count as two, three, one or none?

By the way, grep doesn't need wc -l to tell the count grep -c will do
# 3  
Old 01-30-2016
StringA can occur along with StringB
StringA can occur along with StringC

StringB and StringC cannot occur along in the same line.
# 4  
Old 01-30-2016
Still you haven't said about the output.

Let's assume your log has the following:

Code:
cat junaid_subhani.log

Code:
StringA and StringB and StringC counts as none
StringA and StringB counts as two times
StringB and StringA counts as two times
StringA alone counts as one time
StringB and StringC counts as none
StringB alone counts as one time
StringC alone counts as one time
StringC and StringB counts as none

variable total equals seven. Is this correct?

If not, please, post a representative portion of your log file and current script.
# 5  
Old 01-31-2016
Code:
awk '
/StringA/ { a++ }
/StringB/ { b++ }
/StringC/ { c++ }
END {
  print a+0
  print b+0
  print c+0
}
' server.log

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to use a grep search to search for a specific string within multiple directories?

Lets say I have a massive directory which is filled with other directories all filled with different c++ scripts and I want a listing of all the scripts that contain the string: "this string". Is there a way to use a grep search for that? I tried: grep -lr "this string" * but I do not... (3 Replies)
Discussion started by: Circuits
3 Replies

2. UNIX for Beginners Questions & Answers

Grep/awk using a begin search pattern and end search pattern

I have this fileA TEST FILE ABC this file contains ABC; TEST FILE DGHT this file contains DGHT; TEST FILE 123 this file contains ABC, this file contains DEF, this file contains XYZ, this file contains KLM ; I want to have a fileZ that has only (begin search pattern for will be... (2 Replies)
Discussion started by: vbabz
2 Replies

3. Shell Programming and Scripting

Grep for search.

hi, i got files delimited by |^ character. eg. apple|^ball|^cat|^dog|^egg i need to find if the above file contains | (pipe ) character inside the data.eg. apple|^ball|^cat|^d|og|^egg. above file contains | character in the dog data. how can i search that. (4 Replies)
Discussion started by: krk
4 Replies

4. Shell Programming and Scripting

Re: grep exact search

Hi, x=GATE ps -ef|grep pmon grid 552 1 0 Aug08 ? 00:00:51 asm_pmon_+ASM1 oracle 4314 1 0 Aug08 ? 00:04:08 ora_pmon_GATE1 oracle 5018 1351 0 10:14 pts/1 00:00:00 grep pmon oracle 7329 1 0 Aug08 ? 00:02:19 ora_pmon_NRID1 ps... (3 Replies)
Discussion started by: rcc50886
3 Replies

5. Shell Programming and Scripting

search for a pattern using grep

Hi I am facing the below problem. I have set of lines in which i have to search for only the line which matches with the pattren "/" only. input:- /*+ some text */ /*+ some text */ /* Remove rows from a table of survey results. */ /* Add a survey respondent's name and answers. */ /*... (7 Replies)
Discussion started by: manasa_vs
7 Replies

6. Shell Programming and Scripting

Grep Search for both First and last word

Hi Pretty sure this is very simple hence im banging my head o the wall as to why i cant get this to work: Im greping against many files where there will be the following string: Date: Thu, Aug 23 2001 09:27 PM (of course values such as date ,time and PM/AM will vary.) Im trying to grep... (1 Reply)
Discussion started by: duonut
1 Replies

7. UNIX for Dummies Questions & Answers

Speeding/Optimizing GREP search on CSV files

Hi all, I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min). Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search. Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies

8. Shell Programming and Scripting

search using grep

Hi all, I have a folder with many files. I need to get all the filenames with the value "<f id=audio-time>*:*:*</f>" word. * may be any integer. I used find . -type f -exec grep -H '<f id=audio-time>.:.</f>' {} \; command to get. but i am not able to find the correct values. Any... (2 Replies)
Discussion started by: ananthi_ku
2 Replies

9. Shell Programming and Scripting

how to search -n in a file with grep

can anybody let me know how can i search for "-n" in a text file; or if my file name is "-n" (which is legal in Unix) how do i deal with that? thanks (2 Replies)
Discussion started by: lamilami
2 Replies

10. Shell Programming and Scripting

search pattern by grep

hai folks, i am vijay very new to this website. My query: Search patterns from root directory to all directories by using grep (3 Replies)
Discussion started by: vijaysabari
3 Replies
Login or Register to Ask a Question