Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Filter logs for key words and create an index

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Filter logs for key words and create an index
# 1  
Old 02-16-2013
Filter logs for key words and create an index
well, im a total novice of unix commands and shell scripting, i never made anything.

But yesterday in my work i was working with a lot of log files and i was wondering if there is a command for doing something like this:

i have a bunch of files with text like this:
Code:
blablabla errorcode1 blablabla
blablablablalblalbla errorcode2 blablabla
blablabla errorcode1 blablabla
blablablabbasb errorcode3 blablabla

is possible with a command to make a search in this files and create a new one with something like this:
Code:
Errorcode1 2
errorcode2 1
errorcode3 1

i dont know if there is a way with a command to search for certaing pattern like Errorcode* and write it in a file, but when the pattern already exist inthe new file add 1 to the number of repetitions of the errorcode finded.

is possible or the only way is to make a shell script?

thanks for all!

Moderator's Comments:
Mod Comment changed the thread title to something meaningful and added CODE-tags. Please use CODE-tags yourself for file content, terminal output, commands and the like and give your threads some meaningful title. "Please help me" is hardly descriptive at all. Thank you for your consideration.
Last edited by bakunin; 02-16-2013 at 03:42 PM..
# 2  
Old 02-16-2013
You did not tell us what is your OS, but if you have GNU grep try:
Code:
grep -o "errorcode[0-9]" file | sort | uniq -c

Last edited by Yoda; 02-16-2013 at 12:06 PM.. Reason: Wrap expression in double quotes. Thanks MadeInGermany for pointing that out.
This User Gave Thanks to Yoda For This Post:
matius_88
# 3  
Old 02-16-2013
i have aix, thanks for your reply this monday i will try that, do you know some kind of "manual" or tutorial for making this kind of commands?

thanks for all!
# 4  
Old 02-16-2013
You better quote grep expressions:
Code:
grep -o "errorcode[0-9]" file | sort | uniq -c

Proof:
Code:
touch errorcode0
echo errorcode[0-9]
echo "errorcode[0-9]"

QED

---------- Post updated at 11:12 AM ---------- Previous update was at 11:04 AM ----------

On AIX you'll need to replace grep -o "errorcode[0-9]" by sed -n 's/.*\(errorcode[0-9]\).*/\1/p'
These 2 Users Gave Thanks to MadeInGermany For This Post:
matius_88 Yoda
# 5  
Old 02-16-2013
Or you can use an awk program
Code:
awk ' {
        for(i=1;i<=NF;i++) {
                if($i ~ /errorcode[0-9]/)
                        a[$i]++;
        }
} END {
        for(i in a)
                print i, a[i];
}' file

Quote:
Originally Posted by matius_88
do you know some kind of "manual" or tutorial for making this kind of commands?
For command manuals you can refer the manual pages:

Syntax:
Code:
man <command>

E.g.
Code:
man grep

This User Gave Thanks to Yoda For This Post:
matius_88
# 6  
Old 02-17-2013
thanks a lot i have a doubt about that AWK program
when i run it i need to specify the file i want to get the info?, or it will get all the files of the dir?, i only have to copy that code and execute it or is a certain type of command to execute a AWK?

because i want to analyze 1200 log files, the file you put in the last line is the output file?, it has to be created or it will be created by this program?

in a case of a
COD-ASD-00000

in that case i have to put in pattern something like this?
Code:
COD-[A-Z][A-Z][A-Z]-[0-9][0-9][0-9][0-9][0-9](

thanks for all your help!
# 7  
Old 02-17-2013
matius,
Yes, you specifiy the file or files you want to read as input at end of awk command and then you can redirect output to a file.

Using bipinajith's awk command, here is example of reading in 3 input files in current directory and redirecting output to output.file:

Code:
awk ' {
        for(i=1;i<=NF;i++) {
                if($i ~ /errorcode[0-9]/)
                        a[$i]++;
        }
} END {
        for(i in a)
                print i, a[i];
}' COD-ASD-00000 COD-ASD-00001 COD-ASD-00002 > output.file


Here is example of reading all files in current directory that match your pattern:

Code:
awk ' {
        for(i=1;i<=NF;i++) {
                if($i ~ /errorcode[0-9]/)
                        a[$i]++;
        }
} END {
        for(i in a)
                print i, a[i];
}' COD-[A-Z][A-Z][A-Z]-[0-9][0-9][0-9][0-9][0-9] > output.file

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Files summary using awk based on index key

Hello , I have several files which are looking similar to : file01.txt keyA001 350 X string001 value001 keyA001 450 X string002 value007 keyA001 454 X string002 value004 keyA001 500 X string003 value005 keyA001 255 X string004 value006 keyA001 388 X string005 value008 keyA001 1278 X... (4 Replies)
Discussion started by: alex2005
4 Replies

2. Shell Programming and Scripting

Read log file to create Performance index

I am required to create a CSV file reading last 200000 lines form a log file. I have to grep 3 parameters from this log file and write these parameters in the .csv file, with time stamp. This script will be setup in a cron job which will run every 10 minutes. I have written the script but it is... (5 Replies)
Discussion started by: Crazy_Nix
5 Replies

3. Shell Programming and Scripting

Search string or words in logs without using Grep

I'm in need of some kind of script that will search for a string in each logfile in a directory but we don't want to use GREP. GREP seems to use up to much of our memory causing the server to use up a lot of swap space. Our log files are bigger than 500M on a daily basis. We lately started... (8 Replies)
Discussion started by: senormarquez
8 Replies

4. Shell Programming and Scripting

Filter all the lines with minimum specified length of words of a text file

Hi Can someone tell me which script will work best (in terms of speed and simplicity to write and run) for a large text file to filter all the lines with a minimum specified length of words ? A sample script with be definitely of great help !!! Thanks in advance. :) (4 Replies)
Discussion started by: my_Perl
4 Replies

5. Shell Programming and Scripting

Extract key words and print their values

Input file (HTTP request log file): GET... (2 Replies)
Discussion started by: buptwy
2 Replies

6. UNIX for Dummies Questions & Answers

create table file from different files with index

Hi, I've several files with two collumns, where first collumn can be used as index. filename1 and filename2 how to create a file I should start with cat all files and extract first collumn to create an index? (4 Replies)
Discussion started by: sargotrons
4 Replies

7. Shell Programming and Scripting

Linux Script create index.html file

I need a script that can do this: A script that searches all directories and subdirectories for .html files When a .html file is found it creates a index.html file in that folder. It then edits the index.html file and inserts links to all of the .html files that are in that folder into the... (5 Replies)
Discussion started by: seashell11
5 Replies

8. Shell Programming and Scripting

Sed filter words from string

I have following string in a variable: str="sstring garbage adfsdf tab.col1 lkad rjfj tab2.col2 adja tab2.col4 garbage" I want to filter "word.word" pattern from it. I assume the pattern won't occur in start or end of the string. Output shoud be: tab.col1 tab2.col2 tab2.col4 Can this be... (7 Replies)
Discussion started by: amicon007
7 Replies

9. Shell Programming and Scripting

Filter NOT words

Hi, I have string like this (for example) $str="orange AND grapes NOT pineapple"; $str="orange AND grapes NOT pineapple AND papaya"; $str="orange AND grapes NOT pineapple AND jackfruit NOT "fruit juice""; I have to filter out NOT terms. How can i do that? I used this... (3 Replies)
Discussion started by: vanitham
3 Replies

10. UNIX for Advanced & Expert Users

How to filter the words, if that word contains the expected letter

Hi, I am trying to filter the words from a file which contain 'abc'. But I am unable to. Could any one help me. For eg: The file contents are 123ab 12hnj1 123abc456 123cgbcahjkf23 23134abchfhj43 gc32abc abc1 2abc3 sd uiguif fhwe 21242 uh123 jkcas124d123 u3hdbh23u ffsd8 Output... (3 Replies)
Discussion started by: venu_eie
3 Replies
Login or Register to Ask a Question