Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Filter logs for key words and create an index

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Filter logs for key words and create an index
# 8  
Old 02-17-2013
Quote:
Originally Posted by matius_88
thanks a lot i have a doubt about that AWK program
when i run it i need to specify the file i want to get the info?, or it will get all the files of the dir?, i only have to copy that code and execute it or is a certain type of command to execute a AWK?

because i want to analyze 1200 log files, the file you put in the last line is the output file?, it has to be created or it will be created by this program?

in a case of a
COD-ASD-00000

in that case i have to put in pattern something like this?
Code:
COD-[A-Z][A-Z][A-Z]-[0-9][0-9][0-9][0-9][0-9](

thanks for all your help!
Do you want a single output file that counts the number of occurrences of "errorcode<digit>" in all of the COD-* files, or do you want an output file for each COD-* file?

Do you want to count occurrences of "errorcode" followed by a single digit, or occurrences of "errorcode" followed by a string of one or more digits?

With more than a thousand files to be processed, you may have to worry about limits on the length of arguments passed to a command. Are all of the COD-* files in a single directory? If so, are there any subdirectories in that directory?
# 9  
Old 02-18-2013
i want to count each COD-ABC-00000 of 1025 log files

i have to specify each file??
thanks for all
# 10  
Old 02-18-2013
You didn't answer most of my questions. But, here is a wild guess that may produce something you can use as a template.
Code:
awk '
# Print totals for a given file
# Usage: p(filename)
function p(fn) {
        # There will not be a filename the first time this function is called.
        if(fn == "") return
        # Print a header for this filename
        print "File " f ":"
        # Print the accumulated counts for this file and add them to the array
        # accumulating the sum for all files.
        for(i in c) {
                print i, c[i]
                t[i] += c[i]
                delete c[i]
        }
}
FNR == 1 {
        # This is run for the 1st record in each input file.
        # Print the results for the previous file.
        p(f)
        # Save the filename for the new file.
        f = FILENAME
}
{       # For each input field in each line, if the field matches the string
        # "errorcode" immediately followed by a digit, increment a counter for
        # the number of times we have seen this field in this file.
        for(i=1;i<=NF;i++) if($i ~ /errorcode[0-9]/) c[$i]++
}
END {   # We have found the end of all of the input files, print the results
        # from the last file.
        p(f)
        # And, print the number of times each matching field appeared in all of
        # the input files.
        if(f != "") {
                print "Total:"
                for(i in t) print i, t[i]
        }
}' COD-[A-Z][A-Z][A-Z]-[0-9][0-9][0-9][0-9][0-9] > output

As always, if you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk or nawk instead of awk.

This script will print the number of times each errorcode appears in each input file and the number of times each errorcode appears in all of the input files combined. If this fails with a diagnostic indicating that the argument list is too long, you will need to use something to split the argument list into smaller chunks and make the awk script smarter if you want to produce the list showing the number of times each errorcode appears in the combined files.

When tested with ten files containing a copy of your sample input file, it produces the output:
Code:
File COD-ABC-00001:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00002:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00003:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00004:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00005:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00006:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00007:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00008:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00009:
errorcode1 2
errorcode2 1
errorcode3 1
File COD-ABC-00010:
errorcode1 2
errorcode2 1
errorcode3 1
Total:
errorcode1 20
errorcode2 10
errorcode3 10

Last edited by Don Cragun; 02-18-2013 at 08:19 AM.. Reason: I copied the wrong output file... This edit shows the output this script will produce.
This User Gave Thanks to Don Cragun For This Post:
matius_88
# 11  
Old 02-18-2013
thanks for all the information!!!! Smilie i already make it work! when i was making it i have this questions they are not "needed" for the answer but i cant find information about it

- its there a way use a functionr or search for a bracket? "(" or search strings with Space in it like "The string" without changing the field separator??

- its there a way to return a xls that when you click it you can see the information formatted? because when i open the result the xls opens for example something like this:
Code:
a 1
b 1
c 1

but all in the same cell i wanted the name in one cell and the result of the count in other.

- its there a way to get a couple of awk scripts and run them together executing a file? (something like spools when you execute a file and the file executes de functions in orther)

and the last one!, can you get two file inputs or two/more files for outputs?, or use diferent sections of the script for one file or another?
for example i get two inputs, i use one of them in certain part of the script and other in the other part, and i put some results in one output and some other results in the other.

sorry for asking so much i have all internet blocked in my work and this is the only web i can access and im learning a lot because of you!

thanks a lot!!!

tomorrow i will post my script, to bad a switch cant be done in awk
# 12  
Old 02-19-2013
Quote:
Originally Posted by matius_88
thanks for all the information!!!! Smilie i already make it work! when i was making it i have this questions they are not "needed" for the answer but i cant find information about it

- its there a way use a functionr or search for a bracket? "(" or search strings with Space in it like "The string" without changing the field separator??
Yes. How to do so depends on what you are trying to do.
Quote:
Originally Posted by matius_88
- its there a way to return a xls that when you click it you can see the information formatted? because when i open the result the xls opens for example something like this:
Code:
a 1
b 1
c 1

but all in the same cell i wanted the name in one cell and the result of the count in other.
The awk utility is intended to process text files (both for input and output). I believe that XLS files are binary files; not text files. But I believe the programs that process XLS files can export and import CSV files that can be read and written by awk.
Quote:
Originally Posted by matius_88
- its there a way to get a couple of awk scripts and run them together executing a file? (something like spools when you execute a file and the file executes de functions in orther)
I think we have a language barrier. Obviously, you can have awk run one script that reads an input file, produces output that will be fed into another awk running another script, and have that awk script produce another output file (or files).
Quote:
Originally Posted by matius_88
and the last one!, can you get two file inputs or two/more files for outputs?, or use diferent sections of the script for one file or another?
for example i get two inputs, i use one of them in certain part of the script and other in the other part, and i put some results in one output and some other results in the other.
The script I gave you processes hundreds of input files. It could easily have produced one or more output files for each input file it was given to process.
Having two input files to awk is very common and you can easily find dozens of examples in this forum. Have you tried reading the man page for awk on your system?
Quote:
Originally Posted by matius_88
sorry for asking so much i have all internet blocked in my work and this is the only web i can access and im learning a lot because of you!

thanks a lot!!!

tomorrow i will post my script, to bad a switch cant be done in awk
I have no idea what you mean by "a switch can't be done in awk". It is easy to make awk scripts produce different outputs based on data found in an input file, based on arguments passed to awk when it is started, and based on values read from environment variables.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Files summary using awk based on index key

Hello , I have several files which are looking similar to : file01.txt keyA001 350 X string001 value001 keyA001 450 X string002 value007 keyA001 454 X string002 value004 keyA001 500 X string003 value005 keyA001 255 X string004 value006 keyA001 388 X string005 value008 keyA001 1278 X... (4 Replies)
Discussion started by: alex2005
4 Replies

2. Shell Programming and Scripting

Read log file to create Performance index

I am required to create a CSV file reading last 200000 lines form a log file. I have to grep 3 parameters from this log file and write these parameters in the .csv file, with time stamp. This script will be setup in a cron job which will run every 10 minutes. I have written the script but it is... (5 Replies)
Discussion started by: Crazy_Nix
5 Replies

3. Shell Programming and Scripting

Search string or words in logs without using Grep

I'm in need of some kind of script that will search for a string in each logfile in a directory but we don't want to use GREP. GREP seems to use up to much of our memory causing the server to use up a lot of swap space. Our log files are bigger than 500M on a daily basis. We lately started... (8 Replies)
Discussion started by: senormarquez
8 Replies

4. Shell Programming and Scripting

Filter all the lines with minimum specified length of words of a text file

Hi Can someone tell me which script will work best (in terms of speed and simplicity to write and run) for a large text file to filter all the lines with a minimum specified length of words ? A sample script with be definitely of great help !!! Thanks in advance. :) (4 Replies)
Discussion started by: my_Perl
4 Replies

5. Shell Programming and Scripting

Extract key words and print their values

Input file (HTTP request log file): GET... (2 Replies)
Discussion started by: buptwy
2 Replies

6. UNIX for Dummies Questions & Answers

create table file from different files with index

Hi, I've several files with two collumns, where first collumn can be used as index. filename1 and filename2 how to create a file I should start with cat all files and extract first collumn to create an index? (4 Replies)
Discussion started by: sargotrons
4 Replies

7. Shell Programming and Scripting

Linux Script create index.html file

I need a script that can do this: A script that searches all directories and subdirectories for .html files When a .html file is found it creates a index.html file in that folder. It then edits the index.html file and inserts links to all of the .html files that are in that folder into the... (5 Replies)
Discussion started by: seashell11
5 Replies

8. Shell Programming and Scripting

Sed filter words from string

I have following string in a variable: str="sstring garbage adfsdf tab.col1 lkad rjfj tab2.col2 adja tab2.col4 garbage" I want to filter "word.word" pattern from it. I assume the pattern won't occur in start or end of the string. Output shoud be: tab.col1 tab2.col2 tab2.col4 Can this be... (7 Replies)
Discussion started by: amicon007
7 Replies

9. Shell Programming and Scripting

Filter NOT words

Hi, I have string like this (for example) $str="orange AND grapes NOT pineapple"; $str="orange AND grapes NOT pineapple AND papaya"; $str="orange AND grapes NOT pineapple AND jackfruit NOT "fruit juice""; I have to filter out NOT terms. How can i do that? I used this... (3 Replies)
Discussion started by: vanitham
3 Replies

10. UNIX for Advanced & Expert Users

How to filter the words, if that word contains the expected letter

Hi, I am trying to filter the words from a file which contain 'abc'. But I am unable to. Could any one help me. For eg: The file contents are 123ab 12hnj1 123abc456 123cgbcahjkf23 23134abchfhj43 gc32abc abc1 2abc3 sd uiguif fhwe 21242 uh123 jkcas124d123 u3hdbh23u ffsd8 Output... (3 Replies)
Discussion started by: venu_eie
3 Replies
Login or Register to Ask a Question