grep'ing and sed'ing chunks in bash... need help on speeding up a log parser. Post: 302309278

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

grep'ing for text within a bunch of files...?

I have, say, a dozen files, and I want to grep for a string of text within them. I don't remember the exact syntax, but let me give it a shot and show you an idea here... find . -type f -exec grep thisword {} \; ...and there's a way to put more than one grep into the statement, so it will tell...

2. Shell Programming and Scripting

grep'ing for specific directories, and using the output to move files

Hello, this is probably another really simple tasks for most of you gurus, however I am trying to make a script which takes an input, greps a specific file for that input, prints back to screen the results (which are directory names) and then be able to use the directory names to move files....

3. Shell Programming and Scripting

grep'ing a file until a certain message appears

Hello, I'm writing a script that will automate the launch of some services on my AIX machine. However, some services are dependent on the successful startup of others. When I start these services manually, I usually just check a log file until I see a message that confirms a successful...

4. Shell Programming and Scripting

pipe'ing grep output to awk

This script is supposed to find out if tomcat is running or not. #!/bin/sh if netstat -a | grep `grep ${1}: /tomcat/bases | awk -F: '{print $3}'` > /dev/null then echo Tomcat for $1 running else echo Tomcat for $1 NOT running fi the /tomcat/bases is a file that...

5. Programming

Obfuscate'ing a.out ... ???

Hi all, I've search the forums regarding posts similar to this already but can't find the suitable response. Am actually looking for something very trivial I think. I just want to mask/obfuscate the a.out file and run it like a normal UNIX program. I've look at gpg and encryption but it requires...

6. Shell Programming and Scripting

grep'ing a variable that contains a metacharacter ($) with a while loop

This is driving me crazy, and I'm hoping someone can help me out with this. I'm trying to do a simple while loop to go through a log file. I'm pulling out all of the lines with a specific log line, getting an ID from that line, and once I have a list of IDs I want to loop back through the log and...

7. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ...

8. Shell Programming and Scripting

grep'ing dot history file

Hi, I tried to grep ".sh_history" (DOTsh_history) file and did not return anything though I found the word in .sh _history file through vi editor in Linux. Then I tried to grep ".profile" to check if it is the prob with hidden files and I got results. Then I verified the same with my friend...

9. Shell Programming and Scripting

Grep'ing information from a log file on SUN OS 5

Hi Guys, I'm trying to write an script that will be launched by a user. The script will look at a log file and check for alerts with the date (supplied by user) and a machine's hostname (also supplied by the user). I'm trying to get the output formatted just like the log file. The logfile looks...

10. UNIX for Dummies Questions & Answers

grep'ing a variable - why not working

Hi all, Am writing a ksh script where I am looking for processes that has gone defunct and all of which has the same PPID PID is the variable that I need to match as this is the process ID of the processes that has gone defunct Am just curious how come the following DOES NOT work? ps...

LEARN ABOUT DEBIAN

logtop

LOGTOP(1)						      General Commands Manual							 LOGTOP(1)

NAME

       logtop - Realtime log line rate analyser

SYNOPSIS

       logtop [OPTIONS]

DESCRIPTION

	logtop is a System Administrator tool analyzing line rate on stdin.
	It reads on stdin and print a constantly updated result
	displaying, in columns:
	Line number, count, frequency, and the actual line.

	 $ tail -f FILE | logtop
	is the friendly version of:
	 $ watch 'tail FILE | sort | uniq -c | sort -gr'

OPTIONS

       -s, --size=K
	      Only keep K lines in memory, instead of 10000.

       -q, --quiet
	      Do not display a live view of the data, only display a top at exit.

       -l, --line-by-line=K
	      Print result line by line, in a machine friendly format, K is the number of result to print per line.

		Line by line format is : [%d %f %s	]*

		  %d : Number of occurences
		  %f : Frequency of apparition
		  %s : String (Control chars replaced by dots.

       -i, --interval=K
	      Interval between graphical updates, in seconds. Defaults to 1.

       -h, --help
	      Show summary of options.

       -v, --version
	      Show version of program.

EXAMPLES

       Here are some logtop usage examples.

       tail -f cache.log | grep -o "HIT|MISS" | logtop

       Realtime hit / miss ratio on some caching software log file.

       tail -f access.log | cut -d' ' -f1 | logtop -s 10000

       Realtime most querying IPs on your server, as long as log lines in access.log starts with the client IP.

       tail -f access.log | cut -d' ' -f7 | logtop -s 10000

       Realtime most requested web pages in a NCSA like log file.

       cat auth.log | grep -v "CRON" | grep -o ": .*" | logtop -q -s 100000

       Display a one-shot simple analyse of your auth.log.

SEE ALSO

       watch(1)

AUTHOR

       logtop was written by Julien Palard.

       This manual page was written by Julien Palard <julien@palard.fr>, for the Debian project (and may be used by others).

								  April 16, 2011							 LOGTOP(1)