gzip to read logs


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers gzip to read logs
# 1  
Old 06-04-2010
gzip to read logs

Ok here is my trouble. I run scripts whenever I am investigating problems. One of those scripts simply goes to a log folder and runs

Code:
gzip -cd * | grep error | wc -l

This tells me the number of errors in their log's and how much I should pay attention to them.

The problem is sometimes the number of files in this folder to ridiculously long and gzip just spews out argument list too long and the command won't run. Or occasionally there is a stray file in there that isn't a compressed file and therefore it won't run.

Is there any way I can tell gzip to simply use the last 100 or so compressed files rather than just saying *

It would be nice if I could just tail -100 the directory and say use these files but I can't seem to make that work.
# 2  
Old 06-04-2010
Hi.

The command ls provides a list of files. So:
Code:
gzip -cd $( ls -1 *.gz  | tail -100 ) ...

would first filter for gz files, placing them in a single column ("1" is one, not "L"), then the final 100 would be provided to gzip as a source of input file names for it.

Try it with 3 instead of 100 to test it ... cheers, drl
# 3  
Old 06-06-2010
side track on your problem, you can also use gzcat to read the files...


Code:
gzcat

# 4  
Old 06-06-2010
Hi, MrEddy.

Sorry, I over-looked the problem with too many files. The usual way to avoid the argument list too long issue is to use find. Here is a complete example:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate limiting a long list files.

# Uncomment to run script as external user.
# export PATH="/usr/local/bin:/usr/bin:/bin"
# Infrastructure details, environment, commands for forum posts. 
set +o nounset
pe() { for i;do printf "%s" "$i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe ; pe "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
pe "(Versions displayed with local utility \"version\")"
c=$( ps | grep $$ | awk '{print $NF}' )
version >/dev/null 2>&1 && s=$(_eat $0 $1) || s=""
[ "$c" = "$s" ] && p="$s" || p="$c"
version >/dev/null 2>&1 && version "=o" $p find gzip
set -o nounset

# Remove and create a few *.gz files.
# Make all files except t7 contain string "error".

rm -f t? *.gz

# Display source files.
pe
pe " generic.txt:"
cat generic.txt

pe
pe " plain.txt:"
cat plain.txt

for i in 1 2 3 4 5 6 8 9
do
  cp generic.txt t$i
  gzip t$i
done

# Create file *without* string error.
cp plain.txt t7
gzip t7

pl " Seeing the list of filenames for comfort:"
find . -name '*.gz' | sort | tail -5

pl " Feeding list of files into gzip, expecting \"4\":"
gzip -cd $( find . -name '*.gz' | sort | tail -5 ) | grep error | wc -l

exit 0

producing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
find (GNU findutils) 4.4.0
gzip 1.3.12

 generic.txt:
The file has an error.

 plain.txt:
This is a plain file.

-----
 Seeing the list of filenames for comfort:
./t5.gz
./t6.gz
./t7.gz
./t8.gz
./t9.gz

-----
 Feeding list of files into gzip, expecting "4":
4

I needed the sort to be sure of the order so that the file without the string error was included in the shortened list, but you may not need that.

Good luck ... cheers, drl
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. Shell Programming and Scripting

Script to read last 30mins logs

Hi All, I want to read the log file for last 30mins logs with time stamps. Am using below command but, it is not working for me awk -F - -vDT="$(date --date="30 minutes ago" "+%b %_d %H:%M:%S")" ' DT < $1' log.file >tmp.txt log file time format is 2016-09-27 14:00:25,192 Use code... (1 Reply)
Discussion started by: Prashanth.K
1 Replies

3. Shell Programming and Scripting

Extracting logs using gunzip awk and gzip

Hi All I am trying to use a hard coded script into shell scripting but I am unable to . Kindly find the Script below along with the command Please help gunzip -c FilePath/FileName_*.gz | awk '$0 > "" && $0 < ""'|\ gzip >> FilePath/Outputfile.log.gz I Am trying to use this... (9 Replies)
Discussion started by: pulkitbahl
9 Replies

4. Shell Programming and Scripting

gzip

how to zip all log file in a folder expect the latest gzip * ---> will zip all log files but I don't want the latest file to be zipped ex: file1, file2, file3, file4, file5 any single command to gzip all files excpet file5 ? (2 Replies)
Discussion started by: rmann
2 Replies

5. Shell Programming and Scripting

Help with GZIP

Hi Gurus, I have a requirement to zip a file using gzip and ftp it to target server. I am using a gzip script as below. gzip.sh #!/bin/ksh /usr/bin/gzip -9 $1 Filename for gzip.sh is passed by an application program. so the output for ./gzip.sh Test_YYYYMMDDHHMMSS.txt (file name is... (1 Reply)
Discussion started by: PRVARMA
1 Replies

6. UNIX for Advanced & Expert Users

gzip vs pipe gzip: produce different file size

Hi All, I have a random test file: test.txt, size: 146 $ ll test.txt $ 146 test.txt Take 1: $ cat test.txt | gzip > test.txt.gz $ ll test.txt.gz $ 124 test.txt.gz Take 2: $ gzip test.txt $ ll test.txt.gz $ 133 test.txt.gz As you can see, gzipping a file and piping into gzip... (1 Reply)
Discussion started by: hanfresco
1 Replies

7. Shell Programming and Scripting

script to constantly read the last 500 new logs in a log file

Hello, I would like to write a bash script that would monitor a log file for a certain number of logs, let's say 500 logs and when it reaches that number to write the last log to another file. For example, I want to watch the /var/adm/messages and everytime, there is 500 new logs that are... (1 Reply)
Discussion started by: Pouchie1
1 Replies

8. Shell Programming and Scripting

help using read in menu script to cat out lines in logs

What is wrong with my menu script? Do I need to continue with the read statements? All I want to do with option 4 is to cat some /var/log/files and awk out a few lines? How do I do that please? $ cat menu.sh ... (11 Replies)
Discussion started by: taekwondo
11 Replies

9. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""... (3 Replies)
Discussion started by: harish.parker
3 Replies

10. UNIX for Dummies Questions & Answers

GZIP help, please!

Gurus, My own stupidity (or ignorance...or both) put me in the situation I am in and I need help please: 1-My shell account (OS: HP UX v11) contains several work directories (/docs, /scripts...) 2-Our sysadmin has implemented aggressive disk quotas so I have to compress the files I put here... (2 Replies)
Discussion started by: alan
2 Replies
Login or Register to Ask a Question