Search Results

The comparison is taking a trivial amount of time compared to the time required to:

read your compressed data,
uncompress your data,
write your uncompressed data into a pipe, and
read your...

Expanding on what Corona688 said, your two commands:
zcat file1.gz | head -n 1
zcat file1.gz | tail -n 1
decompress the file twice (maybe not completing the first decompression), and if you find...

Unfortunately you can't seek to the end of a compressed file without decompressing the entire thing, so it really doesn't save you time. Otherwise that'd be a really good idea.

How big is your...

Corona688's reasoning is absolutely correct. I created a large file and timed several awk selection methods:




time awk '{if ($9 == "(200)" && $3 > "[13/Jul/2018:17:00:00" && $3 <...

Assuming your input is sorted by date (which seems likely, given logfiles), processing files individually makes another really big optimization possible - once the date exceeds the cutoff, quit! You...

How big are your files, how long do they take, and how fast is your disk? If you're hitting throughput limits optimizing your programs won't help one iota. If you're not, however, you can process...

i.e.


PROC=0
MAXPROC=4

mkdir /tmp/$$

let N=0

for FILE in nginx*
do
FNAME=$(printf "%04d" "$N") # Filenames like 0001 will sort properly in *
gunzip < "$FILE" | awk...