UNIX for Dummies Questions & Answers

Hi,

I have a huge list of archives (.gz). Each archive is about 40MB. A file is generated every minute so if I want to analyze the data for 1 hour I get already 60 files for example.
These are text files, ';' separated, each line having about 300 fields (columns).

What I need to do is to extract only the lines where a given field (e..g 252) is non-null. Then all this lines I can export in a txt file and analyze the data.

I tried to use cut command, but this one does not allow to put any condition on the field value. So I took the awk command. But this one is limited to 99 fields and I need field 252 for example.
Finally I am using both commands in pipe.
This is what I am using:

Code:

gzip -c -d *.gz | cut -d";" -f2,252 | awk -F';' '$2!=""'

Now this gives me only 2 fields, but not entire line. Even so it is ok as f2 is the ID of the line so I can get later the entire line.

My question is:
1. how to make the above command work faster?
2. how to get directly the entire lines for which a given field is non-null? I might need to do the same for other fields too. And I would like to get all the lines in one shot from several .gz files.

I must say I am pretty new in bash...

Thank you!
Nenad.

What operating system are you using?

What version of awk are you using?

What is the maximum number of bytes in a line (including the terminating <newline> character) in your unzipped files?

What is the output from the command:

Code:

getconf LINE_MAX

on your system?

If you have zcat and Perl, please try the following:

Code:

#!/usr/bin/env perl

use strict;

for (glob '*.gz') {
    open my $fh, sprintf("zcat %s |", $_ ) or die $!;

    {
        local $_;
        while (<$fh>) {
            print if (split ";")[251];
        }
    }
    close $fh or die $!;
}

Save as search.pl
Run as perl search.pl > result.txt

Hi Don,

Thank you for editing and updating my post.
The command you sent gives this:

Code:

bash-3.2$ getconf LINE_MAX
2048

I am not sure what is the max number of bytes of one line.
How can I find this?

If I try to use

Code:

awk

to get field 252 I get this:

Code:

awk: trying to access field 252

If I try to get field number 99:

Code:

bash-3.2$ awk -F';' '{print $99}' myFile
awk: record `9;02bbba84;311:480:f...' has too many fields
 record number 1

.

The OS version is

Code:

SunOS 5.10

.

I tried to get the

Code:

awk

version too but couldn't:

Code:

bash-3.2$ awk -W version
awk: syntax error near line 1
awk: bailing out near line 1
bash-3.2$

Nenad

Hello Nenad,

On a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk , or nawk else you will see errors like as follows.
Thanks,
R. Singh

Aia I tried the perl script but it gives the following error:

Code:

bash-3.2$ perl searchField.pl >resPerl.txt
myFile.gz.Z: No such file or directory
Died at searchField.pl line 14.
bash-3.2$

See what happens if you try the commands:

Code:

gzip -c -d *.gz | /usr/xpg4/bin/awk -F';' '$252 != ""'

and:

Code:

gzip -c -d *.gz | nawk -F';' '$252 != ""'

If neither of them work, check to see if the GNU utilities have been installed on your system. If they have been installed, try gawk (from whatever directory they were installed into) instead of /usr/xpg4/bin/awk and nawk.