awk & zcat not working together


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk & zcat not working together
# 1  
Old 04-13-2013
awk & zcat not working together

I have a tar file which i want to read and check some specific fields basis on which i want to get output.
Code

Code:
 
zcat samplefile.tar.gz | awk 'FNR==1 {++counter}
counter ==2 {BB[$1]=1;next}
substr($0,26,2) =="01") {next}
(substr($0,28,12) ~ "^[A-Z,a-z]") {next}
(substr($0,184,3) in BB) {next}
1
' bb.txt

bb.txt
Code:
1234
5678

samplefile.tar.gz:
Code:
000000000000003255816850602YD-MTSBAL 519131689141165735 1120120901213655201209032136550000000001132005060000000001 405891364839056 2377 00000000000004294967040405899136999995
 
000000000000005255817273002919875129904 119132018104840216 1120120901213658201209032136580000000000033000160000000000405891742168097 161A 12283000032014294967040405899136999995

Need output as
Code:
000000000000005255817273002919875129904 119132018104840216 1120120901213658201209032136580000000000033000160000000000405891742168097 161A 12283000032014294967040405899136999995


Last edited by Scrutinizer; 04-13-2013 at 05:18 AM.. Reason: Additional code tags for data files
# 2  
Old 04-13-2013
You cannot use both stdin and file input like that, you would need the "-" file operand..
Try:
Code:
zcat samplefile.tar.gz | awk '...' bb.txt -

This User Gave Thanks to Scrutinizer For This Post:
# 3  
Old 04-13-2013
Your code can't work for several reasons.
Code:
awk 'FNR==1 {++counter}                      # fine, you count no. of files
     counter ==2 {BB[$1]=1; next}            # from the second file, keep $1. But, there's no second file, as scrutinizer pointed out.
     substr($0,26,2) =="01") {next}          # is never met in your sample input, which is fine if you want to keep all lines. One closing paranthesis too many!
     (substr($0,28,12) ~ "^[A-Z,a-z]") {next}   # would succeed on your tar's second input line if you came that far
     (substr($0,184,3) in BB) {next}         # your longest line has 177 chars
     1
    ' bb.txt                                 # just one file to work upon; see scrutinizer's post

All your code does is print your bb.txt lines.

If you need further help, specify what output you want from which input based on which logic, in plain English.
# 4  
Old 04-13-2013
Thanks Rudi & Scrutinzer for the inputs. Reiterating my requirement once again as follows:

1. Input is from tar file named samplefile.tar.gz, so used following code
Code:
zcat sampleinputfile.tar.gz

2. Dont want to untar the above file so used zcat.
3. Check the fields at 26th location for "01" from the tar file , so used following code
Code:
(substr($0,26,2) =="01") {next}

4. Also check the first character of 28th field, so used following code
Code:
(substr($0,28,12) ~ "^[A-Z,a-z]") {next}

5. Also check for 184th location for 3 bytes in BB array from bb.txt file, so used following code
Code:
(substr($0,184,3) in BB) {next}


Output required is to print all the lines from input (tar file) which is not matching the above conditions
# 5  
Old 04-13-2013
How about applying our advices and creating / testing a revised code. If that does not work, come back for help. One more comment: As the bb.txt file entries and thus BB elements are having 4 chars, they will never match substr ($0, 184,3)...
This User Gave Thanks to RudiC For This Post:
# 6  
Old 04-13-2013
tried changing code to as follows(used if condition) in case required i can pull other data using else if condition(for future requirement if any), but this prints the complete samplefile data (matching the condition)and not the substrs as mentioned in code below(see the printf). Please look into this & suggest

Code:
zcat samplefile.tar.gz | awk 'FNR==1 {++counter}
counter==1 {BB[$1]=1;next}
{
if (( (substr($0,26,2) =="02") && (substr($0,28,12)  ~ "^[A-Z,a-z]") && (substr($0,184,24) in BB)  ))
 print substr($0,28,12)","substr($0,54,16) 
}
1
' bb.txt -

# 7  
Old 04-14-2013
You weren't that far off with your first idea. After some polishing your code could look like
Code:
awk     'FNR==NR        {BB[$1] = 1; next}
          substr ($0,  26, 2) == "01"           {next}
          substr ($0,  28, 1) ~ /[A-Z,a-z]/    {next}
         (substr ($0, 184, 4) in BB)            {next}
         1
        ' file2 file

and should be working (replace my file names).
If testing long unstructured lines, sometimes single character columns come in handy (use FS=""):
Code:
awk     'FNR==NR                {BB[$1] = 1; next}
         $26$27 == "01"         {next}
         $28    ~  /[A-Z,a-z]/ {next}
         ($184$185$186$187 in BB)               {next}
         1
        ' file2 FS="" file

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

ZCAT xxx.Z|tar -xvf - decompression slow ?

I have recently built a new server and patched Soalris 10 up tp latest bundle etc... When I run a decompress using the format zcat fred.Z |tar -xvf - it runs at a very slow rate. A similiar server which is less powerful runs over twice as quick. Is there any work arounds to configure decompress... (4 Replies)
Discussion started by: smcart
4 Replies

2. Shell Programming and Scripting

[zcat] [gunzip -c] header

Hello, When i lauches this command ssh -n server_name gunzip -c "/REP/xxxx.gz" > server.logThere are a few stray characters like NULNULNULNUL100644 NUL000000NUL000000NUL00024002704 12135060747 012607NUL at the beginning of server.log. In the xxxx.gz there is a tar file I'm not sure but it... (5 Replies)
Discussion started by: amazigh42
5 Replies

3. Shell Programming and Scripting

awk Help: quick and easy question may be: How to use &&

Hi Guru's. I am trying to use to check if $5 is greater than 80 & if not 100, then to print $0 : awk '{ if ($5>80) && if ($5 != 100) print $0} But getting error: >bdf1|sed 's/%//g'|awk '{ if ($5>80) && if ($5 != 100) print $0}' syntax error The source line is 1. The error... (6 Replies)
Discussion started by: rveri
6 Replies

4. UNIX for Dummies Questions & Answers

How zcat works?

Hi, just i would like to know, how will be the response if you try to read a 40GB compressed file using zcat. a)Uncompress the 40GB file and have it in the disk. use cat to view the steps. b)Use zcat directly to view the compressed file? What are the steps being occurred in step (b)? Where... (3 Replies)
Discussion started by: pandeesh
3 Replies

5. Shell Programming and Scripting

BASH ZCAT EGREP Shell Script

I created a backup script that emails all the admins when the backup is complete and attaches a log file of what what backed up. On occasion, something happens in which the backups stop working, I started "grep"ing around /var/log/syslog and I usually find the smoking gun. My goal is to zcat... (8 Replies)
Discussion started by: metallica1973
8 Replies

6. Shell Programming and Scripting

zcat two files

Hi, Like cat file1 file2 > file Can I do zcat file1.gz file2.gz > file.gz (11 Replies)
Discussion started by: jacobs.smith
11 Replies

7. UNIX for Advanced & Expert Users

Awk expressions working & not working

Hi, Putting across a few awk expressions. Apart from the last, all of them are working. echo a/b/c | awk -F'/b/c$' '{print $1}' a echo a/b/c++ | awk -F'/b/c++' '{print $1}' a echo a/b/c++ | awk -F'/b/c++$' '{print $1}' a/b/c++ Request thoughts on why putting a '$' post double ++... (12 Replies)
Discussion started by: vibhor_agarwali
12 Replies

8. Linux

zcat on Linux

Hi I am trying to extract a <filename>.tar.Z on a SLES machine using zcat. The command I am using is zcat <filename>.tar.Z | tar xf - When I use the above I get the following message tar: Read 7168 bytes from - I think the message is benign because I see that my files where... (2 Replies)
Discussion started by: felixmat1
2 Replies

9. UNIX for Dummies Questions & Answers

zcat --> Arg list too long

Hi all I have more than 1000 files in a folder and when ever i use a "compress" or "zcat" command it give error /bin/zcat: Arg list too long. . any solution for this :o (3 Replies)
Discussion started by: muneebr
3 Replies

10. Shell Programming and Scripting

Awk: Version && nextfile

How can I find which version of Awk is installed? OpSystem is HPUX 11.x I am getting an error when trying to use the keyword nextfile and I dont know why! (Well, I can only assume that I have am using a version of Awk that does not support nextfile. However, according to O'Reilly, nextfile is... (3 Replies)
Discussion started by: google
3 Replies
Login or Register to Ask a Question