Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk out an unknown column ?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk out an unknown column ?
# 1  
Old 08-02-2008
awk out an unknown column ?
I need a solution to awk out an unknown column. I am unable to say '{print $x}' because the location changes. I would like to find a perl or awk solution to this. I do not know either very well but am trying to delve deeper into both.
I am looking for the version of pkg8 in this example. Please keep in mind the location of pkg8 will change depending...

Here is the line I am looking at :

Code:
# grep pkg8 /tmp/trash
XXX Packages             : XXXpkg1(6.2.3.1) XXXpkg2(1.0.0.3) XXXpkg3(4.0.0.5) XXXpkg4(2.0.0.1) XXXpkg5(A0)  XXXpkg6(2.0.1.1)  XXXpkg8(5.7.0.10) XXXpkg9(1.5) XXXpkg10(2.1.2.3)

I was able to find a UGLY solution which includes a for loop and if statement. But this is very slow. I am looking at this file on 200 + systems, so the execution does matter.

Here is the entire script for this :

Code:
#!/bin/ksh
HOSTNAME=`uname -n`
OS_VER=`uname -r`

PKG=XXXpkg8
CORRECT_VER=5.7.0.10
COLLECT=/tmp/trash
for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
    if [ `echo $x|grep "XXXpkg8("` ];then
        PKG_VER=`echo $x|grep "XXXpkg8("`
    else
        continue
    fi
done
VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`



if [[ $VER = $CORRECT_VER ]];then
        if [[ $STATUS = NO ]];then
            continue
        else
            STATUS=YES
        fi
else
    STATUS=NO
    MESSAGE="$MESSAGE XXXpkg8:$VER"
fi

echo "$STATUS $MESSAGE"

I would like to replace the entire for loop with a perl/awk one liner. I have searched for days and frankly my head hurts already. I have seen grep and for loop used inside of awk but I really don't understand what it is doing.
problems in grep inside awk - Computing.Net
[url=http://www.linuxquestions.org/questions/programming-9/can-i-use-grep-inside-awk-174287[/url]


If anyone can offer some help I would appreciate it.
# 2  
Old 08-02-2008
Code:
awk '{
  for( ......) #go through all fields
  {
   if( field contains what i want ){
     print field
   }
  }

}' file

# 3  
Old 08-03-2008
Replace the following lines :
Code:
for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
    if [ `echo $x|grep "XXXpkg8("` ];then
        PKG_VER=`echo $x|grep "XXXpkg8("`
    else
        continue
    fi
done
VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`

by :
Code:
VER=$( sed -n 's/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )

Jean-Pierre.
# 4  
Old 08-04-2008
ghostdog74,

While I appreciate your post, I am not an expert with awk. I was unable to get your example to work. This is what I tried.
Code:
awk '{
  for( `grep $PKG $COLLECT|awk -F: '{print $2}'` ) #go through all fields
  {
   if( XXXpkg8 ){
     print field
   }
  }

}' $COLLECT

I am not sure how this will populate the $VER which is what I am after.

Aigles,

Thanks for the example. I was able to drop the one line in and it worked perfectly. Since I am no expert with sed/awk/perl, I really appreciate you providing an example that I could just drop in. However it turned out to be very slow. Since I will be running this within another for loop (about 200 times), I need it to run faster.

I ran your sed in ./check_pkg and my code was in check_pkg2.
Code:
root@smsdr2# time ./check_pkg
YES

real    0m1.737s
user    0m1.500s
sys     0m0.000s
root@smsdr2# time ./check_pkg2
YES

real    0m0.365s
user    0m0.040s
sys     0m0.060s

Code:
root@smsdr2# diff check_pkg check_pkg2
8,16c8,15
< #for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
< #     if [ `echo $x|grep "XXXpkg8("` ];then
< #             PKG_VER=`echo $x|grep "XXXpkg8("`
< #     else
< #             continue
< #     fi
< #done
< #VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
< VER=$( sed -n 's/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )
---
> for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
>       if [ `echo $x|grep "XXXpkg8("` ];then
>               PKG_VER=`echo $x|grep "XXXpkg8("`
>       else
>               continue
>       fi
> done
> VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
17a17,18
>
>

Any other ideas?
# 5  
Old 08-04-2008
Why do you need to run it in another loop? Perhaps you could show us the whole code so we can see what to optimize. (I'm thinking maybe you could run the whole loop in sed.)
# 6  
Old 08-04-2008
Quote:
Originally Posted by i9300
ghostdog74,

While I appreciate your post, I am not an expert with awk. I was unable to get your example to work. This is what I tried.
Code:
awk '{
  for( `grep $PKG $COLLECT|awk -F: '{print $2}'` ) #go through all fields
  {
   if( XXXpkg8 ){
     print field
   }
  }

}' $COLLECT

I am not sure how this will populate the $VER which is what I am after.

Aigles,

Thanks for the example. I was able to drop the one line in and it worked perfectly. Since I am no expert with sed/awk/perl, I really appreciate you providing an example that I could just drop in. However it turned out to be very slow. Since I will be running this within another for loop (about 200 times), I need it to run faster.

I ran your sed in ./check_pkg and my code was in check_pkg2.
Code:
root@smsdr2# time ./check_pkg
YES

real    0m1.737s
user    0m1.500s
sys     0m0.000s
root@smsdr2# time ./check_pkg2
YES

real    0m0.365s
user    0m0.040s
sys     0m0.060s

Code:
root@smsdr2# diff check_pkg check_pkg2
8,16c8,15
< #for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
< #     if [ `echo $x|grep "XXXpkg8("` ];then
< #             PKG_VER=`echo $x|grep "XXXpkg8("`
< #     else
< #             continue
< #     fi
< #done
< #VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
< VER=$( sed -n 's/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )
---
> for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
>       if [ `echo $x|grep "XXXpkg8("` ];then
>               PKG_VER=`echo $x|grep "XXXpkg8("`
>       else
>               continue
>       fi
> done
> VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
17a17,18
>
>

Any other ideas?
A little modification to the sed command will increase performances :
Code:
VER=$( sed -n "/$PKG/"'s/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )

I have tested the three solutions with a file containing 6900 lines (package versions on last line) on my PC (cygwin) and on my AIX box.

The initial version run faster under AIX only (i am very surprise by the difference between Cygwin and AIX results).
The second sed version is the fastest in the two case.

Cygwin times :
Code:
---- Initial version ---
YES  
real    0m3.140s
user    0m1.047s
sys     0m0.719s
---- sed version -------
YES  
real    0m0.406s
user    0m0.215s
sys     0m0.137s
---- sed version 2 -----
YES  

real    0m0.360s
user    0m0.231s
sys     0m0.090s

AIX times :
Code:
---- Initial version ---
YES  
réel    0m0,14s
util    0m0,00s
sys     0m0,09s
---- sed version -------
YES  
réel    0m1,55s
util    0m1,52s
sys     0m0,03s
---- sed version 2 -----
YES  

réel    0m0,03s
util    0m0,02s
sys     0m0,01s

Jean-Pierre.
# 7  
Old 08-04-2008
@OP
to go through the fields in awk
Code:
awk '
{
 for(i=1;i<=NF;i++ ){
   if ( $i == "what i want" ) {
      # do something.
   } 
 } 

}

' file
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk script to append suffix to column when column has duplicated values

Please help me to get required output for both scenario 1 and scenario 2 and need separate code for both scenario 1 and scenario 2 Scenario 1 i need to do below changes only when column1 is CR and column3 has duplicates rows/values. This inputfile can contain 100 of this duplicated rows of... (1 Reply)
Discussion started by: as7951
1 Replies

2. Shell Programming and Scripting

awk to Sum columns when other column has duplicates and append one column value to another with Care

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (1 Reply)
Discussion started by: as7951
1 Replies

3. Shell Programming and Scripting

awk to update unknown value in file using range of another

I am trying to use awk to update all the unknown values in $6 of file2, if the $4 value in file 2 is within the range of $1 of file1. If there is already a value in $6 other then unknown, it is skipped and the next line is processed. In my awk attempt below the final output is 6 tab-delimited... (6 Replies)
Discussion started by: cmccabe
6 Replies

4. Shell Programming and Scripting

Print Unknown Number of User Inputs in awk

Hello, I am new to awk and I am trying to figure out how to print an output based on user input. For example: ubuntu:~/scripts$ steps="step1, step2, step3" ubuntu:~/scripts$ echo $steps step1, step2, step3 I am playing around and I got this pattern that I want: ... (3 Replies)
Discussion started by: tattoostreet
3 Replies

5. Shell Programming and Scripting

Problems with awk (fatal error) and paste (two variables into one column-by-column)

Hello, I have a script extracting columns of useful numbers from a data file, and manipulating the numbers with awk commands. I have problems with my script... 1. There are two lines assigning numbers to $BaseForAveraging. If I use the commented line (the first one) and let the second one... (9 Replies)
Discussion started by: vgbraymond
9 Replies

6. Shell Programming and Scripting

awk Print New Column For Every Two Lines and Match On Multiple Column Values to print another column

Hi, My input files is like this axis1 0 1 10 axis2 0 1 5 axis1 1 2 -4 axis2 2 3 -3 axis1 3 4 5 axis2 3 4 -1 axis1 4 5 -6 axis2 4 5 1 Now, these are my following tasks 1. Print a first column for every two rows that has the same value followed by a string. 2. Match on the... (3 Replies)
Discussion started by: jacobs.smith
3 Replies

7. Shell Programming and Scripting

awk code to ignore the first occurence unknown number of rows in a data column

Hello experts, Shown below is the 2 column sample data(there are many data columns in actual input file), Key, Data A, 1 A, 2 A, 2 A, 3 A, 1 A, 1 A, 1 I need the below output. Key, Data A, 2 A, 2 A, 3 A, 1 A, 1 A, 1 (2 Replies)
Discussion started by: ks_reddy
2 Replies

8. Shell Programming and Scripting

for each different entry in column 1 extract maximum values from column 2 in unix/awk

Hello, I have 2 columns (1st column has multiple entries but the corresponding values in the column 2 may be the same or different.) however I want to extract unique values for each entry in column 1 by assigning the max value from column 2 SDF4 -0.211654 SDF4 0.978068 ... (1 Reply)
Discussion started by: Diya123
1 Replies

9. Shell Programming and Scripting

Dropping Records for unknown reason in awk script

Hi, I have written the following it is pretty sloppy but I don't see any reason why I should be losing 54 records from a 3.5 million line file after using it. What I am doing: I have a 3.5 million record file with about 80,000 records need a correction. They are missing the last data from... (8 Replies)
Discussion started by: mkastin
8 Replies

10. Solaris

PING - Unknown host 127.0.0.1, Unknown host localhost - Solaris 10

Hello, I have a problem - I created a chrooted jail for one user. When I'm logged in as root, everything work fine, but when I'm logged in as a chrooted user - I have many problems: 1. When I execute the command ping, I get weird results: bash-3.00$ usr/sbin/ping localhost ... (4 Replies)
Discussion started by: Przemek
4 Replies
Login or Register to Ask a Question