Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Variable as input to awk command

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Variable as input to awk command
# 1  
Old 12-03-2011
Variable as input to awk command
Hi Gurus,

I need a suggestion, please help. I have a input file as below :
abc.txt :
Code:
*
xxxx:              00000
xxxxx:              00000
xxxx:              RANDOM
xxx:              RANDOM
**************************xxxxxxx***
*        abc
******************************
abc:
abc:              6213000
abx:            89234010
abc:             01179
******************************
*        acbxyz
******************************
Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2
0117943621,89234010001179436212,,621300020985821,,2347064000500,,,
0117943622,89234010001179436220,,621300020985822,,2347062347000,,,

I am using a perl script as below :
Code:
     1  #!usr/bin/perl -w
     2
     3  $test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' awktext.txt`;
     4  print $test2;
     5  $test2 = `awk 'BEGIN { count=0;}  { if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/) count++; else print "Unmatching line" ; print } END { print "Number of Lines = ",count;}' `;
     6  print $test2;

The above input file (abc.txt) that I have mentioned is an example file (since the original file will have upto 5 million records after the line : Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2)

Initially we had an perl script where we were validating each line by loading in array and running foreach for every line, after the below line in the input file (abc.txt): Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2
with an appropriate Regex as shown in the above perl script and we really faced performance issues. So, I was advised to use awk (though I see that it actually uses a new shell) for increase in performance, please suggest if you think otherwise. I have to use a perl script for few rules in organization.

Now, please suggest in the above script, how do i use the variable $test as input for the awk command ( i.e. $test2 = `awk 'BEGIN { count=0;} { if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/) count++; else print "Unmatching line" ; print } END { print "Number of Lines = ",count;}' abc.txt`Smilie since I want to process (run regex) the file for the regex only after the line as below (and not before it) : Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2

Running the awk as above will even process the above line and before it.

In short, how can I validate for Regex using a variable ($test1) as input to awk command and store the same in $test2.

If you have any other suggestion apart from the above, kindly let me know.

Thank you
Last edited by jim mcnamara; 12-04-2011 at 09:31 AM.. Reason: code tags please
# 2  
Old 12-04-2011
Bug
I think your perl script should be like this (note the bold line).

Code:
#!usr/bin/perl -w

$test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' awktext.txt`;
print $test1;
$test2 = `awk 'BEGIN { count=0;}  { if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/) count++; else print "Unmatching line" ; print } END { print "Number of Lines = ",count;}' `;
print $test2;

Quote:
The above input file (abc.txt) that I have mentioned is an example file (since the original file will have upto 5 million records after the line : Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2)

Initially we had an perl script where we were validating each line by loading in array and running foreach for every line, after the below line in the input file (abc.txt): Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2
with an appropriate Regex as shown in the above perl script and we really faced performance issues. So, I was advised to use awk (though I see that it actually uses a new shell) for increase in performance, please suggest if you think otherwise. I have to use a perl script for few rules in organization.
Using the above code you've provided, you are using awk twice to parse the same data and that is the reason being slow. Instead, you could just use one awk command and parse the data and count the number of lines starting from a match of the following line
Code:
Kitnb/ICCID1/ICCID2/IMSI1/IMSI2/MSISDN1/MSISDN2/VOUCHER1/VOUCHER2

I think this is something you want to achieve? If that's the case, then you could either use awk or perl to accomplish.

Code:
awk --re-interval 'BEGIN { count=0; found=0 } { if(/Kitnb\/ICCID1\/ICCID2/) { found=1; next } if(/^([0-9]*\,){8}([0-9]*)$/ && found) { count++; } else if(! /^([0-9]*\,){8}([0-9]*)$/ && found) { print "Unmatching line"; print } } END { print "number of lines = " count } ' data.txt

Quote:
In short, how can I validate for Regex using a variable ($test1) as input to awk command and store the same in $test2.
To answer your original question and use the original code given, you could change it this way
Code:
#!usr/bin/perl -w

$test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' awktext.txt`;
print $test1;
$test2 = ` echo "$test1" | awk 'BEGIN { count=0;}  {  if(/^([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,([0-9])*,\$/)  count++; else print "Unmatching line" ; print } END { print "Number of  Lines = ",count;}' `;
print $test2;

Last edited by MR.bean; 12-05-2011 at 12:05 AM..
# 3  
Old 12-05-2011
For the below Suggestion I got the error as below.
Code:
awk --re-interval 'BEGIN { count=0; found=0 } { if(/Kitnb\/ICCID1\/ICCID2/) { found=1; next } if(/^([0-9]*\,){8}([0-9]*)$/ && found) { count++; } else
 if(! /^([0-9]*\,){8}([0-9]*)$/ && found) { print "Unmatching line"; print } } END { print "number of lines = " count } ' data.txt

=> perl Regex.pl
Code:
 Usage: awk [-F fs][-v Assignment][-f Progfile|Program][Assignment|File] ...

So, I modified the script as below (implementing the suggestion of using the variable) and it worked fine:

$test1 = `awk '/Kitnb\\/ICCID1\\/ICCID2/{f=1;next}f' 10k.txt | awk 'BEGIN { count=0;} { if('"$regex"') count++; else print "Correct the output file, The line is :" ; print ;} END {} '`;

Thank you MR.bean Smilie

Thanks,
Arun

---------- Post updated at 04:14 PM ---------- Previous update was at 04:13 PM ----------

BTW, could you please suggest why I got below error :

Code:
awk --re-interval 'BEGIN { count=0; found=0 } { if(/Kitnb\/ICCID1\/ICCID2/) { found=1; next } if(/^([0-9]*\,){8}([0-9]*)$/ && found) { count++; } else
 if(! /^([0-9]*\,){8}([0-9]*)$/ && found) { print "Unmatching line"; print } } END { print "number of lines = " count } ' data.txt

=> perl Regex.pl
Code:
 Usage: awk [-F fs][-v Assignment][-f Progfile|Program][Assignment|File] ...

Thanks,
Arun
Last edited by Scott; 12-05-2011 at 06:48 AM.. Reason: Code tags, please...
# 4  
Old 12-05-2011
It may be because your awk doesnt support --re-interval option.
Can you post the output of
Code:
awk --version

If you are on Solaris, try to run it with 'nawk'.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk command input string too long, limit

cat filename| awk '{ $1=""; print $0}' in my file there are few lines that has more than 3000 characters per line and as soon as I run the above command it cores, strings core reveals that the awk is failing because input string too long, limit. can i get some help from the experts to find... (8 Replies)
Discussion started by: knijjar
8 Replies

2. Shell Programming and Scripting

Variable input to awk script

Hi guys, I wrote the following function to compare two csv files column by column. However, sometimes the input needs to be sorted before parsing it to awk. I can do this by changing the awk arguments, but I would like to make this variable if possible. The below doesn't work since the... (3 Replies)
Discussion started by: Subbeh
3 Replies

3. Shell Programming and Scripting

Call a awk script with variable and input filename

HI, MY question is a very simple one: if i want to call an awk script with the input file name and also pass a variable value , then how to do it. #>awk -f my_script.awk -v variable=value my_inputfile.txt I can't do it like this. throws error: awk: my_script.awk:18:... (0 Replies)
Discussion started by: Onkar Banerjee
0 Replies

4. Shell Programming and Scripting

Passing variable as an input file to AWK comand

Hi, I would like to compare 2 files using awk, which I can do by using: awk 'NR==FNR{a;next} (NR > 32 && $2 in a) {print $0}' File1 and File2. If the name of the File1 is in another file (for example, column 4 in File 3) then how can I pass this column 4 to the awk command. Thanks in... (1 Reply)
Discussion started by: ezhil01
1 Replies

5. Programming

take input from a variable as pattern to awk

Hi everyone, Can anyone tell me how to take contents of a variable as a pattern for awk command. Am doing as below, but doesnt get any output: $c = "Tue Dec"; $log = ` awk '/ \$c /' in.txt`; print $log; (7 Replies)
Discussion started by: anandrec
7 Replies

6. Shell Programming and Scripting

awk built-in variable for input file

Hi guys, Does awk have a built-in variable which I can use to display the input file it's currently reading? I'm currently concatenating multiple files using awk and later on do some parsing. But for now, I want to add an extra column in the main output data file - basically putting in the... (3 Replies)
Discussion started by: Det7
3 Replies

7. Shell Programming and Scripting

Awk command without input file

i have a requirement to compare two time stamps in IF condition and return true whenever the second timestamp is greater than first, i will also be checking, if the timestamp in HHMMSS format( 6 digit time stamp ).Im able to achieve it using awk, however i dont want to give any input file to awk... (3 Replies)
Discussion started by: saikiran_1984
3 Replies

8. Shell Programming and Scripting

Input variable in command line

i have this script that reads a file "listall_101111" where 101111 varies. awk '{print $0,$1}' listall_101111 | sed -e 's/\(.*\).$/\1/g' | awk '{print $6,$2,$3,$4,$5}' | sort -nrk5 | nawk 'NR==1{m=$5;a++;b=(b)?b:$0;next}$5==m{a++;b=(b)?b:$0}END{for (i in a){print b,a}}' | grep -v ^LG | sort... (4 Replies)
Discussion started by: aydj
4 Replies

9. UNIX and Linux Applications

Input a variable and write to a file using awk

Hi I am trying to edit a csv file. Bacically I need to input a search variable and the value that must be changed in one of the fields corresponding to that searched variable. My csv file looks like so: 1,1A,5 1,1B,2 1,1C,3 2,2A,7 2,2B,4 2,2C,0 3,3A,1 3,3B,6 3,3C,4 I want to... (4 Replies)
Discussion started by: ladyAnne
4 Replies

10. UNIX for Dummies Questions & Answers

AWK command giving wrong input

Hi all, I have a problem with qwk command. i have to check process status and for that i am using command prstat -mvL 1 1 and it gives me the entire output but when i use this command with awk like this: prstat -mvL 1 1 | awk -F" " '{print $1,$15}' to get first and 15th arguments. ... (3 Replies)
Discussion started by: usha rao
3 Replies
Login or Register to Ask a Question