Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Efficient awk way to add numbers in line fields

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Efficient awk way to add numbers in line fields
# 8  
Old 10-16-2017
Without knowing the pattern nor the input file, it's difficult to explain that behaviour, esp. if you say it "worked beautifully" before. Was that with the same data?
Last edited by RudiC; 10-16-2017 at 04:55 PM..
# 9  
Old 10-16-2017
Quote:
Originally Posted by Don Cragun
Your code prints the lines for 10pm on Sunday, October 15, 2017 (from the 2nd statement in the first line of your awk code. I assume that the value you seem to want from those lines total up to zero. But the calculations in your script (from the statement on lines 2 through 10 in your awk script) are performed on every line in the file; not just those that you print (since the condition on that statement is the empty expression).

Again, you say the output is not what you want, but do not clearly show us what output you are trying to produce.

And, you have not shown us the input that you are feeding this script. So, we have no idea where the 5 is coming from either. If your input file contained the line:
Code:
,,,,,,=5

plus the first three lines shown in your output and $pattern expands to an empty string, you would likely get the output you showed us.
maybe if i word this differently, it'll be clearer.

the data file that the awk code is scanning as many lines similar to this (and they all have different time stamps on them):

Code:
0,data,Sun Oct 15 22:01:23 2017,6906,/var/log/catalina.out,524K,data,529086,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6894 6906,148       
0,data,Sun Oct 15 22:06:23 2017,6906,/var/log/catalina.out,524K,data,529086,emxCommonAppInitialization__Error_while_initializing=0  INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0  java_lang_NoClassDefFoundError=0  java_lang_OutOfMemoryError=0  Master_Item_Service_is_down=0  The_file_or_directory_is_corrupted_and_unreadable=0,6906 6906,448  
0,data,Sun Oct 15 22:11:23 2017,6914,/var/log/catalina.out,524K,data,529580,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6906 6914,100

the lines i care about, in this particular scenario are all lines that have the 10PM timestamp on them.

ok, so when the awk code finds those lines that contain 10PM on them, it should do what my original request in the original post of this thread was about....which is to add up all the numbers in the 9th field, provided a specific pattern was not specified.

if a specific pattern to search for in field 9 was not provided, then the "$pattern" variable would be empty. And yes, in this case, it is empty. i did not specify a pattern.

so
Code:
 pattern=""

i hope its clear now.
# 10  
Old 10-16-2017
OK.

So now you have shown us three out of an unknown number of lines in your input file and you have told us how pattern is set.

Is it really that hard for you to actually show us a few more lines of input that would allow us to test code that might meet your needs so we know that we are correctly extracting only the desired lines from input files AND show us the output you want your script to produce?
# 11  
Old 10-16-2017
Quote:
Originally Posted by Don Cragun
OK.

So now you have shown us three out of an unknown number of lines in your input file and you have told us how pattern is set.

Is it really that hard for you to actually show us a few more lines of input that would allow us to test code that might meet your needs so we know that we are correctly extracting only the desired lines from input files AND show us the output you want your script to produce?
all the lines will look similar to this:
Code:
0,data,Sun Aug 15 22:01:23 2017,6906,/var/log/catalina.out,524K,data,529086,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6894 6906,148
0,data,Sun Aug 15 22:06:23 2017,6906,/var/log/catalina.out,524K,data,529086,emxCommonAppInitialization__Error_while_initializing=0  INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0  java_lang_NoClassDefFoundError=0  java_lang_OutOfMemoryError=0  Master_Item_Service_is_down=0  The_file_or_directory_is_corrupted_and_unreadable=0,6906 6906,448
0,data,Sun Aug 15 22:11:23 2017,6914,/var/log/catalina.out,524K,data,529580,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6906 6914,100
0,data,Sun Sep 15 22:01:23 2017,6906,/var/log/catalina.out,524K,data,529086,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6894 6906,148
0,data,Sun Sep 15 22:06:23 2017,6906,/var/log/catalina.out,524K,data,529086,emxCommonAppInitialization__Error_while_initializing=0  INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0  java_lang_NoClassDefFoundError=0  java_lang_OutOfMemoryError=0  Master_Item_Service_is_down=0  The_file_or_directory_is_corrupted_and_unreadable=0,6906 6906,448
0,data,Sun Sep 15 22:11:23 2017,6914,/var/log/catalina.out,524K,data,529580,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6906 6914,100
0,data,Sun Oct 15 22:01:23 2017,6906,/var/log/catalina.out,524K,data,529086,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6894 6906,148
0,data,Sun Oct 15 22:06:23 2017,6906,/var/log/catalina.out,524K,data,529086,emxCommonAppInitialization__Error_while_initializing=0  INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0  java_lang_NoClassDefFoundError=0  java_lang_OutOfMemoryError=0  Master_Item_Service_is_down=0  The_file_or_directory_is_corrupted_and_unreadable=0,6906 6906,448
0,data,Sun Oct 15 22:11:23 2017,6914,/var/log/catalina.out,524K,data,529580,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6906 6914,100
0,data,Sun Nov 15 22:01:23 2017,6906,/var/log/catalina.out,524K,data,529086,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6894 6906,148       
0,data,Sun Nov 15 22:06:23 2017,6906,/var/log/catalina.out,524K,data,529086,emxCommonAppInitialization__Error_while_initializing=0  INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0  java_lang_NoClassDefFoundError=0  java_lang_OutOfMemoryError=0  Master_Item_Service_is_down=0  The_file_or_directory_is_corrupted_and_unreadable=0,6906 6906,448  
0,data,Sun Nov 15 22:11:23 2017,6914,/var/log/catalina.out,524K,data,529580,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6906 6914,100

# 12  
Old 10-16-2017
Quote:
Originally Posted by SkySmart
[..]
What is wrong with the modified code below:

Code:
awk -F, -v pat="$Pattern" 'BEGIN{t=0} /'Sun' 'Oct' '15' '22':[0-9][0-9]:[0-9][0-9] '2017',/
  {
[..]

The search pattern must be before the opening brace, not on the line above it (otherwise it is seen by awk as two separate condition-action sequences):

Code:
awk -F, -v pat="$Pattern" '
  BEGIN {
    t=0
  }

  /Sun Oct 15 22:[0-9][0-9]:[0-9][0-9] 2017,/ {
[..]

And you should leave out al those single quotes..



--
Note: this is also what Don Cragun mentioned in post #7
Last edited by Scrutinizer; 10-16-2017 at 11:06 PM..
This User Gave Thanks to Scrutinizer For This Post:
SkySmart
# 13  
Old 10-17-2017
seems to be working now.

thanks all!!!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to identify empty fields in line

I am trying to use awk to identify and print out records in fields that are empty along with which line they are in. I hope the awk below is close, it runs but nothing results. Thank you :). awk awk -F'\t' 'FNR==NR ~ /^*$/ { print "NR is empty" }' file file 123 GOOD ID 45... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

awk - set numbers [ 1 ... n] from the 6 line

Hi, i have a file, where measurement-data is stored in the first column. The file has also a header of 5 lines. I want to set counting up numbers in front of any particular measurement-value; should start at the 6. line with starting number 1. i try to solve it with ... awk 'NR > 6 { print... (6 Replies)
Discussion started by: IMPe
6 Replies

3. Shell Programming and Scripting

How to add line numbers (multiples of 5: 0,5,10,15,20) to a text file?

Hi, I need to number the lines in my text file. I know how to do this with standard numbering (1,2,3,4, etc) but I need to count in multiples of 5, beginning 0,5,10,15... example existing file: abcd efg hijklm nopqrs desired output 0 abcd 5 efg 10 hijklm 15 ... (11 Replies)
Discussion started by: livbaddeley
11 Replies

4. Shell Programming and Scripting

awk (or other) script that assigns fields from a line to multiple variables

Hey all, Unfortunately I have only basic knowledge of awk and/or scripting. If I have a file with lines that can look similar to this: Name=line1 Arg1=valueA Arg2=valueB Arg3=valueC Name=line2 Arg1=valueD Name=line3 Arg1=valueE Arg3=valueF Name=line4 Arg2=valueG ... (4 Replies)
Discussion started by: Rike255
4 Replies

5. Shell Programming and Scripting

Using Awk to Add Numbers

echo "0.1 2.0 0.4 2.0 4.3 1.0 6.0 9.0" | awk 'BEGIN {total=0} {total += $1} END {print total}' I want to add the above output from the echo command, but i can't figure this out. The output above always spits out inaccurate numbers. can someone please provide me with a one liner similar to... (4 Replies)
Discussion started by: SkySmart
4 Replies

6. Shell Programming and Scripting

AWK multiple line fields sorting

I have a bash script which takes a log file with each record separated by a #. The records have multiple fields but field $1 is always the date and time. When the script is run it prints the record just fine from oldest to newest. I need to have records print out from newest first. Here is the... (7 Replies)
Discussion started by: numele
7 Replies

7. Shell Programming and Scripting

Add to constant fields at the end of every line

Hi, I want to add two fields with values '1000' and 'XYZ-1234' at the end of every line in a comma delimited file. Should I use any command in a loop to add the fields or using any single command Shall I acheive it? Kindly help me in code. Thanks, Poova. (6 Replies)
Discussion started by: poova
6 Replies

8. Shell Programming and Scripting

how to add line numbers in text file

Hi all How to add line numbers in text file.. ex abcd cdef result 1. abcd 2. cdef thx in advance (4 Replies)
Discussion started by: suryanarayana
4 Replies

9. UNIX for Advanced & Expert Users

Add line numbers to end of each line

Hi i would like to add line numbers to end of each line in a file. I am able to do it in the front of each line using sed, but not able to add at the end of the file. Can anyone suggest The following code adds line number to start of each line sed = filename | sed 'N;s/\n/\t/' how can i... (5 Replies)
Discussion started by: rudoraj
5 Replies

10. Shell Programming and Scripting

add line numbers

Hello.. I have got one file ... I want to add line numbers with space form starting to ending.. for example...if the file is -------------------------- sand sorcd 2345 345 recds 234 234 5687 yeres 568 988 erfg4 67 -------------------------- I need the output ... (4 Replies)
Discussion started by: esham
4 Replies
Login or Register to Ask a Question