Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Getting lines between patterns and perform operations

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Getting lines between patterns and perform operations
# 1  
Old 08-16-2013
Getting lines between patterns and perform operations
Hi, I'm currently dev'ing using awk and I'm currently stuck.

Here's the file, with comments on "<--- ":
Code:
Record            <--- First Pattern
Amount 1          <--- Amount on first transaction
TotalSales 0      <--- Total Sales Prior from previous transactions
Time 1:00:00      <--- Time
Orange
Record            <--- Second Pattern, First Pattern for next Record
Amount 10         <--- Amount on next transaction
TotalSales 1      <--- Total Sales from previous transactions
Time 1:00:00
Orange
Grapes
Record            <--- Second Pattern, First Pattern for next Record
Amount 100        
TotalSales 11   <--- Total Sales from previous transactions
Time 1:00:00
Grapes
Record            <--- Second Pattern, First Pattern for next Record
Amount 10       <--- Missing total sales, ignored set
Time 1:00:00
Grapes
Record            <--- Second Pattern, First Pattern for next Record
TotalSales 111   <--- Missing amount, ignored set
Time 1:00:00
Grapes
Record            <--- Second Pattern, First Pattern for next Record

Basically, I want to extract the lines in between the pattern match, then perform operation such as addition and validation for the lines between the pattern. If the line does not have Amount or TotalSales, it will ignore the pattern and move to the next occurrence.

Output pattern:
Code:
Amount, TotalSales, Time

Here's a sample output:
Code:
1,0,1:00:00 
10,1,1:00:00 
100,11,1:00:00 
2,111,1:00:00

So here's my current code, however it is printing to many duplicate lines:
Code:
awk '/Record/ { start = 1; next }
/Record/ { start = 0 ; exit }
(/Amount/) && start == 1 { amount=$8 };
(/TotalSales/) && start == 1 { totalSales=$8 };
(/Time/) && start == 1 { time=$8 };
{ print amount,totalSales,time}' record.csv

Thanks!
Last edited by Jin_; 08-16-2013 at 02:49 AM..
# 2  
Old 08-16-2013
Code:
awk '{if($1=="Record"){print a;a="Record"} else {a=a" "$0}}' record.csv | awk 'BEGIN{print "Amount, TotalSales, Time";OFS=","} /Amount/&&/TotalSales/&&/Time/ {print $3,$5,$7}'

This User Gave Thanks to krishmaths For This Post:
Jin_
# 3  
Old 08-16-2013
Another awk
Code:
awk '/Record/ {getline;if ($1!~/Amount/) next;a=$2",";getline;if ($1!~/Total/) next;a=a$2",";getline;print a$2}' file
1,0,1:00:00
10,1,1:00:00
100,11,1:00:00

This User Gave Thanks to Jotne For This Post:
Jin_
# 4  
Old 08-16-2013
gawk, mawk:
Code:
awk '/Amount/ && /TotalSales/{print $2,$4,$6}' RS=Record OFS=, file

--
other awks:
Code:
awk '/Amount/{a=$2} /TotalSales/{t=$2} /Time/{if(a!="" && t!="") print a,t,$2; a=t=""}' OFS=, file

Last edited by Scrutinizer; 08-16-2013 at 05:14 AM..
This User Gave Thanks to Scrutinizer For This Post:
Jin_
# 5  
Old 08-16-2013
Holy crap! Good stuff guys.

Thanks!
# 6  
Old 08-16-2013
@Scrutinizer
I do need to remember that changing the Record separator i a very powerful way to do it. Tanks Smilie

You can even save some more:
Code:
awk '/Amount.+TotalSales/{print $2,$4,$6}' RS=Record OFS=, file

Just to show other what is going on:
Code:
awk '{$1=$1;print}' RS=Record OFS=,  file

Amount,1,TotalSales,0,Time,1:00:00,Orange
Amount,10,TotalSales,1,Time,1:00:00,Orange,Grapes
Amount,100,TotalSales,11,Time,1:00:00,Grapes
Amount,10,Time,1:00:00,Grapes
TotalSales,111,Time,1:00:00,Grapes

This User Gave Thanks to Jotne For This Post:
Jin_
# 7  
Old 08-16-2013
@Jotne Smilie Yes, but only with gawk and mawk. Regular awk can only use a single character as a record separator (or two consecutive new lines if left blank)...
This User Gave Thanks to Scrutinizer For This Post:
Jin_
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to print lines from a files with specific start and end patterns and pick only the last lines?

Hi, I need to print lines which are matching with start pattern "SELECT" and END PATTERN ";" and only select the last "select" statement including the ";" . I have attached sample input file and the desired input should be as: INPUT FORMAT: SELECT ABCD, DEFGH, DFGHJ, JKLMN, AXCVB,... (5 Replies)
Discussion started by: nani2019
5 Replies

2. UNIX for Beginners Questions & Answers

Copy last few lines of a file, perform math operation and iterate further

Hi, I am trying to generate a data of following order: 4 0 1 642 643 4 642 643 1283 1284 4 1283 1284 1924 1925 4 1924 1925 2565 2566 4 2565 2566 3206 3207 4 3206 3207 3847 3848 4 3847 3848 4488 4489 4 4488 4489 5129 5130 ---------------------- 4 1 2 643 644 4 643 644 1284... (6 Replies)
Discussion started by: SaPa
6 Replies

3. UNIX for Beginners Questions & Answers

Delete multiple lines between blank lines containing two patterns

Hi all, I'm looking for a way (sed or awk) to delete multiple lines between blank lines containing two patterns ex: user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 16... (3 Replies)
Discussion started by: ce9888
3 Replies

4. Shell Programming and Scripting

How to perform multiple operations on a number before storing to a variable?

(I am using bash) I have a command that will find the line number in a file by searching for a string where it exists. linenumber=$(grep -n "string" $FILENAME | cut -d : -fi) This returns the line number and removes the string. Now that I have the line number I want to subtract 4 from it and... (5 Replies)
Discussion started by: prodigious8
5 Replies

5. Shell Programming and Scripting

Perform operations as a non root user

Hi All, I need a help in the below scenario. I want to perform few operations as a non root user but those operations can be performed only by the root user. For example I need to modify /etc/hosts file as a non root user. This is just one scenario. Could you please provide... (3 Replies)
Discussion started by: kalpeer
3 Replies

6. Shell Programming and Scripting

Perform Operations on One File Conditional on Data in Another File

Hello all, I am looking for a solution to the following problem. Perl or python solutions also welcome. Given this input: And this input: I want to get this output. The rule being that if the number in the first file is < 0.9, then the corresponding two columns on... (2 Replies)
Discussion started by: hydrabane
2 Replies

7. Shell Programming and Scripting

How can I get the lines between two patterns?

hi, I have the following file hello world this is to say bye to everyone so bye I want to get the lines from hello to the first bye inclusive into another file? how can I do this (11 Replies)
Discussion started by: JamesByars
11 Replies

8. Shell Programming and Scripting

Operations on a file with Deletion , Modification and Insertion of lines

Hi All , Given a file , I need to delete , modify and insert lines matching certain patterns in that file using shell scripting. e.g. If a file FILE1 has following content : CUST.ABC.DEF = CUST.ABC.DEF * CUST.ABC.DEF PRINTF(CUST.ABC.DEF) CUST.ABC.DEF = mid(CUST.ABC.DEF,10.34)... (5 Replies)
Discussion started by: swapnil.nawale
5 Replies

9. Shell Programming and Scripting

Searching patterns in 1 file and deleting all lines with those patterns in 2nd file

Hi Gurus, I have a file say for ex. file1 which has 3500 lines in it which are different account numbers and another file (file2) which has 230000 lines in it. I want to read all the lines in file1 and delete all those lines from file2 which has that same pattern as in file1. I am not quite... (4 Replies)
Discussion started by: toms
4 Replies

10. Shell Programming and Scripting

Can sed perform editing operations ONLY in the matched region?

Hi: Let's suppose I want to replace all the | by > ONLY when | is between . Usually (and it works) I would do something like sed -e 's/\(\*\)|\(*\]\)/\1>\2/g' where I have to "save" some portions of the matched region and use them with the \n metacharacter. I was wondering if I could... (2 Replies)
Discussion started by: islegmar
2 Replies
Login or Register to Ask a Question