Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Copy data to new file based on input pattern

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Copy data to new file based on input pattern
# 1  
Old 10-20-2017
Copy data to new file based on input pattern
Hi All,

I want to create a new file based on certain conditions and copy only those conditioned data to new file.

Input Data is as it looks below.

Code:
ORDER|Header|Add|32|32|1616
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1616
ORDER|Header|Modify|32|32|1617
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1617
ORDER|Header|Add|32|32|1618
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1618
ORDER|Header|Add|32|32|1619
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1619

I want to copy only those data to a new file which has the Header starting with #ORDER|Header|Add|32|32|1618 and followed by details lines until I reach Termination line which looks as #ORDER|T|32|32|1618

1618,1619 all these keeps getting incremented by 1 and is considered as 1 chunk of data. So in the data mentioned above I need only the below data.

Sample output:

Code:
ORDER|Header|Add|32|32|1616
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1616
ORDER|Header|Add|32|32|1618
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1618
ORDER|Header|Add|32|32|1619
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1619

Thanks & Regards
Gaurav
Last edited by rbatte1; 10-20-2017 at 08:51 AM.. Reason: Added CODE tags
# 2  
Old 10-20-2017
Hello grvk101,

Welcome to forums, hope you will enjoy learning and sharing knowledge here. Please use code tags for sample Input_file and expected output too.
Please try following and let me know if this helps you.
Code:
awk '!/ORDER\|Header\|Add\|32\|32\|161[68]/ && !/ORDER\|Details/ && !/ORDER\|T/{flag=""} /ORDER\|Header\|Add\|32|\32\|161[68]/{flag=1} flag'   Input_file

Output will be as follows.
Code:
ORDER|Header|Add|32|32|1616
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1616
ORDER|Header|Add|32|32|1618
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1618
ORDER|Header|Add|32|32|1619
ORDER|Details1.........
ORDER|Details2.........
ORDER|Details3.........
ORDER|Details4.........
ORDER|T|32|32|1619

EDIT: Adding a non-one liner form of solution too here.
Code:
awk '
!/ORDER\|Header\|Add\|32\|32\|161[68]/ && !/ORDER\|Details/ && !/ORDER\|T/{
  flag=""
}
/ORDER\|Header\|Add\|32|\32\|161[68]/{
  flag=1
}
flag
'   Input_file

Thanks,
R. Singh
# 3  
Old 10-20-2017
Hello Ravinder,

Getting error as below.

Code:
$ awk !/ORDER\|Header\|Add\|32\|32\|161[68]/ && !/ORDER\|Details/ && !/ORDER\|T/{flag=""} /ORDER\|Header\|Add\|32|\32\|161[68]/{flag=1} flag samplefile.dat
./test.sh[6]: $:  not found


Added to it I need to search data based only on the initial conditions and the rest of the columns in the Header needs to skipped.

Eg -
Search only on basis of below format

Code:
ORDER|Header|Add..........................
Followed by details
and then by Terminate ie ORDER|T|............

If possible kindly share the entire code at once so that I can get a feel of it and make use of the same in my next codes which are related to it.

Moderator's Comments:
Mod Comment
Please wrap all code, files, input & output/errors in CODE tags.
It makes them easier to read and preserves spaces for indenting of fixed-width data.
Last edited by rbatte1; 10-20-2017 at 09:53 AM.. Reason: Code tags
# 4  
Old 10-20-2017
Not clear. What except removing "Modify" blocks needs to be done?
# 5  
Old 10-20-2017
Hello grvk101,

Not sure if you have pasted complete command, please copy complete command of mine and try, also you haven't showed us which O.S you are on, so in case you are on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk.

Also seems your very first post and last post having different requirement, please mention all details of your question in code tags.

Thanks,
R. Singh
# 6  
Old 10-20-2017
Hi,
If i understand correctly ?
This sed command write in a file all text under ..1618 and ..1618.
Code:
sed -E '/([^|]*\|){5}1618$/!d;:A;N;/\n([^|]*\|){4}1618$/!bA;w savefile' infile

# 7  
Old 10-20-2017
I'm expecting another shoe to drop soon with the rest of your requirements, but here are three simple awk scripts and one simple sed script that seem to do what you requested in post #1:
Code:
awk -F'|' '$1 == "ORDER" && $2 == "Header" && $3 == "Add",$1 == "ORDER" && $2 == "T"' file

awk '/^ORDER\|Header\|Add\|/,/^ORDER\|T\|/' file

awk 'substr($0, 1, 17) == "ORDER|Header|Add|",substr($0, 1, 8) == "ORDER|T|"' file

sed -n '/^ORDER|Header|Add|/,/^ORDER|T|/p' file

Maybe you can modify one of these to work with your other, unspecified requirements to get something that will work for you.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

2. Shell Programming and Scripting

Insert data based on pattern

if it is finding some data based on pattern 'test' then insert else if has no data based on the pattern 'test' then exit successfully cat file | grep test > file2 (3 Replies)
Discussion started by: jagu
3 Replies

3. Shell Programming and Scripting

Splitting textfile based on pattern and name new file after pattern

Hi there, I am pretty new to those things, so I couldn't figure out how to solve this, and if it is actually that easy. just found that awk could help:(. so i have a textfile with strings and numbers (originally copy pasted from word, therefore some empty cells) in the following structure: SC... (9 Replies)
Discussion started by: luja
9 Replies

4. Shell Programming and Scripting

Generate tabular data based on a column value from an existing data file

Hi, I have a data file with : 01/28/2012,1,1,98995 01/28/2012,1,2,7195 01/29/2012,1,1,98995 01/29/2012,1,2,7195 01/30/2012,1,1,98896 01/30/2012,1,2,7083 01/31/2012,1,1,98896 01/31/2012,1,2,7083 02/01/2012,1,1,98896 02/01/2012,1,2,7083 02/02/2012,1,1,98899 02/02/2012,1,2,7083 I... (1 Reply)
Discussion started by: himanish
1 Replies

5. Shell Programming and Scripting

how to get data from hex file using SED or AWK based on pattern sign

I have a binary (hex) file I need to parse to get some data which are encoded this way: .* b4 . . . 01 12 .* af .* 83 L1 x1 x2 xL 84 L2 y1 y2 yL By another words there is a stream of hexadecimal bytes (in my example separated by space for better readability). I need to get value stored in... (3 Replies)
Discussion started by: sameucho
3 Replies

6. Shell Programming and Scripting

Split a file into multiple files based on the input pattern

I have a file with lines something like. ...... 123_start ...... ....... 123_end .... ..... 456_start ...... ..... 456_end .... ..... 789_start .... .... 789_end (6 Replies)
Discussion started by: abinash
6 Replies

7. Shell Programming and Scripting

Merge two file data together based on specific pattern match

My input: File_1: 2000_t g1110.b1 abb.1 2001_t g1111.b1 abb.2 abb.2 g1112.b1 abb.3 2002_t . . File_2: 2000_t Ali england 135 abb.1 Zoe british 150 2001_t Ali england 305 g1111.b1 Lucy russia 126 (6 Replies)
Discussion started by: patrick87
6 Replies

8. Shell Programming and Scripting

Copy input file based on condition

Hi, I am new to unix shell programming. I want to write a shell script for a functionality existing in mainframe system. I have one file as below as input 123456 &__987 &12yuq abcdef _ referes to blank condition:whenever the input file is having &__ ,it should be replaced... (4 Replies)
Discussion started by: charan0703
4 Replies

9. Shell Programming and Scripting

Need script to take input from file, match on it in file 2 and input data

All, I am trying to figure out a script to run in windows that will allow me to match on First column in file1 to 8th Column in File2 then Insert file1 column2 to file2 column4 then create a new file. File1: 12345 Sam 12346 Bob 12347 Bill File2:... (1 Reply)
Discussion started by: darkoth
1 Replies

10. Shell Programming and Scripting

Truncating FILE data BASED ON A PATTERN

HI I HAVE A PROBLEM,MY SOURCE FILE IS OF PATTERN S1,E-Certified,29,29,2.7,Certified,4,3,2.7,,0,0,0 S2,Certified,4,3,2.7,,0,0,0,,0 S3,E-Certified,29,29,2.7,,0,0,0 S4,,0,0,0,,0,0,0,,0,0,0,,0,0,0 AND THE EXPECTED OUTPUT IS S1,E-Certified,29,29,2.7 S1,Certified,4,3,2.7... (1 Reply)
Discussion started by: pkumar3
1 Replies
Login or Register to Ask a Question