Read text between regexps and write into files based on a field in the text


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read text between regexps and write into files based on a field in the text
# 1  
Old 10-16-2013
Read text between regexps and write into files based on a field in the text

Hi,

I have a huge file that has data something like shown below:

huge_file.txt
Code:
start regexp
Name=Name1
Title=Analyst
Address=Address1
Department=Finance
end regexp
some text 
some text
start regexp
Name=Name2
Title=Controller
Address=Address2
Department=Finance
end regexp
some text 
some text
start regexp
Name=Name3
Title=Associate
Address=Address3
Department=Marketing
end regexp
some text
some text

I can extract the records between the start and end regular expressions using either Awk or Sed. It will really help me if I can read these records and write them into multiple files named after the Department, so they can be reviewed rather quickly.

Output I am expecting is -

Finance.txt
Code:
start regexp
Name=Name1
Title=Analyst
Address=Address1
Department=Finance
end regexp

start regexp
Name=Name2
Title=Controller
Address=Address2
Department=Finance
end regexp

Marketing.txt
Code:
start regexp
Name=Name3
Title=Associate
Address=Address3
Department=Marketing
end regexp

I thought using single command line as follows would work, but clearly I am missing something.
Code:
cat huge_file.txt | sed -n '/start regexp/,/end regexp/p' | tee record_buffer | grep Department | awk -F\= '{print $2}' | xargs cat record_buffer > {}

I think the problem was that output from sed is not single record but rather all the records between regexps. Any suggestions?
# 2  
Old 10-17-2013
Here is a solution in awk:

Code:
awk 'F{F=F "\n" $0}
/start regexp/ {F=$0}
/end regexp/ { 
  $0=F "\n"
  if(gsub(/.*Department=/, "", F)) gsub(/\n.*/, ".txt", F)
  else F="Unknown.txt"
  print > F
  F=x}' huge_file.txt


Last edited by Chubler_XL; 10-17-2013 at 01:21 AM.. Reason: -Added .txt to outfile names -Put missing Department blocks into Unknown.txt file
This User Gave Thanks to Chubler_XL For This Post:
# 3  
Old 10-17-2013
Hi,

Could you please use the following code.

Code:
awk '/start regexp/,/end regexp/' file_name | awk '/start regexp/ {print$NR}1'  | sed 's/^start$//g'

Output will be as follows.

Code:
start regexp
Name=Name1
Title=Analyst
Address=Address1
Department=Finance
end regexp
 
start regexp
Name=Name2
Title=Controller
Address=Address2
Department=Finance
end regexp
 
start regexp
Name=Name3
Title=Associate
Address=Address3
Department=Marketing
end regexp


Hope this wil help.


Thanks,
R. Singh
# 4  
Old 10-17-2013
Quote:
Originally Posted by RavinderSingh13
Hi,

Could you please use the following code.

Code:
awk '/start regexp/,/end regexp/' file_name | awk '/start regexp/ {print$NR}1'  | sed 's/^start$//g'

Output will be as follows.

Code:
start regexp
Name=Name1
Title=Analyst
Address=Address1
Department=Finance
end regexp
 
start regexp
Name=Name2
Title=Controller
Address=Address2
Department=Finance
end regexp
 
start regexp
Name=Name3
Title=Associate
Address=Address3
Department=Marketing
end regexp

Hope this wil help.


Thanks,
R. Singh
@ RavinderSingh13 User requirement is different please read thread before posting answer
# 5  
Old 10-17-2013
The awk script provided by ChublerXL should work well as long as huge_file.txt contains a small number of different departments. The number of open files allowed varies with different implementations of awk. If you have more than nine different departments, you might want to consider a slightly more complex awk script, such as:
Code:
awk -F'=' '
$0 == "start regexp" && !inre {
        inre = 1
        out = $0
        next
}
$0 == "end regexp" && inre {
        out = out "\n" $0 "\n"
        if(dept == "") dept = "Unknown"
        if(!(dept in depts)) {
                depts[dept] = dept ".txt"
                print "" > depts[dept]
        }
        print out >> depts[dept]
        close(depts[dept])
        out = dept = ""
        inre = 0
        next
}
inre {  out = out "\n" $0
        if($1 == "Department") dept = $2
}' huge_file.txt

As always, if you want to use awk on a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of just awk.
These 2 Users Gave Thanks to Don Cragun For This Post:
# 6  
Old 10-17-2013
@Chubler_XL, @Don Cragun, thank you very much for your help. Both of these scripts worked on the sample I posted. When I tried it on the actual text file I have (about 600K lines, around 300-400 lines between start and end regexs), the scripts are taking a lot of time. Do you have any suggestions on reducing the process time?
# 7  
Old 10-17-2013
Limits are most definitely implementation specific, GNU awk, for example, seems to have no limit on open files, apart from those enforced by the OS limits.

GNU awk 3.1.7 on RHEL 6.4 will allow 24768 and with GNU awk 4.1.0 on cygwin 6.1 I could get 1834.

---------- Post updated at 12:11 PM ---------- Previous update was at 11:54 AM ----------

I'm surprised this solution is running slow for you, awk is fairly efficient and even if it's re-coded in C I wouldn't expect much improvement.

Are you writing the output department files to the same disk as the huge_file.txt is stored on, contention between read and write may be slowing it down. If you have more than 1 drive in the system try writing the output elsewhere eg make current directory on /disk1/tmp and read input file from /disk2/data.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to adjust text and count based on value in field

The below awk executes as is and produces the current output. It isvery close but what Ican not seem to do is add the -exon..., the ... portion comes from $1 and the _exon is static and will never change. If there is + sign in $4 then the ... is in acending order or sequential. If there is a - in... (2 Replies)
Discussion started by: cmccabe
2 Replies

2. Shell Programming and Scripting

awk to print lines based on text in field and value in two additional fields

In the awk below I am trying to print the entire line, along with the header row, if $2 is SNV or MNV or INDEL. If that condition is met or is true, and $3 is less than or equal to 0.05, then in $7 the sub pattern :GMAF= is found and the value after the = sign is checked. If that value is less than... (0 Replies)
Discussion started by: cmccabe
0 Replies

3. UNIX for Beginners Questions & Answers

awk GSUB read field values from multiple text files

My program run without error. The problem I am having. The program isn't outputting field values with the column headers to file.txt. Each of the column headers in file.txt has no data. MEMSIZE SECOND SASFoundation Filename The output results in file.txt should show: ... (1 Reply)
Discussion started by: dellanicholson
1 Replies

4. Shell Programming and Scripting

awk to skip lines find text and add text based on number

I am trying to use awk skip each line with a ## or # and check each line after for STB= and if that value in greater than or = to 0.8, then at the end of line the text "STRAND BIAS" is written in else "GOOD". So in the file of 4 entries attached. awk tried: awk NR > "##"' "#" -F"STB="... (6 Replies)
Discussion started by: cmccabe
6 Replies

5. Shell Programming and Scripting

Read n lines from a text files getting n from within the text file

I dont even have a sample script cause I dont know where to start from. My data lookes like this > sat#16 #data: 15 site:UNZA baseline: 205.9151 0.008 -165.2465 35.8109 40.6685 21.9148 121.1446 26.4629 -18.4976 33.8722 0.017 -165.2243 48.2201 40.6908 ... (8 Replies)
Discussion started by: malandisa
8 Replies

6. Shell Programming and Scripting

Read in 2-column CSV, output many files based on field

Is there a way to read in a two-columned CSV file, and based on the fields in 1st column, output many different files? The input/output looks something like: input.csv: call Call Mom. call Call T-Mobile. go Go home. go Go to school. go Go to gas station. play Play music. play Play... (4 Replies)
Discussion started by: pxalpine
4 Replies

7. Shell Programming and Scripting

how read specific line in a file and write it in a new text file?

I have list of files in a directory 'dir'. Each file is of type HTML. I need to read each file and get the string which starts with 'http' and write them in a new text file. How can i do this shell scripting? file1.html <head> <url>http://www.google.com</url> </head> file2.html <head>... (6 Replies)
Discussion started by: vel4ever
6 Replies

8. Shell Programming and Scripting

Aligning text files by max field length

Hello, Is there anyway that I can align a pipe delimited text file by the maxium field length where the field is separated out by pipes for large text files with more than 100,000 rows? So, far I have searched other forums and google about aligning text files in unix and I have noticed that... (7 Replies)
Discussion started by: physalis2099
7 Replies

9. Shell Programming and Scripting

match text from two files and write to a third file

Hi all I have two files X.txt and Y.txt. Both file contains same number of sentences. The content of X.txt is The filter described above may be combined. and the content of Y.txt is The filter describ+ed above may be combin+ed. Some of the words are separated with "+"... (2 Replies)
Discussion started by: my_Perl
2 Replies

10. Shell Programming and Scripting

Remove spaces from first field, and write entire contents into other text file

Hi all, I have searched and found various threads about removing spaces from a field within a text file. Unfortunately, I have not found exactly what I'm looking for, nor am I adept enough to modify what I've found into what I need. I use the following command to remove the first line... (3 Replies)
Discussion started by: carriehoff
3 Replies
Login or Register to Ask a Question