Sponsored Content
Top Forums Shell Programming and Scripting Read text between regexps and write into files based on a field in the text Post 302865193 by Don Cragun on Thursday 17th of October 2013 10:55:16 PM
Old 10-17-2013
Quote:
Originally Posted by r3d3
@Chubler_XL, @Don Cragun, thank you very much for your help. Both of these scripts worked on the sample I posted. When I tried it on the actual text file I have (about 600K lines, around 300-400 lines between start and end regexs), the scripts are taking a lot of time. Do you have any suggestions on reducing the process time?
What OS are you using? (I.e., what is the output from uname -a?)

How many different departments are in huge_file.txt?

Is there any chance that start_regexp and end_regexp occur in unmatched pairs? (My code will copy a start_regexp line found between a start_regext and the next end_regexp without restarting a copy, and will ignore an end_regexp if there was no start_regexp since the last seen end_regexp.)

Will there ever be a sequence of lines between the start and end lines that does not contain a Department=value line?

Answers to the above questions could be used to improve speed with an increased chance of things going wrong if the input data is malformed for some reason.

Expanding on what Chubler_XL said: If your input and output files are on different disk controllers, that might improve performance. If your input files are on one filesystem on a disk drive and your output files are on a different filesystem on the same drive, that will be worse than having the input and output files on the same filesystem.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove spaces from first field, and write entire contents into other text file

Hi all, I have searched and found various threads about removing spaces from a field within a text file. Unfortunately, I have not found exactly what I'm looking for, nor am I adept enough to modify what I've found into what I need. I use the following command to remove the first line... (3 Replies)
Discussion started by: carriehoff
3 Replies

2. Shell Programming and Scripting

match text from two files and write to a third file

Hi all I have two files X.txt and Y.txt. Both file contains same number of sentences. The content of X.txt is The filter described above may be combined. and the content of Y.txt is The filter describ+ed above may be combin+ed. Some of the words are separated with "+"... (2 Replies)
Discussion started by: my_Perl
2 Replies

3. Shell Programming and Scripting

Aligning text files by max field length

Hello, Is there anyway that I can align a pipe delimited text file by the maxium field length where the field is separated out by pipes for large text files with more than 100,000 rows? So, far I have searched other forums and google about aligning text files in unix and I have noticed that... (7 Replies)
Discussion started by: physalis2099
7 Replies

4. Shell Programming and Scripting

how read specific line in a file and write it in a new text file?

I have list of files in a directory 'dir'. Each file is of type HTML. I need to read each file and get the string which starts with 'http' and write them in a new text file. How can i do this shell scripting? file1.html <head> <url>http://www.google.com</url> </head> file2.html <head>... (6 Replies)
Discussion started by: vel4ever
6 Replies

5. Shell Programming and Scripting

Read in 2-column CSV, output many files based on field

Is there a way to read in a two-columned CSV file, and based on the fields in 1st column, output many different files? The input/output looks something like: input.csv: call Call Mom. call Call T-Mobile. go Go home. go Go to school. go Go to gas station. play Play music. play Play... (4 Replies)
Discussion started by: pxalpine
4 Replies

6. Shell Programming and Scripting

Read n lines from a text files getting n from within the text file

I dont even have a sample script cause I dont know where to start from. My data lookes like this > sat#16 #data: 15 site:UNZA baseline: 205.9151 0.008 -165.2465 35.8109 40.6685 21.9148 121.1446 26.4629 -18.4976 33.8722 0.017 -165.2243 48.2201 40.6908 ... (8 Replies)
Discussion started by: malandisa
8 Replies

7. Shell Programming and Scripting

awk to skip lines find text and add text based on number

I am trying to use awk skip each line with a ## or # and check each line after for STB= and if that value in greater than or = to 0.8, then at the end of line the text "STRAND BIAS" is written in else "GOOD". So in the file of 4 entries attached. awk tried: awk NR > "##"' "#" -F"STB="... (6 Replies)
Discussion started by: cmccabe
6 Replies

8. UNIX for Beginners Questions & Answers

awk GSUB read field values from multiple text files

My program run without error. The problem I am having. The program isn't outputting field values with the column headers to file.txt. Each of the column headers in file.txt has no data. MEMSIZE SECOND SASFoundation Filename The output results in file.txt should show: ... (1 Reply)
Discussion started by: dellanicholson
1 Replies

9. Shell Programming and Scripting

awk to print lines based on text in field and value in two additional fields

In the awk below I am trying to print the entire line, along with the header row, if $2 is SNV or MNV or INDEL. If that condition is met or is true, and $3 is less than or equal to 0.05, then in $7 the sub pattern :GMAF= is found and the value after the = sign is checked. If that value is less than... (0 Replies)
Discussion started by: cmccabe
0 Replies

10. Shell Programming and Scripting

awk to adjust text and count based on value in field

The below awk executes as is and produces the current output. It isvery close but what Ican not seem to do is add the -exon..., the ... portion comes from $1 and the _exon is static and will never change. If there is + sign in $4 then the ... is in acending order or sequential. If there is a - in... (2 Replies)
Discussion started by: cmccabe
2 Replies
g3cat(1)						       mgetty+sendfax manual							  g3cat(1)

NAME
g3cat - concatenate multiple g3 documents SYNOPSIS
g3cat [-l] [-a] g3-file1 ... DESCRIPTION
g3cat concatenates g3 files. These can either be 'raw', that is, bitmaps packed according to the CCITT T.4 standard for one-dimensional bitmap encoding, or 'digifax' files, created by GNU's GhostScript package with the digifax drivers. Its output is a concatenation of all the input files, in raw G3 format, with two white lines in between. If a - is given as input file, stdin is used. If the input data is malformed, a warning is printed to stderr, and the output file will have a blank line at this place. OPTIONS
-l separate files with a one-pixel wide black line. -h <blank lines> specifies the number of blank lines g3cat should prepend to each page. Default is 0. -L <lines> limit lenght of output page to maximum <lines> lines. SPECIAL-CASE OPTIONS -w <width> specifies the desired page width in pixels per line. Default is 1728 PELs, and this is mandatory if you want to send the fax to a standard fax machine. If one of the input files doesn't match this line width (for example because it was created by a broken G3 creator), a warning is printed, and the line width is transparently fixed. -a byte-align the end-of-line codes (EOL) in the file. Every EOL will end at a byte boundary, that is, with a 01 byte. -p <pad> specifies a minimum number of bytes that each output line must be padded to. Padding is done with 0-bits before the EOL code. -R suppress output of end-of-page code (RTC). Example The following example will put a header line on a given g3 page, 'page1' and put the result into 'page2': echo '$header' | pbmtext | pbm2g3 | g3cat - page1 >page2 FILES
-- BUGS
Hopefully none :-). SEE ALSO
g32pbm(1), sendfax(8), faxspool(1) AUTHORS
g3cat is Copyright (C) 1993 by Gert Doering, <gert@greenie.muc.de> greenie 27 Oct 93 g3cat(1)
All times are GMT -4. The time now is 02:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy