Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Complex text parsing with speed/performance problem (awk solution?)
Top Forums Shell Programming and Scripting Complex text parsing with speed/performance problem (awk solution?) Post 302781097 by Michael Stora on Friday 15th of March 2013 02:08:50 PM
Old 03-15-2013
RudiC, not to over complicate things, assume there is a loop through all the data files and the variables FilePathName, FilePath, FileName, Var1, ..., Var6 are defined within the loop before the code is called.

The transpose with padding code will be called for one file at a time within the loop. If awk is used, the variables can be passed using the -v option as needed. Var1-6 are not needed in the awk so they can be passed as one concatenated string.

After the loop, the sort is done and duplicate columns will be deleted. If you don't want to keep it general, assume it is the last two colums FilePath and FileName that are not part of the duplicate line determination.

Mike
 

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk parsing problem

I need help with a problem that I have not been able to figure out. I have a file that is about 650K lines. Records are seperated by blank lines, fields seperated by new lines. I was trying to make a report that would add up 2 fields and associate them with a CP. example output would be... (11 Replies)
Discussion started by: timj123
11 Replies

2. Shell Programming and Scripting

Parsing a complex log file

I have a log file that has many SQL statements/queries/blocks and their resultant output (success or failure) added to each of them. I need to pick up all the statements which caused errors and write them to a separate file. On most cases, the SQL statement is a single line, like DROP . And if... (1 Reply)
Discussion started by: exchequer598
1 Replies

3. Shell Programming and Scripting

Difficult problem: Complex text file manipulation in bash script.

I don't know if this is a big issue or not, but I'm having difficulties. I apoligize for the upcoming essay :o. I'm writing a script, similar to a paint program that edits images, but in the form of ANSI block characters. The program so far is working. I managed to save the image into a file,... (14 Replies)
Discussion started by: tinman47
14 Replies

4. Shell Programming and Scripting

Complex awk problem

hello, i have a complex awk problem... i have two tables, one with a value (0 to 1) and it's corresponding p-value, like this: 1. table: ______________________________ value p-value ... ... 0.254 0.003 0.245 0.005 0.233 0.006 ... ... ______________________________ and a... (6 Replies)
Discussion started by: dietmar13
6 Replies

5. Shell Programming and Scripting

Text string parsing in awk

I have a awk script that parses many millions of lines so performance is critical. At one point I am extracting some variables from a space delimited string. alarm = $11; len = split(alarm,a," "); ent = a; chem = a; for (i = 5; i<= len; i++) {chem = chem " " a}It works but is slow. Adding the... (7 Replies)
Discussion started by: Michael Stora
7 Replies

6. Shell Programming and Scripting

awk parsing problem

Hello fellow unix geeks, I am having a small dilemna trying to parse a log file I have. Below is a sample of what it will look like: MY_TOKEN1(group) TOKEN(other)|SSID1 MY_TOKEN2(group, group2)|SSID2 What I need to do is only keep the MY_TOKEN pieces and where there are multiple... (7 Replies)
Discussion started by: dagamier
7 Replies

LEARN ABOUT PLAN9

pr

PR(1)							      General Commands Manual							     PR(1)

NAME

       pr - print file

SYNOPSIS

       pr [ option ...	] [ file ...  ]

DESCRIPTION

       Pr  produces  a printed listing of one or more files on its standard output.  The output is separated into pages headed by a date, the name
       of the file or a specified header, and the page number.	With no file arguments, pr prints its standard input.

       Options apply to all following files but may be reset between files:

       -n     Produce n-column output.

       +n     Begin printing with page n.

       -b     Balance columns on last page, in case of multi-column output.

       -d     Double space.

       -en    Set the tab stops for input text every n spaces.

       -h     Take the next argument as a page header (file by default).

       -in    Replace sequences of blanks in the output by tabs, using tab stops set every n spaces.

       -f     Use formfeeds to separate pages.

       -ln    Take the length of the page to be n lines instead of the default 66.

       -m     Print all files simultaneously, each in one column.

       -n     Number the lines of each file.

       -on    Offset the left margin n character positions.

       -sc    Separate columns by the single character c instead of aligning them with white space.  A missing c is taken to be a tab.

       -t     Do not print the 5-line header or the 5-line trailer normally supplied for each page.

       -wn    For multi-column output, take the width of the page to be n characters instead of the default 72.

SOURCE

       /sys/src/cmd/pr.c

SEE ALSO

       cat(1), lp(1)

																	     PR(1)
All times are GMT -4. The time now is 07:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy