Sponsored Content
Top Forums Shell Programming and Scripting Complex text parsing with speed/performance problem (awk solution?) Post 302781097 by Michael Stora on Friday 15th of March 2013 02:08:50 PM
Old 03-15-2013
RudiC, not to over complicate things, assume there is a loop through all the data files and the variables FilePathName, FilePath, FileName, Var1, ..., Var6 are defined within the loop before the code is called.

The transpose with padding code will be called for one file at a time within the loop. If awk is used, the variables can be passed using the -v option as needed. Var1-6 are not needed in the awk so they can be passed as one concatenated string.

After the loop, the sort is done and duplicate columns will be deleted. If you don't want to keep it general, assume it is the last two colums FilePath and FileName that are not part of the duplicate line determination.

Mike
 

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk parsing problem

I need help with a problem that I have not been able to figure out. I have a file that is about 650K lines. Records are seperated by blank lines, fields seperated by new lines. I was trying to make a report that would add up 2 fields and associate them with a CP. example output would be... (11 Replies)
Discussion started by: timj123
11 Replies

2. Shell Programming and Scripting

Parsing a complex log file

I have a log file that has many SQL statements/queries/blocks and their resultant output (success or failure) added to each of them. I need to pick up all the statements which caused errors and write them to a separate file. On most cases, the SQL statement is a single line, like DROP . And if... (1 Reply)
Discussion started by: exchequer598
1 Replies

3. Shell Programming and Scripting

Difficult problem: Complex text file manipulation in bash script.

I don't know if this is a big issue or not, but I'm having difficulties. I apoligize for the upcoming essay :o. I'm writing a script, similar to a paint program that edits images, but in the form of ANSI block characters. The program so far is working. I managed to save the image into a file,... (14 Replies)
Discussion started by: tinman47
14 Replies

4. Shell Programming and Scripting

Complex awk problem

hello, i have a complex awk problem... i have two tables, one with a value (0 to 1) and it's corresponding p-value, like this: 1. table: ______________________________ value p-value ... ... 0.254 0.003 0.245 0.005 0.233 0.006 ... ... ______________________________ and a... (6 Replies)
Discussion started by: dietmar13
6 Replies

5. Shell Programming and Scripting

Text string parsing in awk

I have a awk script that parses many millions of lines so performance is critical. At one point I am extracting some variables from a space delimited string. alarm = $11; len = split(alarm,a," "); ent = a; chem = a; for (i = 5; i<= len; i++) {chem = chem " " a}It works but is slow. Adding the... (7 Replies)
Discussion started by: Michael Stora
7 Replies

6. Shell Programming and Scripting

awk parsing problem

Hello fellow unix geeks, I am having a small dilemna trying to parse a log file I have. Below is a sample of what it will look like: MY_TOKEN1(group) TOKEN(other)|SSID1 MY_TOKEN2(group, group2)|SSID2 What I need to do is only keep the MY_TOKEN pieces and where there are multiple... (7 Replies)
Discussion started by: dagamier
7 Replies
foreach(n)						       Tcl Built-In Commands							foreach(n)

__________________________________________________________________________________________________________________________________________________

NAME
foreach - Iterate over all elements in one or more lists SYNOPSIS
foreach varname list body foreach varlist1 list1 ?varlist2 list2 ...? body _________________________________________________________________ DESCRIPTION
The foreach command implements a loop where the loop variable(s) take on values from one or more lists. In the simplest case there is one loop variable, varname, and one list, list, that is a list of values to assign to varname. The body argument is a Tcl script. For each element of list (in order from first to last), foreach assigns the contents of the element to varname as if the lindex command had been used to extract the element, then calls the Tcl interpreter to execute body. In the general case there can be more than one value list (e.g., list1 and list2), and each value list can be associated with a list of loop variables (e.g., varlist1 and varlist2). During each iteration of the loop the variables of each varlist are assigned consecutive values from the corresponding list. Values in each list are used in order from first to last, and each value is used exactly once. The total number of loop iterations is large enough to use up all the values from all the value lists. If a value list does not contain enough elements for each of its loop variables in each iteration, empty values are used for the missing elements. The break and continue statements may be invoked inside body, with the same effect as in the for command. Foreach returns an empty string. EXAMPLES
This loop prints every value in a list together with the square and cube of the value: set values {1 3 5 7 2 4 6 8} ;# Odd numbers first, for fun! puts "Value Square Cube" ;# Neat-looking header foreach x $values { ;# Now loop and print... puts " $x [expr {$x**2}] [expr {$x**3}]" } The following loop uses i and j as loop variables to iterate over pairs of elements of a single list. set x {} foreach {i j} {a b c d e f} { lappend x $j $i } # The value of x is "b a d c f e" # There are 3 iterations of the loop. The next loop uses i and j to iterate over two lists in parallel. set x {} foreach i {a b c} j {d e f g} { lappend x $i $j } # The value of x is "a d b e c f {} g" # There are 4 iterations of the loop. The two forms are combined in the following example. set x {} foreach i {a b c} {j k} {d e f g} { lappend x $i $j $k } # The value of x is "a d e b f g c {} {}" # There are 3 iterations of the loop. SEE ALSO
for(n), while(n), break(n), continue(n) KEYWORDS
foreach, iteration, list, looping Tcl foreach(n)
All times are GMT -4. The time now is 01:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy