Complex text parsing with speed/performance problem (awk solution?) Post: 302781115

Sponsored Content

Top Forums Shell Programming and Scripting Complex text parsing with speed/performance problem (awk solution?) Post 302781115 by DGPickett on Friday 15th of March 2013 02:43:53 PM

03-15-2013

Registered User

I gotta take a sed swipe at this:

Code:

echo "var1,var2,var3,var4,var5,var6,$(
sed '
   1,10d
   s/^[^,]*, *\([^,]*[^ ,]\).*/\1/
   :l
   N
   s/\n[^,]*, *\([^,]*[^ ,]\).*/,\1/
   34q
   b l
  ' original_data_file_name | dos2unix
 ),/original/directory/path/of/data/file/,original_data_file_name"

Narrative for sed: delete 10 lines, make line 11 into csv column two without leading spaces and stop before any trailing spaces (assumes never blank, else we can either trim trailing spaces in one more 's' line or 't' detect no sub and sub in a bare comma), discarding the rest of that line, set a loop branch target 'l', get the next line, turn the linefeed and second line into a comma and column 2 as before without white space, quit if line 34, branch to 'l' otherwise. We can work on the raw file since space, comma are not different code in DOS, and maybe we do not need dos2unix unless there are other codes that need fixing; any carriage returns got tossed.

If the files are long, make your first command head?

DGPickett

View Public Profile for DGPickett

Find all posts by DGPickett

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk parsing problem

I need help with a problem that I have not been able to figure out. I have a file that is about 650K lines. Records are seperated by blank lines, fields seperated by new lines. I was trying to make a report that would add up 2 fields and associate them with a CP. example output would be...

2. Shell Programming and Scripting

Parsing a complex log file

I have a log file that has many SQL statements/queries/blocks and their resultant output (success or failure) added to each of them. I need to pick up all the statements which caused errors and write them to a separate file. On most cases, the SQL statement is a single line, like DROP . And if...

3. Shell Programming and Scripting

Difficult problem: Complex text file manipulation in bash script.

I don't know if this is a big issue or not, but I'm having difficulties. I apoligize for the upcoming essay :o. I'm writing a script, similar to a paint program that edits images, but in the form of ANSI block characters. The program so far is working. I managed to save the image into a file,...

4. Shell Programming and Scripting

Complex awk problem

hello, i have a complex awk problem... i have two tables, one with a value (0 to 1) and it's corresponding p-value, like this: 1. table: ______________________________ value p-value ... ... 0.254 0.003 0.245 0.005 0.233 0.006 ... ... ______________________________ and a...

5. Shell Programming and Scripting

Text string parsing in awk

I have a awk script that parses many millions of lines so performance is critical. At one point I am extracting some variables from a space delimited string. alarm = $11; len = split(alarm,a," "); ent = a; chem = a; for (i = 5; i<= len; i++) {chem = chem " " a}It works but is slow. Adding the...

6. Shell Programming and Scripting

awk parsing problem

Hello fellow unix geeks, I am having a small dilemna trying to parse a log file I have. Below is a sample of what it will look like: MY_TOKEN1(group) TOKEN(other)|SSID1 MY_TOKEN2(group, group2)|SSID2 What I need to do is only keep the MY_TOKEN pieces and where there are multiple...

LEARN ABOUT CENTOS

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk parsing problem

Discussion started by: timj123

2. Shell Programming and Scripting

Parsing a complex log file

Discussion started by: exchequer598

3. Shell Programming and Scripting

Difficult problem: Complex text file manipulation in bash script.

Discussion started by: tinman47

4. Shell Programming and Scripting

Complex awk problem

Discussion started by: dietmar13

5. Shell Programming and Scripting

Text string parsing in awk

Discussion started by: Michael Stora

6. Shell Programming and Scripting

awk parsing problem

Discussion started by: dagamier

LEARN ABOUT CENTOS

igawk