SED help - cleaning up code, extra spaces won't go away

04-24-2010

Registered User

6, 0

Join Date: May 2009

Last Activity: 10 February 2011, 1:41 PM EST

Posts: 6

Thanks Given: 1

Thanked 0 Times in 0 Posts

SED help - cleaning up code, extra spaces won't go away

Hello,

W/in the script I'm working on, I have a need to take a column from a file, and format it so I can have a variable that will egrep for & invert the regex from another file.

My solution is this:

Code:

VAR=`awk -F, '{print $2}' $FAIL | sed 's/-i/\|/g'`
VAR2=`echo $VAR | sed 's/ //g;s/^.\{1\}//g'`

egrep "$VAR2" file.txt >> newfile.txt
egrep -v "$VAR2" file.txt >> newfile.txt

The above works, but it's just ugly sed code.

All together...
input.txt:

Code:

-h a, -i b, -j c
-h d, -i e, -j f
-h g, -i h, -j i

Code:

VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g'`

... echo $VAR, prints: | b | e | h

So I'm running the variable again, through "another filter":

Code:

VAR2=`echo $VAR | sed 's/ //g;s/^.\{1\}//g'`

... echo $VAR2, prints b|e|h
... which is what I'd need for a good egrep command.

What I don't understand, why 'this' isn't working:

Code:

VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`

???
It'll produce something like this:
b e h
... I'm losing the "|", and the spaces are still there?

Can someone please help me understand what I'm doing wrong w/ SED? I can get the results I want, but it's ugly. Is what I'm doing correct? Lastly... if there's nothing wrong w/ how I'm doing things... is there a better or more efficient way?

Thanks everybody.

Last edited by vgersh99; 04-24-2010 at 09:53 AM.. Reason: code tag added

Matthias03

View Public Profile for Matthias03

Find all posts by Matthias03

04-24-2010

Moderator

8,825, 1,112

Join Date: Feb 2005

Last Activity: 23 August 2021, 11:26 AM EDT

Location: Foxborough, MA

Posts: 8,825

Thanks Given: 579

Thanked 1,112 Times in 1,003 Posts

nawk -f matt.awk input.txt file.txt

matt.awk:

Code:

BEGIN {
  FS=","
}
FNR==NR {
   match($2, "[^ ][^ ]*$")
   str=substr($2, RSTART)
   regex=(!regex)?str:regex "|" str
   next
}
$0 !~ regex

vgersh99

View Public Profile for vgersh99

Find all posts by vgersh99

04-24-2010

Registered User

2,100, 402

Join Date: Apr 2009

Last Activity: 11 February 2020, 10:24 AM EST

Posts: 2,100

Thanks Given: 26

Thanked 402 Times in 360 Posts

Quote:

Originally Posted by Matthias03

...
What I don't understand, why 'this' isn't working:

Code:

VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`

???
It'll produce something like this:
b e h
... I'm losing the "|", and the spaces are still there?
...

Well, I do see the "|" characters in my output:

Code:

$ 
$ cat -n input.txt
     1    -h a, -i b, -j c
     2    -h d, -i e, -j f
     3    -h g, -i h, -j i
$ 
$ awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'
|b
|e
|h
$

But if I assign the quoted output of that pipeline to a shell variable, then spaces are introduced.

Code:

$ 
$ VAR="`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`"
$ 
$ echo $VAR
|b |e |h
$

The command pipeline's output did *not* have spaces at the end:

Code:

$ 
$ awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g' | od -bc
0000000 174 142 012 174 145 012 174 150 012
          |   b  \n   |   e  \n   |   h  \n
0000011
$

So I guess the shell replaces those newlines by blank spaces when it is assigned to a (shell) variable:

Code:

$ 
$ VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`
$ echo $VAR
|b |e |h
$ 
$ ## or quoted
$ VAR="`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`"
$ echo $VAR
|b |e |h
$ 
$ echo $VAR | od -bc
0000000 174 142 040 174 145 040 174 150 012
          |   b       |   e       |   h  \n
0000011
$

Here's another way to extract a pipe-delimited output from the file using plain awk:

Code:

$ 
$ cat -n input.txt
     1    -h a, -i b, -j c
     2    -h d, -i e, -j f
     3    -h g, -i h, -j i
$ 
$ awk '{sub(",","",$4); x = NR==1 ? $4 : x"|"$4} END{print x}' input.txt
b|e|h
$

tyler_durden

durden_tyler

View Public Profile for durden_tyler

Find all posts by durden_tyler

04-24-2010

Registered User

3,231, 978

Join Date: Dec 2009

Last Activity: 11 June 2014, 8:40 PM EDT

Posts: 3,231

Thanks Given: 179

Thanked 978 Times in 791 Posts

Quote:

Originally Posted by durden_tyler

But if I assign the quoted output of that pipeline to a shell variable, then spaces are introduced.

The value of $VAR does contain the newlines. It may seem that the shell is converting newlines to spaces, but it is not. Since the variable expansion is unquoted, it's splitting the result into words at each of those newlines (and spaces and tabs, assuming a default value for IFS). echo never sees the newlines. echo does its job, printing its arguments as space-delimited list. If VAR is double-quoted, echo will be invoked with one argument which will contain newlines.

Code:

$ VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g'`
$ echo $VAR
| b | e | h
$ echo "$VAR"
 | b
 | e
 | h

Sidenote: Just as echo isn't seeing any of the newlines (because the shell "consumed" them during the field splitting step), echo is not seeing any of the spaces either. You are just less likely to miss them since echo prints a space to delimit its arguments, which are often space delimited to begin with. (Although you would notice that multiple-consecutive spaces are squeezed into one.)

Code:

$ VAR='a b c'
$ # Gives the impression that field splitting did not happen, but it did.
$ echo $VAR
a b c
$ VAR='a          b              c'
$ echo $VAR
a b c
$ echo "$VAR"
a          b              c

Regards,
Alister

Last edited by alister; 04-24-2010 at 12:40 PM..

alister

View Public Profile for alister

Find all posts by alister

04-24-2010

Moderator

12,296, 3,792

Join Date: Nov 2008

Last Activity: 1 January 2021, 1:47 AM EST

Location: Amsterdam

Posts: 12,296

Thanks Given: 679

Thanked 3,792 Times in 3,282 Posts

In short, try using:

Code:

awk -F, '{printf $2}' input.txt

Scrutinizer

View Public Profile for Scrutinizer

Find all posts by Scrutinizer

04-24-2010

Registered User

3,231, 978

Join Date: Dec 2009

Last Activity: 11 June 2014, 8:40 PM EDT

Posts: 3,231

Thanks Given: 179

Thanked 978 Times in 791 Posts

Quote:

Originally Posted by Matthias03

What I don't understand, why 'this' isn't working:

Code:

VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`

I cannot reproduce the "b e h" pipe-loss result.

My results:

Code:

$ cat input.txt 
-h a, -i b, -j c
-h d, -i e, -j f
-h g, -i h, -j i
$ VAR=`awk -F, '{print $2}' input.txt | sed 's/-i/\|/g;s/^.\{1\}//g;s/ //g'`
$ echo $VAR
|b |e |h

Personally, I'm partial to ...

Code:

awk -F'[, ]+' '{print $4}' input.txt | paste -sd\| -
b|e|h

... and ...

Code:

awk -F'[, ]+' '{printf("%s", (NR!=1 ? "|" : "") $4)}' input.txt
b|e|h

Regards,
Alister

Last edited by alister; 04-24-2010 at 03:01 PM..

alister

View Public Profile for alister

Find all posts by alister

Shell Programming and Scripting

SED help - cleaning up code, extra spaces won't go away

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep string causes extra spaces

Discussion started by: baris35

2. Shell Programming and Scripting

Have problem with extra EOLs in my CSV - need help cleaning out

Discussion started by: richardsantink

3. Shell Programming and Scripting

Removing extra unwanted spaces

Discussion started by: anshaa

4. Shell Programming and Scripting

Removing extra unwanted spaces

Discussion started by: anshaa

5. Shell Programming and Scripting

Remove of extra spaces from the trailing

Discussion started by: bhanudhingra

6. Shell Programming and Scripting

clear extra spaces and tabs in a file

Discussion started by: goofist

7. Shell Programming and Scripting

How to remove extra spaces from a string??

Discussion started by: vanitham

8. Shell Programming and Scripting

remove extra spaces between fields

Discussion started by: srilaxmi

9. Shell Programming and Scripting

Remove extra spaces in a line

Discussion started by: vikas027

10. UNIX for Dummies Questions & Answers

To remove the extra spaces in unix

Discussion started by: Sho