Understanding sed


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Understanding sed
# 1  
Old 03-01-2013
Understanding sed

Hi,

can some one suggest me,how "sed" is managed to delete the second field here.

Any explanation on , how the below code is working would be appreciated.

Code:
sed 's/^\([^:]*\):[^:]:/\1::/'  /etc/passwd

sed 's/[^:]*:/:/2'  /etc/passwd

# 2  
Old 03-01-2013
Hi, the first one deletes the second field only if it consists of one character... It replace the first field and a colon + a character and a colon with the first field (that was captured with \( .. \) and recalled by\1) and two colons

The second replaces any number of non-colons and a colon by a colon, the number two means the it does this with the second occurrence on a line...
This User Gave Thanks to Scrutinizer For This Post:
# 3  
Old 03-01-2013
Hi Scrutinizer,

Thanks for the reply. I got your second correctly.

Your first answer is slightly confusing me!..

Let's say the sample input is:

Code:
abcd:x:panyam:panyam:Panyam:512

echo "abcd:x:panyam:panyam:Panyam:512" | sed 's/^\([^:]*\):[^:]:/\1::/' gives me:

abcd::panyam:panyam:Panyam:512

\([^:]*\) matches : abcd
[^:] matches :x

\1 : prints only "abcd"..

Now, how the rest of the line is coming in output as it is? I mean
Code:
"panyam:panyam:Panyam:512"

.

Is it because sed prints the non matching patterns as it is?

Is my understanding correct?
# 4  
Old 03-01-2013
Hi Panyam, yes the rest of the line remains unaltered. It is not part of the substitution, so it gets printed like it is.
# 5  
Old 03-01-2013
An attempt to further explain the regular expression, with an ulterior motive of setting up for a question.

Given:
Code:
sed 's/^\([^:]*\):[^:]:/\1::/'

search for a pattern in the string matching:
Code:
^    = start of the line
\(   = Start of first remembered pattern
[^:] = followed by any character that is not a :
*    = followed by any number of the previous character class
       (characters that are not colons)
\)   = end of first remembered pattern
:    = followed by a colon
[^:] = followed by any character that is not a colon
:    = followed by a colon

<describes the first 2 fields, along with their seperators>

Replace with:
Code:
\1   = the first remembered pattern (the first field)
::   = followed by 2 literal colons

In other words, replace the first 2 colon separated fields
with the first field and 2 colons (deletes the 2nd field).

Question: if this sed command was in a script, could it be commented like I did above in the code? Can a sed regex be multi-line with comments?

One could also do:
Code:
s/:[^:]*:/::/

# 6  
Old 03-01-2013
Quote:
Originally Posted by gary_w
Question: if this sed command was in a script, could it be commented like I did above in the code? Can a sed regex be multi-line with comments?
Nope. You can have comments in a sed script, but not within a regular expression. What you are asking is possible with perl if you use the /x regular expression modifier.

Regards,
Alister

---------- Post updated at 10:54 PM ---------- Previous update was at 10:49 PM ----------

Quote:
Originally Posted by gary_w
One could also do:
Code:
s/:[^:]*:/::/

s/:[^:]*/:/ would work just as well, unless it's necessary to prevent the last field in a line without a trailing colon from matching, or even s/[^:]*//2.

REgards,
Alister
# 7  
Old 03-02-2013
Quote:
Originally Posted by alister
[..] or even s/[^:]*//2
That is what I would think too, but this does not work like that everywhere.. This works wiith GNU sed and sed on AIX7 and with regular sed on Solaris, but not with /usr/xpg4/bin/sed on Solaris nor with sed on HPUX and OSX and some other UNIX flavor.

In those cases where it does not work, the desired effect was obtained when s/[^:]*//3 was used instead (and for the 3rd field s/[^:]*//5 and so on).

How can this be? What I think this may have to do with how the respective regex engines interpret a zero match after a previous match. The first match of

echo aaa:bbb:ccc:ddd:eee:fff | sed 's/[^:]*//'

renders
Code:
:bbb:ccc:ddd:eee:fff

On this every engine agrees. After the first match the engine arrives after the previous match and before the first colon. But what then constitutes the next match? For GNU sed and some other mentioned above this apparently means the next iteration of non-colon characters after the first colon. But the other engines apparently interpret zero repetitions of the non-colons before the colon as the next match, which constitutes an empty string and which I guess could be labeled as a "strict" interpretation of [^:]*.

Anyway, it seems safest to include one colon in the match line in the OP's second example, or insist a pattern of 1 or more non-colons, i.e. sed 's/[^:][^:]*//2' or sed 's/[^:]\{1,\}//2'

Regards,

S.

Last edited by Scrutinizer; 03-02-2013 at 05:44 AM..
These 2 Users Gave Thanks to Scrutinizer For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need Quick help on Understanding sed Regex

Hi Guys, Could you please kindly explain what exactly the below SED command will do ? I am quite confused and i assumed that, sed 's/*$/ /' 1. It will remove tab and extra spaces .. with single space. The issue is if it is removing tab then it should be Î right .. please assist.... (3 Replies)
Discussion started by: Nandy
3 Replies

2. Shell Programming and Scripting

Need your help in understanding this

Hi, I found this in a script and I would like to know how this works Code is here: # var1=PART1_PART2 # var2=${var1##*_} # echo $var2 PART2 I'm wondering how ##* makes the Shell to understand to pick up the last value from the given. (2 Replies)
Discussion started by: sathyaonnuix
2 Replies

3. Shell Programming and Scripting

help understanding regex with grep & sed

I have the following line of code that works wonders. I just don't completely understand it as I am just starting to learn regex. Can you help me understand exactly what is happening here? find . -type f | grep -v '^\.$' | sed 's!\.\/!!' (4 Replies)
Discussion started by: trogdortheburni
4 Replies

4. UNIX for Dummies Questions & Answers

understanding sed command

Hi Friends, I need a small help in understanding the below sed command. $ cat t4.txt 1 root 1 58 0 888K 368K sleep 4:06 0.00% init 1 root 1 58 0 888K 368K sleep 4:06 0.00% init last $ sed 's/*$//' t4.txt 1 root 1 58 0 888K ... (3 Replies)
Discussion started by: forroughuse
3 Replies

5. UNIX for Dummies Questions & Answers

understanding {%/*}/

Hi Gurus: I am trying to understand the following line of code.I did enough of googling to understand but no luck.Please help me understand the follow chunk of code: X=$0 MOD=${X%/*}/env.ksh X is the current script from which I am trying to execute. Say if X=test.ksh $MOD is echoing :... (3 Replies)
Discussion started by: vemana
3 Replies

6. Shell Programming and Scripting

need help understanding mv

I just started shell coding and I'm a bit confused on how 'mv' works can someone explain to me how it works and if i did this correctly. Thanks. echo "Enter Name of the first file:" read file1 #echo $file1 if ; then echo "Sorry, file does not exist." exit 1 ... (16 Replies)
Discussion started by: taiL
16 Replies

7. UNIX for Dummies Questions & Answers

Help understanding sed

I am trying to create a basic script that converts an Oracle script into a Sybase script. The only things im changing are Datatypes and the to_char and to_date functions. I am not really 100% sure of the way it works. I have tried running the functions through a loop to replace each word line... (6 Replies)
Discussion started by: Makaer
6 Replies

8. Shell Programming and Scripting

understanding the sed command

Guys, I am trying to understand the sed command here. adx001 $ a=/clocal/dctrdata/user/dctrdat1/trdroot/recouncil adx001 $ b=`echo $a | sed 's/\//\\\\\//g'` adx001 $ echo $b \/clocal\/dctrdata\/user\/dctrdat1\/trdroot\/recouncil The sed command i took it from the script. Please... (3 Replies)
Discussion started by: mac4rfree
3 Replies

9. UNIX for Advanced & Expert Users

need further understanding of init.d

I needt o know how what init.d does and how it knows which dameons/applications to turn off and how to restart the applications after reboot. any OS - solaris/hp-ux (1 Reply)
Discussion started by: jigarlakhani
1 Replies
Login or Register to Ask a Question