Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using sed, awk or perl to remove substring of all lines except the first

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using sed, awk or perl to remove substring of all lines except the first
# 1  
Old 04-28-2013
Using sed, awk or perl to remove substring of all lines except the first
Greetings All,

I would like to find all occurences of a pattern and delete a substring from the all matching lines EXCEPT the first. For example:

1234::group:user1,user2,user3,blah1,blah2,blah3
2222::othergroup:user9,user8
4444::othergroup2:user3,blah,blah,user1
1234::group3:user5,user1


This should be for all combinations of gid and user. If this can be accomplished using a sed or awk one liner that would be great. Otherwise, I guess I'll try a Perl script.

Any ideas to get me started will be greatly appreciated. I'm thinking a nested for loop with an internal sed call would be a start.
# 2  
Old 04-28-2013
It's unclear to me what output you want, and no point in guessing. Smilie

Also, please use code tags. Smilie
# 3  
Old 04-28-2013
I think this awk script does what you want (and you can turn it into a 1-liner if you insist, but I prefer readable). Note this this script also removes one of the duplicated group (blah) entries from the input line:
Code:
4444::othergroup2:user3,blah,blah,user1

Code:
awk '
BEGIN { FS = OFS = ":" }
{       n=split($4, g, /,/)
        for(i = 1; i <= n; i++)
                if(($1,g[i]) in key) {
                        for(j = i + 1; j <= n; j++) g[j - 1] = g[j]
                        i--
                        n--
                        c = 1
                } else  key[$1,g[i]]
        if(c) { c = 0
                $4 = n ? g[1] : ""
                for(j = 2; j <= n; j++) $4 = $4 "," g[j]
        }
        print
}' data

With your sample input, this script produces the output:
Code:
1234::group:user1,user2,user3,blah1,blah2,blah3
2222::othergroup:user9,user8
4444::othergroup2:user3,blah,user1
1234::group3:user5

As always, if you're using a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of awk.
# 4  
Old 04-28-2013
Here's a little background information. Due to group member limitations we had to split groups into separate lines by using different group names and identical GIDs. A few help desk admins maintaining the NIS map entered users into all of the groups instead of selecting the latest group to add the user. A single GID could have several group names. I want to loop through the group file and remove the redundant entries.
# 5  
Old 04-28-2013
Quote:
Originally Posted by jacksolm
Here's a little background information. Due to group member limitations we had to split groups into separate lines by using different group names and identical GIDs. A few help desk admins maintaining the NIS map entered users into all of the groups instead of selecting the latest group to add the user. A single GID could have several group names. I want to loop through the group file and remove the redundant entries.
That was what I understood when I wrote the awk script for you. Did you decide not to try it because it is more than 1 line?
# 6  
Old 04-29-2013
Don,

Sorry for the delay. I just had a chance to try your solution. I had a few typos which resulted in syntax errors. After debugging, the script was able to execute. It is strange that awk complained about the single quote before BEGIN. It didn't appear to delete the user from the redundant group. Any ideas? I will post the edited script. I appreciate your time and awk expertise. Smilie
# 7  
Old 04-29-2013
Quote:
Originally Posted by jacksolm
Don,

Sorry for the delay. I just had a chance to try your solution. I had a few typos which resulted in syntax errors. After debugging, the script was able to execute. It is strange that awk complained about the single quote before BEGIN. It didn't appear to delete the user from the redundant group. Any ideas? I will post the edited script. I appreciate your time and awk expertise. Smilie
I suggest you copy the script I provided and save it into a file and execute that file. When I ran that code it produced exactly the output i listed right after the script. If it isn't doing that for you, you must still have some typos.

What are the results of running the command:
Code:
uname -a

on your system? What shell are you using?
This User Gave Thanks to Don Cragun For This Post:
jacksolm
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk with sed to combine lines and remove specific odd # pattern from line

In the awk piped to sed below I am trying to format file by removing the odd xxxx_digits and whitespace after, then move the even xxxx_digit to the line above it and add a space between them. There may be multiple lines in file but they are in the same format. The Filename_ID line is the last line... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

Remove lines matching a substring in a specific column

Dear group, I have following input text file: Brit 2016 11 18 12 00 10 1.485,00 EUR Brit 2016 11 18 12 00 10 142,64 EUR Brit 2016 11 18 12 00 10 19,80 EUR Brit 2016 11 18 12 00 10 545,00 EUR Brit 2016 11 18 12 00 10 6.450,00 EUR... (3 Replies)
Discussion started by: gfhsd
3 Replies

3. Shell Programming and Scripting

Sed/awk/perl substitution with multiple lines

OSX I have been grinding my teeth on a portion of code. I am building a bash script that edits a html email template. In the template, I have place holders for SED (or whatever program is appropriate) to use as anchors for find and replace, with user defined corresponding html code. The HTML code... (3 Replies)
Discussion started by: sudo
3 Replies

4. Shell Programming and Scripting

Process alternate lines in awk/sed/perl

hi.. i have a fasta file with the following format >sequence1 CCGGTTTTCGATTTGGTTTGACT >sequence2 AAAGTGCCGCCAGGTTTTGAGTGT >sequence3 AGTGCCGCAGAGTTTGTAGTGT Now, i want to read alternate line and add "GGGGGGGGGGG" to end of every sequence Desired output: >sequence1... (4 Replies)
Discussion started by: empyrean
4 Replies

5. Shell Programming and Scripting

Need an awk / sed / or perl one-liner to remove last 4 characters with non-unique pattern.

Hi, I'm writing a ksh script and trying to use an awk / sed / or perl one-liner to remove the last 4 characters of a line in a file if it begins with a period. Here is the contents of the file... the column in which I want to remove the last 4 characters is the last column. ($6 in awk). I've... (10 Replies)
Discussion started by: right_coaster
10 Replies

6. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance (9 Replies)
Discussion started by: cola
9 Replies

7. Shell Programming and Scripting

How to remove spaces using awk,sed,perl?

Input: 3456 565 656 878 235 8 4 8787 3 7 35 878 Expected output: 3456 565 656 878 235 8 4 8787 3 7 35 878 How can i do this with awk,sed and perl? (10 Replies)
Discussion started by: cola
10 Replies

8. Shell Programming and Scripting

perl or awk remove empty lines when condition

Hi Everyone, # cat 1 a b b cc 1 2 3 3 3 4 55 5 a b (2 Replies)
Discussion started by: jimmy_y
2 Replies

9. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks. (5 Replies)
Discussion started by: ashimada
5 Replies

10. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ... (5 Replies)
Discussion started by: Shoeless_Mike
5 Replies
Login or Register to Ask a Question