Using sed, awk or perl to remove substring of all lines except the first Post: 302800027

Sponsored Content

Top Forums Shell Programming and Scripting Using sed, awk or perl to remove substring of all lines except the first Post 302800027 by Don Cragun on Sunday 28th of April 2013 07:42:03 PM

04-28-2013

Registered User

I think this awk script does what you want (and you can turn it into a 1-liner if you insist, but I prefer readable). Note this this script also removes one of the duplicated group (blah) entries from the input line:

Code:

4444::othergroup2:user3,blah,blah,user1

Code:

awk '
BEGIN { FS = OFS = ":" }
{       n=split($4, g, /,/)
        for(i = 1; i <= n; i++)
                if(($1,g[i]) in key) {
                        for(j = i + 1; j <= n; j++) g[j - 1] = g[j]
                        i--
                        n--
                        c = 1
                } else  key[$1,g[i]]
        if(c) { c = 0
                $4 = n ? g[1] : ""
                for(j = 2; j <= n; j++) $4 = $4 "," g[j]
        }
        print
}' data

With your sample input, this script produces the output:

Code:

1234::group:user1,user2,user3,blah1,blah2,blah3
2222::othergroup:user9,user8
4444::othergroup2:user3,blah,user1
1234::group3:user5

As always, if you're using a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of awk.

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ...

2. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks.

3. Shell Programming and Scripting

perl or awk remove empty lines when condition

Hi Everyone, # cat 1 a b b cc 1 2 3 3 3 4 55 5 a b

4. Shell Programming and Scripting

How to remove spaces using awk,sed,perl?

Input: 3456 565 656 878 235 8 4 8787 3 7 35 878 Expected output: 3456 565 656 878 235 8 4 8787 3 7 35 878 How can i do this with awk,sed and perl?

5. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance

6. Shell Programming and Scripting

Need an awk / sed / or perl one-liner to remove last 4 characters with non-unique pattern.

Hi, I'm writing a ksh script and trying to use an awk / sed / or perl one-liner to remove the last 4 characters of a line in a file if it begins with a period. Here is the contents of the file... the column in which I want to remove the last 4 characters is the last column. ($6 in awk). I've...

7. Shell Programming and Scripting

Process alternate lines in awk/sed/perl

hi.. i have a fasta file with the following format >sequence1 CCGGTTTTCGATTTGGTTTGACT >sequence2 AAAGTGCCGCCAGGTTTTGAGTGT >sequence3 AGTGCCGCAGAGTTTGTAGTGT Now, i want to read alternate line and add "GGGGGGGGGGG" to end of every sequence Desired output: >sequence1...

8. Shell Programming and Scripting

Sed/awk/perl substitution with multiple lines

OSX I have been grinding my teeth on a portion of code. I am building a bash script that edits a html email template. In the template, I have place holders for SED (or whatever program is appropriate) to use as anchors for find and replace, with user defined corresponding html code. The HTML code...

9. Shell Programming and Scripting

Remove lines matching a substring in a specific column

Dear group, I have following input text file: Brit 2016 11 18 12 00 10 1.485,00 EUR Brit 2016 11 18 12 00 10 142,64 EUR Brit 2016 11 18 12 00 10 19,80 EUR Brit 2016 11 18 12 00 10 545,00 EUR Brit 2016 11 18 12 00 10 6.450,00 EUR...

10. UNIX for Beginners Questions & Answers

awk with sed to combine lines and remove specific odd # pattern from line

In the awk piped to sed below I am trying to format file by removing the odd xxxx_digits and whitespace after, then move the even xxxx_digit to the line above it and add a space between them. There may be multiple lines in file but they are in the same format. The Filename_ID line is the last line...

LEARN ABOUT DEBIAN

time::duration::parse

Time::Duration::Parse(3pm)				User Contributed Perl Documentation				Time::Duration::Parse(3pm)

NAME

       Time::Duration::Parse - Parse string that represents time duration

SYNOPSIS

	 use Time::Duration::Parse;

	 my $seconds = parse_duration("2 minutes and 3 seconds"); # 123

DESCRIPTION

       Time::Duration::Parse is a module to parse human readable duration strings like 2 minutes and 3 seconds to seconds.

       It does the opposite of duration_exact function in Time::Duration and is roundtrip safe. So, the following is always true.

	 use Time::Duration::Parse;
	 use Time::Duration;

	 my $seconds = int rand 100000;
	 is( parse_duration(duration_exact($seconds)), $seconds );

FUNCTIONS

       parse_duration
	     $seconds = parse_duration($string);

	   Parses duration string and returns seconds. When it encounters an error in a given string, it dies an exception saying "Unknown
	   timespec: blah blah blah". This function is exported by default.

AUTHOR

       Tatsuhiko Miyagawa <miyagawa@bulknews.net>

COPYRIGHT

       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

       Some internal code is taken from Cache and Cache::Cache modules on CPAN.

SEE ALSO

       Date::Manip, DateTime::Format::Duration, <http://use.perl.org/~miyagawa/journal/30310>

perl v5.10.0							    2008-06-02						Time::Duration::Parse(3pm)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sed or Awk to remove specific lines

Discussion started by: Shoeless_Mike

2. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Discussion started by: ashimada

3. Shell Programming and Scripting

perl or awk remove empty lines when condition

Discussion started by: jimmy_y

4. Shell Programming and Scripting

How to remove spaces using awk,sed,perl?

Discussion started by: cola

5. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Discussion started by: cola

6. Shell Programming and Scripting

Need an awk / sed / or perl one-liner to remove last 4 characters with non-unique pattern.

Discussion started by: right_coaster

7. Shell Programming and Scripting

Process alternate lines in awk/sed/perl

Discussion started by: empyrean

8. Shell Programming and Scripting

Sed/awk/perl substitution with multiple lines

Discussion started by: sudo

9. Shell Programming and Scripting

Remove lines matching a substring in a specific column

Discussion started by: gfhsd

10. UNIX for Beginners Questions & Answers

awk with sed to combine lines and remove specific odd # pattern from line

Discussion started by: cmccabe

LEARN ABOUT DEBIAN

time::duration::parse