Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Using sed, awk or perl to remove substring of all lines except the first
Top Forums Shell Programming and Scripting Using sed, awk or perl to remove substring of all lines except the first Post 302800027 by Don Cragun on Sunday 28th of April 2013 07:42:03 PM
Old 04-28-2013
I think this awk script does what you want (and you can turn it into a 1-liner if you insist, but I prefer readable). Note this this script also removes one of the duplicated group (blah) entries from the input line:
Code:
4444::othergroup2:user3,blah,blah,user1

Code:
awk '
BEGIN { FS = OFS = ":" }
{       n=split($4, g, /,/)
        for(i = 1; i <= n; i++)
                if(($1,g[i]) in key) {
                        for(j = i + 1; j <= n; j++) g[j - 1] = g[j]
                        i--
                        n--
                        c = 1
                } else  key[$1,g[i]]
        if(c) { c = 0
                $4 = n ? g[1] : ""
                for(j = 2; j <= n; j++) $4 = $4 "," g[j]
        }
        print
}' data

With your sample input, this script produces the output:
Code:
1234::group:user1,user2,user3,blah1,blah2,blah3
2222::othergroup:user9,user8
4444::othergroup2:user3,blah,user1
1234::group3:user5

As always, if you're using a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of awk.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ... (5 Replies)
Discussion started by: Shoeless_Mike
5 Replies

2. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks. (5 Replies)
Discussion started by: ashimada
5 Replies

3. Shell Programming and Scripting

perl or awk remove empty lines when condition

Hi Everyone, # cat 1 a b b cc 1 2 3 3 3 4 55 5 a b (2 Replies)
Discussion started by: jimmy_y
2 Replies

4. Shell Programming and Scripting

How to remove spaces using awk,sed,perl?

Input: 3456 565 656 878 235 8 4 8787 3 7 35 878 Expected output: 3456 565 656 878 235 8 4 8787 3 7 35 878 How can i do this with awk,sed and perl? (10 Replies)
Discussion started by: cola
10 Replies

5. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance (9 Replies)
Discussion started by: cola
9 Replies

6. Shell Programming and Scripting

Need an awk / sed / or perl one-liner to remove last 4 characters with non-unique pattern.

Hi, I'm writing a ksh script and trying to use an awk / sed / or perl one-liner to remove the last 4 characters of a line in a file if it begins with a period. Here is the contents of the file... the column in which I want to remove the last 4 characters is the last column. ($6 in awk). I've... (10 Replies)
Discussion started by: right_coaster
10 Replies

7. Shell Programming and Scripting

Process alternate lines in awk/sed/perl

hi.. i have a fasta file with the following format >sequence1 CCGGTTTTCGATTTGGTTTGACT >sequence2 AAAGTGCCGCCAGGTTTTGAGTGT >sequence3 AGTGCCGCAGAGTTTGTAGTGT Now, i want to read alternate line and add "GGGGGGGGGGG" to end of every sequence Desired output: >sequence1... (4 Replies)
Discussion started by: empyrean
4 Replies

8. Shell Programming and Scripting

Sed/awk/perl substitution with multiple lines

OSX I have been grinding my teeth on a portion of code. I am building a bash script that edits a html email template. In the template, I have place holders for SED (or whatever program is appropriate) to use as anchors for find and replace, with user defined corresponding html code. The HTML code... (3 Replies)
Discussion started by: sudo
3 Replies

9. Shell Programming and Scripting

Remove lines matching a substring in a specific column

Dear group, I have following input text file: Brit 2016 11 18 12 00 10 1.485,00 EUR Brit 2016 11 18 12 00 10 142,64 EUR Brit 2016 11 18 12 00 10 19,80 EUR Brit 2016 11 18 12 00 10 545,00 EUR Brit 2016 11 18 12 00 10 6.450,00 EUR... (3 Replies)
Discussion started by: gfhsd
3 Replies

10. UNIX for Beginners Questions & Answers

awk with sed to combine lines and remove specific odd # pattern from line

In the awk piped to sed below I am trying to format file by removing the odd xxxx_digits and whitespace after, then move the even xxxx_digit to the line above it and add a space between them. There may be multiple lines in file but they are in the same format. The Filename_ID line is the last line... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT CENTOS

locale::codes::script

Locale::Codes::Script(3)				User Contributed Perl Documentation				  Locale::Codes::Script(3)

NAME

       Locale::Codes::Script - standard codes for script identification

SYNOPSIS

	  use Locale::Codes::Script;

	  $script  = code2script('phnx');		      # 'Phoenician'
	  $code    = script2code('Phoenician'); 	      # 'Phnx'
	  $code    = script2code('Phoenician',
				 LOCALE_CODE_NUMERIC);	      # 115

	  @codes   = all_script_codes();
	  @scripts = all_script_names();

DESCRIPTION

       The "Locale::Codes::Script" module provides access to standards codes used for identifying scripts, such as those defined in ISO 15924.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 15924
       four-letter codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying scripts. A code set may be specified using either a name, or a constant
       that is automatically exported by this module.

       For example, the two are equivalent:

	  $script = code2script('phnx','alpha');
	  $script = code2script('phnx',LOCALE_SCRIPT_ALPHA);

       The codesets currently supported are:

       alpha, LOCALE_SCRIPT_ALPHA
	   This is a set of four-letter (capitalized) codes from ISO 15924 such as 'Phnx' for Phoenician.  It also includes additions to this set
	   included in the IANA language registry.

	   The Zxxx, Zyyy, and Zzzz codes are not used.

	   This is the default code set.

       num, LOCALE_SCRIPT_NUMERIC
	   This is a set of three-digit numeric codes from ISO 15924 such as 115 for Phoenician.

ROUTINES

       code2script ( CODE [,CODESET] )
       script2code ( NAME [,CODESET] )
       script_code2code ( CODE ,CODESET ,CODESET2 )
       all_script_codes ( [CODESET] )
       all_script_names ( [CODESET] )
       Locale::Codes::Script::rename_script  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::Script::add_script  ( CODE ,NAME [,CODESET] )
       Locale::Codes::Script::delete_script  ( CODE [,CODESET] )
       Locale::Codes::Script::add_script_alias	( NAME ,NEW_NAME )
       Locale::Codes::Script::delete_script_alias  ( NAME )
       Locale::Codes::Script::rename_script_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::Script::add_script_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::Script::delete_script_code_alias	( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.unicode.org/iso15924/
	   Home page for ISO 15924.

       http://www.iana.org/assignments/language-subtag-registry
	   The IANA language subtag registry.

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 1997-2001 Canon Research Centre Europe (CRE).
	  Copyright (c) 2001-2010 Neil Bowers
	  Copyright (c) 2010-2013 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.16.3							    2013-02-27						  Locale::Codes::Script(3)
All times are GMT -4. The time now is 04:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy