Using sed, awk or perl to remove substring of all lines except the first


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using sed, awk or perl to remove substring of all lines except the first
# 8  
Old 04-30-2013
Where is the genius button?! I was able to edit the script and change the variables around( I typed in the example in the wrong format) and voila! Many thanks to you for your awk expertise! To answer your question, it is runnning RHEL 5. Bash is the default. Thanks again!

---------- Post updated at 07:03 PM ---------- Previous update was at 06:55 PM ----------

Code:
awk
BEGIN { FS = OFS = ":" }
{       n=split($4, g, /,/)
        for(i = 1; i <= n; i++)
                if(($3,g[i]) in key) 
                {
                        for(j = i + 1; j <= n; j++) g[j - 1] = g[j]
                        i--
                        n--
                        c = 1
                } else  key[$3,g[i]]
                if(c) 
                { 
                    c = 0
                    $4 = n ? g[1] : ""
                    for (j = 2; j <= n; j++) $4 = $4 "," g[j]
                }
        print
}


Last edited by Franklin52; 05-01-2013 at 03:42 AM.. Reason: Please use code tags
# 9  
Old 05-18-2013
Don,

Thanks again for you assistance. I will keep this in my toolbox. One last request if you would, please. Can you post a few comments/pseudo code explaining your approach to this problem? I got as far as the key statement. It appears that you are looping through the array of sids from the 4th column, combining the GID and each sid as a search key. How are you performing the delete? Are you just re-writing the array without the duplicate sid? Is this in a temporary array?
# 10  
Old 05-18-2013
g[] is a list of user names on the current line. Duplicates are eliminated by removing entries from the g[] array and, if it is changed, reconstructing the 4th field on the current line before writing the updated line. key[gid, user_name] is a two dimensional array that keeps track of what user names have been seen for the gid on the current line (ignoring the group name). Here is a copy of my original script with extensive comments added. Let me know if something still is not clear.

Code:
awk '
# Input file format:
#       gid:do_not_care:gname;uname_list
# where:"gid" is a numeric string specifying the group ID number,
#       "do_not_care" is ignored by this script,
#       "gname" is an alphanumeric group name, and
#       "uname_list" is a comma separated list of alphanumeric user names.
#
# The two dimensional array key[] is indexed by the "gid" and a user name.  The
# array starts out empty.  If key[$1, user name] is present, the user name has
# been seen with the "gid" (either earlier on this line or on an earlier line).
# The "gname" is ignored when making this determination, so when we are done, a
# user name will appear only once for each "gid".
BEGIN { # Set the input and output field separators to ":"
        FS = OFS = ":"
}
{       # Split the "uname_list" into n individual user names:
        n = split($4, g, /,/)
        # Update the array of user names seen for this "gid":
        for(i = 1; i <= n; i++)
                # Determine if we have seen this user name with this "gid"
                if(($1,g[i]) in key) {
                        # We have seen this user name with this "gid".  Remove
                        # this name from the list of user names on this line:
                        for(j = i + 1; j <= n; j++) g[j - 1] = g[j]
                        i--     # Repeat the check for user name i.
                        n--     # Decrease the # of user names on this line.
                        c = 1   # Note that we have changed this line.
                } else  # We have not seen this user name with this "gid".
                        # Add an entry for this combination:
                        key[$1,g[i]]
        # Check to see if we need to reformat the "uname_list" (because we
        # removed a user name from the list on this line).
        if(c) { # We do need to reformat the uname_list on this line:
                c = 0     # Clear the flag for the next line.
                # If there are any user names left in the list, initialize the
                # reformatted "uname_list" to the 1st user name that is left;
                # otherwise set the "uname_list" to the empty string.
                # Note that if you want to discard this "gname" if there are no
                # remaining user names in the updated "uname_list", you could
                # do that by replacing the following uncommented line with the
                # next two lines (after removing the "# " in both lines):
                # if(n == 0) next
                # $4 = g[1]
                $4 = n ? g[1] : ""
                # For each additional remaining user name (if any exist), add a
                # comma and that name to the reformatted "uname_list":
                for(j = 2; j <= n; j++) $4 = $4 "," g[j]
        }
        # Print the original or updated line.
        print
}'  data

# 11  
Old 05-19-2013
Now, I see the missing piece. I was missing the algorithm to remove an item from the middle of an array. In other words:

To delete an element from the middle of an array:

Code:
1. There is no need to do anything with the element deleted.
2. Iterate along the array from 1 after the deleted element, to the last element.
3. Copy each element into the location 1 before it.
4. Set the last element to null.

Thanks again for sharing your knowledge and expertise!
# 12  
Old 05-19-2013
Quote:
Originally Posted by jacksolm
Now, I see the missing piece. I was missing the algorithm to remove an item from the middle of an array. In other words:

To delete an element from the middle of an array:

Code:
1. There is no need to do anything with the element deleted.
2. Iterate along the array from 1 after the deleted element, to the last element.
3. Copy each element into the location 1 before it.
4. Set the last element to null.

Thanks again for sharing your knowledge and expertise!
You're welcome. You almost described what the script is doing. Rather than setting the last element to null, the script I provided just reduces the number of elements to be processed in the array (that is what the n-- does). The former last element is still there unchanged, but it will be ignored when updating the last field just before printing the modified line. (Note that all of the elements in the array will be deleted and replaced with new elements the next time split() is called when awk is processing the next line.)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk with sed to combine lines and remove specific odd # pattern from line

In the awk piped to sed below I am trying to format file by removing the odd xxxx_digits and whitespace after, then move the even xxxx_digit to the line above it and add a space between them. There may be multiple lines in file but they are in the same format. The Filename_ID line is the last line... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

Remove lines matching a substring in a specific column

Dear group, I have following input text file: Brit 2016 11 18 12 00 10 1.485,00 EUR Brit 2016 11 18 12 00 10 142,64 EUR Brit 2016 11 18 12 00 10 19,80 EUR Brit 2016 11 18 12 00 10 545,00 EUR Brit 2016 11 18 12 00 10 6.450,00 EUR... (3 Replies)
Discussion started by: gfhsd
3 Replies

3. Shell Programming and Scripting

Sed/awk/perl substitution with multiple lines

OSX I have been grinding my teeth on a portion of code. I am building a bash script that edits a html email template. In the template, I have place holders for SED (or whatever program is appropriate) to use as anchors for find and replace, with user defined corresponding html code. The HTML code... (3 Replies)
Discussion started by: sudo
3 Replies

4. Shell Programming and Scripting

Process alternate lines in awk/sed/perl

hi.. i have a fasta file with the following format >sequence1 CCGGTTTTCGATTTGGTTTGACT >sequence2 AAAGTGCCGCCAGGTTTTGAGTGT >sequence3 AGTGCCGCAGAGTTTGTAGTGT Now, i want to read alternate line and add "GGGGGGGGGGG" to end of every sequence Desired output: >sequence1... (4 Replies)
Discussion started by: empyrean
4 Replies

5. Shell Programming and Scripting

Need an awk / sed / or perl one-liner to remove last 4 characters with non-unique pattern.

Hi, I'm writing a ksh script and trying to use an awk / sed / or perl one-liner to remove the last 4 characters of a line in a file if it begins with a period. Here is the contents of the file... the column in which I want to remove the last 4 characters is the last column. ($6 in awk). I've... (10 Replies)
Discussion started by: right_coaster
10 Replies

6. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance (9 Replies)
Discussion started by: cola
9 Replies

7. Shell Programming and Scripting

How to remove spaces using awk,sed,perl?

Input: 3456 565 656 878 235 8 4 8787 3 7 35 878 Expected output: 3456 565 656 878 235 8 4 8787 3 7 35 878 How can i do this with awk,sed and perl? (10 Replies)
Discussion started by: cola
10 Replies

8. Shell Programming and Scripting

perl or awk remove empty lines when condition

Hi Everyone, # cat 1 a b b cc 1 2 3 3 3 4 55 5 a b (2 Replies)
Discussion started by: jimmy_y
2 Replies

9. Shell Programming and Scripting

How to remove lines before and after with awk / sed ?

Hi guys, I need to remove the pattern (ID=180), one line before and four lines after. Thanks. (5 Replies)
Discussion started by: ashimada
5 Replies

10. Shell Programming and Scripting

Sed or Awk to remove specific lines

I have searched the forum for this - forgive me if I missed a previous post. I have the following file: blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah alter table "informix".esc_acct add constraint (foreign key (fi_id) references "informix".fi ... (5 Replies)
Discussion started by: Shoeless_Mike
5 Replies
Login or Register to Ask a Question