Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

awk to remove redundancy, with conditions

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers awk to remove redundancy, with conditions
# 1  
Old 05-16-2013
awk to remove redundancy, with conditions
Hi Everyone,

I've been using awk '!x[$0]++' to get rid of redundancy in a file, but it's getting rid of some extra lines I need (like the plus signs in the example below).

So how would I go from this:

@XYZ
ABCD
345345
+
@MNO
EFGH
789789
+
@XYZ
ABCD
345345
+

to:

@XYZ
ABCD
345345
+
@MNO
EFGH
789789
+
# 2  
Old 05-17-2013
Take a look at this info:UNIX BASH scripting: Remove duplicates based on fields - AWK
# 3  
Old 05-17-2013
An approach using two awk pipelines:
Code:
$ awk '{$1=$1}1' RS="+" file | awk '{!A[$0]++}END{for(k in A){gsub(/.$/,"& +",k);gsub(OFS,RS,k);if(k) print k}}'
@XYZ
ABCD
345345
+
@MNO
EFGH
789789
+

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk problems - awk ignores conditions

awk 'BEGIN{ if('"$CATE"'<'"${WARN}"') printf ("%s", "'"`Kfunc "" ; break`"'") else if (('"${CATE}"'>='"${WARN}"') && ('"${CATE}"'<'"${CRIT}"')) printf ("%s", "'"`Wfunc ""; break`"'") else if ('"${CATE}"'>='"${CRIT}"') printf... (6 Replies)
Discussion started by: SkySmart
6 Replies

2. Shell Programming and Scripting

If conditions in awk

Hello Friends, I need to find some CDRs in production servers whose 1st field value and 2nd field value = 1 and 11th looks like 45.123... where there are more than 3 digits after comma.so i wrote a one liner, something like below but does not work, however when i used first and second conditions... (8 Replies)
Discussion started by: EAGL€
8 Replies

3. Shell Programming and Scripting

awk three conditions

I'm having a problem pulling UID's from data. The data outputs a user's UID in one of three ways: 1. Error User user_name already assigned with <UID> 2. Success <UID> reserved for user_name 3. <a load of crap because there was a db failure yet somehow the UID is still in there> I typically... (5 Replies)
Discussion started by: MaindotC
5 Replies

4. Shell Programming and Scripting

awk with conditions

Hi All, I have a file with below contents. "en2"/10.185.81.0:cluster_interconnect,"en5"/10.185.81.0:cluster_interconnect,"en6"/169.181.146.0:public I want to take the interface name from the file and convert it as ipaddress using ifconfig command get the output like below en6 ->... (2 Replies)
Discussion started by: kamauv234
2 Replies

5. Shell Programming and Scripting

Conditions in awk

Hi there, here is my command ssh host.local "/path/to/my/perscript/hostconfig.pl -s $HOST -d |awk '{if (\$4 > 120)print \"My error message\";exit}{s=0; for (i=1; i<=NF; i++) s++; if(s == 13) print \$3}'" The problem is if conditional 1 is met (i.e $4 > 120), i don't see "My error message", the... (5 Replies)
Discussion started by: urello
5 Replies

6. Shell Programming and Scripting

awk with two conditions

Hi Everyone, # cat 1 1;2;3;4;5;6 1;2;3;4;5; # awk -F ";" '$5 == "5"' 1 1;2;3;4;5;6 1;2;3;4;5; but the output is should be just "1;2;3;4;5;6" means 1st condition: $5 is 5; 2nd condition: $6 is not empty, please advice. Thanks (2 Replies)
Discussion started by: jimmy_y
2 Replies

7. UNIX for Dummies Questions & Answers

How to get remove duplicate of a file based on many conditions

Hii Friends.. I have a huge set of data stored in a file.Which is as shown below a.dat: RAO 1869 12 19 0 0 0.00 17.9000 82.3000 10.0 0 0.00 0 3.70 0.00 0.00 0 0.00 3.70 4 NULL LEE 1870 4 11 1 0 0.00 30.0000 99.0000 0.0 0 0.00 0 0.00 0.00 0.00 0 ... (3 Replies)
Discussion started by: reva
3 Replies

8. Shell Programming and Scripting

About awk conditions

Hello, Can you explain why in the first 2 commands the awk does not print anything? Is it looking of a specific format ? Thanks. $ echo 12a3 | awk '($1>=2) {print $1}' # prints nothing $ echo 123a | awk '($1>=2) {print $1}' # prints nothing $ echo a123 | awk '($1>=2) {print $1}' a123... (1 Reply)
Discussion started by: majormark
1 Replies

9. Shell Programming and Scripting

awk remove column with conditions?

Folks: I have a file which has 3 columns. Using awk I want to remove rows from column 3 (Col3 <> A) where it is not equal to A. All columns are seperated by "|". Col1|Col2|Col3|Col4 1 | 2 | A | 4 2 | 3 | A | 5 3 | 4 | B | 6 4 | 5 | A | 7 5 | 6 | ... (3 Replies)
Discussion started by: pr2003
3 Replies

10. Shell Programming and Scripting

How to remove the redundancy messages in a file

Hi I have a file in which there are lot of redundancy messages.. Like suppose i have the messages like the following in my file.. 45123|1|784tryr|uirirr|wagras|swyuyau|| 45123|1|784tryr|uirirr|wagras|swyuyau|| I need to remove the extra (45123|1|784tryr|uirirr|wagras|swyuyau||) and need... (1 Reply)
Discussion started by: grajesh_955
1 Replies
Login or Register to Ask a Question