Visit Our UNIX and Linux User Community


Help with uniq or awk??


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with uniq or awk??
# 1  
Old 08-20-2009
Help with uniq or awk??

Hi, my dilemna is this:
example i got a file of fruit.txt which contains:


Apple 6
Apple_new 7
old_orange 9
orange 10


Is there any way for me to have an output of


Apple 13
Orange 19


using shell script:


If its duplicate names, it should have been alright but I'm lost as to what can I do in this scenario.
Any help is greatly appreciated.
# 2  
Old 08-20-2009
Is this a home work question?

What is your "real world problem"?

Last edited by Franklin52; 08-20-2009 at 06:04 AM..
# 3  
Old 08-20-2009
Quote:
Originally Posted by Franklin52
Is this a home work question?

What is your "real world problem"?

Its related to work. I have a system generated file which contains similar entries as stated above. I need to summarize it to something as I have stated in the output.


My algorithm so far is to grab first entry, and store it in a file. Grab second entry and search the file if the entry already exists, if it does, just grab the number and increment the total number for that entry. If the entry doesnt exists den store it in the file. And so on and so forth.


Issue is entries are not the same, and the pattern is not consistent.
Any ideas??
cheers.
# 4  
Old 08-20-2009
It's gonna be difficult to find a balanced solution, your data is not consistent...

Regards
# 5  
Old 08-20-2009
Quote:
Originally Posted by Franklin52
It's gonna be difficult to find a balanced solution, your data is not consistent...

Regards
hmmmm..
ok i went tru again the generated file, there seems to be a pattern..the first 60% of the word is the same, e.g.

Apple_new_101 15
Apple_newMandarin 6
OrangeMango_new 6
OrangeMango_old 5

Algo:
1.grab the first 60% of the name and store it in a temp_name
2.create a file to put the filtered list e.g. filtered.txt
3.search filtered.txt, if temp_name already exists on the file
4. if it does, grab the number then sum it with the existing number on that name.
5.if it doesnt, store the name and number, and proceed with the next entry.

does this sound feasible? its just im not fluid with shell scripts.
thanks

---------- Post updated at 10:36 AM ---------- Previous update was at 09:15 AM ----------

actually thats fine, leave it for now.
i think to make it easier is to have a proper grouping of data and work from there.
thanks for the help.
# 6  
Old 08-20-2009
Something like that?
Code:
awk -F'[_| ]' 'NF{a[$1]+=$NF;next}END{for(i in a)print i,a[i]}' file

# 7  
Old 08-21-2009
assume your post is just some sample data, so really up to your criteria base on what to categorize them into one, if simply as 'old' and 'new'. Then maybe below perl script can help you some:

Code:
while(<DATA>){
	chomp;
	my @tmp = split;
	$tmp[0]=~s/_?(old|new)_?//;
	$hash{$tmp[0]}+=$tmp[1];
}
foreach my $key (keys %hash){
	print $key," ",$hash{$key},"\n";
}
__DATA__
Apple 6
Apple_new 7
old_orange 9
orange 10


Previous Thread | Next Thread
Test Your Knowledge in Computers #28
Difficulty: Easy
Unix was the first fully supported 64-bit operating system for personal computers.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk or uniq

Hi Help, I have a file which looks like 1 20 30 40 50 60 6 2 20 30 40 50 60 8 7 20 30 40 50 60 7 4 30 40 50 60 70 8 5 30 40 50 60 70 9 2 30 40 50 60 70 8 I want the o/p as 1 20 30 40 50 60 6 4 30 40 50 60 70 8 Is there a way I can use uniq command or awk to do this? ... (11 Replies)
Discussion started by: Indra2011
11 Replies

2. Shell Programming and Scripting

awk compare and keep uniq

Hi all I was wondering if you may help me in resolving an issue. In particular I have a file like this: the ... represent different string and what I wrote Cur or Ent are the constant. Well, what I would like to obtain is a file in which are reported only the ID in which the second column... (6 Replies)
Discussion started by: giuliangiuseppe
6 Replies

3. Shell Programming and Scripting

Sort uniq or awk

Hi again, I have files with the following contents datetime,ip1,port1,ip2,port2,number How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up? Please mind the file may contain 100k lines. (8 Replies)
Discussion started by: LDHB2012
8 Replies

4. Shell Programming and Scripting

Rewriting GNU uniq in awk

Within a shell script I use uniq -w 16 -D in order to process all lines in which the first 16 characters are duplicated. Now I want to also run that script on a BSD based system where the included version of uniq does not support the -w (--check-chars) option. To get around this I have... (7 Replies)
Discussion started by: mij
7 Replies

5. Shell Programming and Scripting

awk uniq and longest string of a column as index

I met a challenge to filter ~70 millions of sequence rows and I want using awk with conditions: 1) longest string of each pattern in column 2, ignore any sub-string, as the index; 2) all the unique patterns after 1); 3) print the whole row; input: 1 ABCDEFGHI longest_sequence1 2 ABCDEFGH... (12 Replies)
Discussion started by: yifangt
12 Replies

6. Shell Programming and Scripting

awk - getting uniq count on multiple col

Hi My file have 7 column, FIle is pipe delimed Col1|Col2|col3|Col4|col5|Col6|Col7 I want to find out uniq record count on col3, col4 and col2 ( same order) how can I achieve it. ex 1|3|A|V|C|1|1 1|3|A|V|C|1|1 1|4|A|V|C|1|1 Output should be FREQ|A|V|3|2 FREQ|A|V|4|1 Here... (5 Replies)
Discussion started by: sanranad
5 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

Text Proccessing with sort,uniq,awk

Hello, I have a log file with the following input: X , ID , Date, Time, Y 01,01368,2010-12-02,09:07:00,Pass 01,01368,2010-12-02,10:54:00,Pass 01,01368,2010-12-02,13:07:04,Pass 01,01368,2010-12-02,18:54:01,Pass 01,01368,2010-12-03,09:02:00,Pass 01,01368,2010-12-03,13:53:00,Pass... (12 Replies)
Discussion started by: rollyah
12 Replies

9. Shell Programming and Scripting

How to replicate data using Uniq or awk

Hi, I have this scenario; where there are two classes:- apple and orange. 1,2,3,4,5,6,apple 1,1,0,4,2,3,apple 1,3,3,3,3,4,apple 1,1,1,1,1,1,orange 1,2,3,1,1,1,orange Basically for apple, i have 3 entries in the file, and for orange, I have 2 entries. Im trying to edit the file and find... (5 Replies)
Discussion started by: ahjiefreak
5 Replies

10. Shell Programming and Scripting

using uniq and awk??

I have a file that is populated: hits/books.hits:143.217.64.204 Thu Sep 21 22:24:57 GMT 2006 hits/books.hits:62.145.39.14 Fri Sep 22 00:38:32 GMT 2006 hits/books.hits:81.140.86.170 Fri Sep 22 08:45:26 GMT 2006 hits/books.hits:81.140.86.170 Fri Sep 22 09:13:57 GMT... (13 Replies)
Discussion started by: amatuer_lee_3
13 Replies

Featured Tech Videos