Cleaning up Arrays with duplicate values


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Cleaning up Arrays with duplicate values
# 1  
Old 07-23-2010
Cleaning up Arrays with duplicate values

Hey Gang!

So I have two Arrays. @linecount and @hit. Now these arrays contain numbers which I'm using as line placeholders on files. I need these two arrays to sync up and not repeat a number.

Here are the contents (spam alert)

@linecount
Code:
1
28
53
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
175
175
175
176
176
177

and Here is

@hit
Code:
21
31
54
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
176
177
178
177
178
178

Notice how some of them have the same numbers, multiple times? I need it to quit that and merge them into a singular array with no duplicating numbers.

I hope this makes sense.
# 2  
Old 07-23-2010
Something like this:

Code:
@merge{@linecount, @hit} = (1) x @linecount + @hit;
@all_unique = sort { $a <=> $b } keys %merge;

# 3  
Old 07-24-2010
Hi

you can place the two arrays in two files- file1 and file2.
Code:
cat file2 >> file1 
sort -nu file1


Last edited by radoulov; 07-24-2010 at 04:31 AM.. Reason: Code tags, please!
# 4  
Old 07-24-2010
Quote:
Originally Posted by adelsin
(spam alert)
Spam in, spam out Smilie (Not sure I'm getting you right, though ...)

Code:
[house@leonov] cat data1
175
175
175
176
176
177
[house@leonov] cat data2
176
177
178
177
178
178
[house@leonov] paste data1 data2 | awk 'BEGIN {X=0} {if (X!=$1) {print $0; X=$1}}'
175     176
176     177
177     178

# 5  
Old 07-24-2010
Another way (assume sorted input files)
Code:
$ cat adelsin_1.txt
1
1
2
2
5
6
9
$ cat adelsin_2.txt
2
2
3
4
7
8
8
8
9
9
$ join -a 1 -a 2 adelsin_1.txt adelsin_2.txt  | uniq
1
2
3
4
5
6
7
8
9
$

Jean-Pierre.
# 6  
Old 07-24-2010
Thank you all for you responses. I'm looking at keep this answer primarily OS independant. Radoulov's answer seemed to work the best for me.

Additional question, how do I do the same thing with words and a single file?

---------- Post updated at 01:54 PM ---------- Previous update was at 01:41 PM ----------

The question is in the answer.

Code:
my %hash = map { $_, 1 } @names;
my @name_final = keys %hash;

I wish I fully understood hashes.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

2. Shell Programming and Scripting

Compare two arrays by values [BASH]

I have 2 arrays of values for example A1 ={10 15 3 21} A2 ={10 15 3 22} I need to check which one has greater values. However: A1 ={10 15 3 21} A2 ={10 15 3 21 3} - this one would be greater. A1 ={10 15 5 21} - this one greater A2 ={10 15 3 21} Basically, I want to compare patch... (6 Replies)
Discussion started by: Jutsimitsu
6 Replies

3. Shell Programming and Scripting

Modifying the values of dynamically named arrays

Hi all, In ksh, I'm trying to loop through all of my arrays, named array1, array2, array3..., and update the indices. But I'm getting errors and I'm not sure how to fix them. The errors are ./parse.sh: 6+1: not found The code is: eval \${array$c}=$(eval \${array$c}+1 ) Any help... (12 Replies)
Discussion started by: nicksantos1
12 Replies

4. Shell Programming and Scripting

Using dynamic arrays to extract the values

Hi all, We have requirement to generate load timing based on subject areas HOUSEHOLD, BANKING and TRADING. These values are stored in an array SUB_ARR SUB_ARR=("HOUSEHOLD" "BANKING" "TRADING") Based on indicator files produced while processing data for each type, we need to get the stats (using... (2 Replies)
Discussion started by: sanjaydubey2006
2 Replies

5. Shell Programming and Scripting

How do I find the sum of values from two arrays?

Hi I have redc containing the values 3, 6, 2, 8, and 1. I have work containing the values 8, 2, 11, 7, and 9. Is there a way to find the sum of redc and work? I need to compare the sum of those two arrays to something else, so is it okay to put that into my END? TY! (4 Replies)
Discussion started by: razrnaga
4 Replies

6. Shell Programming and Scripting

arrays and two values

I have requirement where I need to capture the highest values of items from a feed that runs for N hours. For example lets assume my data looks like this first feed ======== appples 10 oranges 20 pears 14 second feed ========== apples 5 oranges 30 pears 1 Last feed... (6 Replies)
Discussion started by: BeefStu
6 Replies

7. Shell Programming and Scripting

Storing values in arrays

I have the following csh script which lets the use pass the following as an argument -legend=tag1/tag2/tag3/tag4/tag5/tag6/tag7 We pass a number of tags separated by '/'. I want to save the legend tags in an array and have done as below. How can I improve on this as things are getting quite... (3 Replies)
Discussion started by: kristinu
3 Replies

8. Shell Programming and Scripting

Storing values in arrays using csh

I am reading a number of files but then I want to put the ranges xmin xmax ymin ymax as arrays for each file. Any idea how I can do this??? set j = 1 echo "Welcome $i times" while ( $j <= $i ) echo "$j" set fname = $fin-bst-misf.xy echo " "$fname ... (0 Replies)
Discussion started by: kristinu
0 Replies

9. Shell Programming and Scripting

storing values in arrays using shell

Friends, I have to execute a command and store its contents into an array using shell. this is what i have tried #!/bin/bash disk_names = ($(`iostat -xtc | egrep -v "device|nfs" | awk '{print $1}'| tr '\n' ' ' `)) But its throwing an error message as ./test-script ./test-script:... (6 Replies)
Discussion started by: achak01
6 Replies

10. UNIX for Dummies Questions & Answers

Concatenating arrays cell values in shell scripting

Hi All, I want to concatenate the array cell values and form a string.. Is it possible? for ex. I have an array word_array contains d u m b and after concatenating the string shld be 'dumb' thanks (2 Replies)
Discussion started by: mathur
2 Replies
Login or Register to Ask a Question