Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Efficient population of array from text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Efficient population of array from text file
# 1  
Old 06-27-2012
Efficient population of array from text file
Hi,

I am trying to populate an array with data from a text file. I have a working method using awk but it is too slow and inefficent. See below.

The text file has 70,000 lines. As awk is a line editor it reads each line of the file until it gets to the required line and then processes it. Then for the next loop it starts at the first line and then reads each line of the file until it gets to the required line and then processes it. This is very inefficent when you get towards the bottom of a large file.

A want to efficently write the 70,000 lines of a text file into an array where the each line of the file contains 5 numbers and is comma seperated. The array index should be equal to the first number. The value of the array for each index can be equal to either the whole line or the last four numbers of the line.

The method does not have to use awk, so long as it is a bash script.
I can't figure out how to use awk to read the data to the array as it reads through the file on the first read through.
Code:
for ((i=1; i<=70000; i++)) do                        
  line_num=$i
  ARRAY_3D_ELSET[$i]=$(awk -F, -v var="$line_num" 'NR==var {print $0}' 3D_ELSET.tmp)     
done

Last edited by Franklin52; 06-28-2012 at 04:51 AM.. Reason: Please use code tags for data and code samples
# 2  
Old 06-27-2012
How about this

Code:
eval ARRAY_3D_ELSET=( $(awk -F, '{print "["$1"]=\""$0"\" "}' 3D_ELSET.tmp) )

# 3  
Old 06-27-2012
Thanks the above works very nicley. But i now have another issue.

I need to efficiently scan through a text file of 70,000 lines and populate an array. This time it is more complex. Each line has 5 numbers, and is comma separated; EL_Num, Node_A, Node_B, Node_C, Node_D. The values for these are all integers.

I want to populate the array with the index equal to the Node number, and the string equal to all the El_num which contain that node.

For example given the below lines out of the file:
Code:
EL_Num, Node_A, Node_B, Node_C, Node_D
      5414,      5249,      5018,      5217,      5113
      5415,      5018,       5035,      5300,      5201
      5416,      5345,      5013,      5018,        5245

It can be seen that all el_num contain node 5018.

Therefore ARRAY[5108] should equal 5014, 5015, 5016.

Like wise ARRAY[5249] should equal 5014


I have a method below but it is very slow, how can this be done efficiently?
Code:
Last_line_3D=$(awk -F, 'END {print NR}' 3D_ELSET.tmp)    
Num_Fields=$(awk -F, 'NR==1 {print NF}' 3D_ELSET.tmp)


for ((i=1; i<=$Last_line_3D; i++)) do            
for ((j=2; j<=$Num_Fields; j++)) do

EL_num=$(awk -F, -v variable="$i" 'NR==variable {print $1}' 3D_ELSETTEST.tmp)
ND_num=$(awk -F, -v vari="$i" -v field="$j" 'NR==vari {print $field}' 3D_ELSETTEST.tmp)


Orig_string=${ARRAY[ND_num]}                # Set variable as original string in array

unset ARRAY[ND_num]                            # Remove current string in the array

New_string="$Orig_string $EL_num"            # Set variable as original string in array plus the new element number

ARRAY[ND_num]=$New_string                    # Place the new string into the array

done
done

Last edited by Franklin52; 06-28-2012 at 04:52 AM.. Reason: Please use code tags for data and code samples
# 4  
Old 06-27-2012
Try:

Code:
eval ARRAY=( $(awk -F, '{for(i=2;i<=NF;i++)v[$i]=(($i in v)?v[$i]" ":"")$1}
             END { for(i in v) print "["i"]=\""v[i]"\""}' 3D_ELSET.tmp) )

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Portable and efficient way to add text after pattern

Shell: sh/bash OS: Linux (all unix flavors) Suppose i have a variable with this content: ArgZ=' import os import sys MySpecialpath = os.path.abspath(sys.argv) # ' ArgZB='#REGEN #REGEN #REGEN ' I want to add this text to a file/script, only under the following conditions: 1. ... (1 Reply)
Discussion started by: SkySmart
1 Replies

2. Shell Programming and Scripting

Efficient way to search array in text file by awk

I have one array SPLNO with approx 10k numbers.Now i want to search the subscriber number from MDN.TXT file (containing approx 1.5 lac record)from the array.if subscriber number found in array it will perform below operation.my issue is that it's taking more time because for one number it's search... (6 Replies)
Discussion started by: siramitsharma
6 Replies

3. Shell Programming and Scripting

Most efficient method to extract values from text files

I have a list of files defined in a single file , one on each line.(No.of files may wary each time) eg. content of ETL_LOOKUP.dat /data/project/randomname /data/project/ramname /data/project/raname /data/project/radomname /data/project/raame /data/project/andomname size of these... (5 Replies)
Discussion started by: h0x0r21
5 Replies

4. Shell Programming and Scripting

Array & text file

Hi all, i have a text file such as: 10 17:54:47,213 10 17:54:47,214 10 17:54:49,338 10 17:54:49,399 10 17:54:50,402 10 17:54:50,403 11 17:54:47,213 11 17:54:47,213 11 17:54:49,362 11 17:54:49,422 11 17:54:50,429 11 17:54:50,429 11 17:54:50,429 11 17:54:50,429 11 17:54:51,510 12... (10 Replies)
Discussion started by: sbamap
10 Replies

5. Shell Programming and Scripting

Reading columns from a text file and to make an array for each column

Hi, I am not so familiar with bash scripting and would appreciate your help here. I have a text file 'input.txt' like this: 2 3 4 5 6 7 8 9 10 I want to store each column in an array like this a ={2 5 8}, b={3 6 9}, c={4 7 10} so that i can access any element, e.g b=6 for the later use. (1 Reply)
Discussion started by: Asif Siddique
1 Replies

6. Homework & Coursework Questions

Efficient Text File Writing

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write a template main.c file via shell script to make it easier for yourself later. The issue here isn't writing... (2 Replies)
Discussion started by: george3isme
2 Replies

7. Shell Programming and Scripting

File transformation - what is most efficient method

I've done quite a bit of searching on this but cannot seem to find exactly what I'm looking for. Say I have a | delimited input file with 6 columns and I need to change the value of a few columns and create an output file. With my limited knowledge I can do this with many lines of code but want... (5 Replies)
Discussion started by: 1superdork
5 Replies

8. Shell Programming and Scripting

Pipe text from a file into an array

Hi Guys I have a question about filling up an array I have a file called USER_FILE.txt it contains the following: Real Name:Thomas A Username:THOMAS_A Real Name:Thomas B Username:THOMAS_B Real Name:Thomas C Username:THOMAS_C Real Name:Thomas D Username:THOMAS_D Real Name:Thomas E... (8 Replies)
Discussion started by: grahambo2005
8 Replies

9. Shell Programming and Scripting

input text from file into 2d array

Hi all I have a little brainscratcher here. I want to draw a pie chart from data in a text file. The drawing of the graph works fine, if I insert the data manually into a 2d array. Now I want to pull the data from a text file (which was created using a uniq -c command) see sample below.... (2 Replies)
Discussion started by: pietie
2 Replies

10. Shell Programming and Scripting

Need more efficient log file grep

I'm writing a script that at one point needs to check the contents of another script's log file to determine how to proceed. An example record from the log file is: "mcref04152006","060417","ANTH0415","282","272","476,983.37","465,268.44","loaded" I want my script to return this record if: ... (3 Replies)
Discussion started by: Glenn Arndt
3 Replies
Login or Register to Ask a Question