Sponsored Content
Top Forums Shell Programming and Scripting Efficient population of array from text file Post 302663249 by carlr on Wednesday 27th of June 2012 07:37:12 PM
Old 06-27-2012
Thanks the above works very nicley. But i now have another issue.

I need to efficiently scan through a text file of 70,000 lines and populate an array. This time it is more complex. Each line has 5 numbers, and is comma separated; EL_Num, Node_A, Node_B, Node_C, Node_D. The values for these are all integers.

I want to populate the array with the index equal to the Node number, and the string equal to all the El_num which contain that node.

For example given the below lines out of the file:
Code:
EL_Num, Node_A, Node_B, Node_C, Node_D
      5414,      5249,      5018,      5217,      5113
      5415,      5018,       5035,      5300,      5201
      5416,      5345,      5013,      5018,        5245

It can be seen that all el_num contain node 5018.

Therefore ARRAY[5108] should equal 5014, 5015, 5016.

Like wise ARRAY[5249] should equal 5014


I have a method below but it is very slow, how can this be done efficiently?
Code:
Last_line_3D=$(awk -F, 'END {print NR}' 3D_ELSET.tmp)    
Num_Fields=$(awk -F, 'NR==1 {print NF}' 3D_ELSET.tmp)


for ((i=1; i<=$Last_line_3D; i++)) do            
for ((j=2; j<=$Num_Fields; j++)) do

EL_num=$(awk -F, -v variable="$i" 'NR==variable {print $1}' 3D_ELSETTEST.tmp)
ND_num=$(awk -F, -v vari="$i" -v field="$j" 'NR==vari {print $field}' 3D_ELSETTEST.tmp)


Orig_string=${ARRAY[ND_num]}                # Set variable as original string in array

unset ARRAY[ND_num]                            # Remove current string in the array

New_string="$Orig_string $EL_num"            # Set variable as original string in array plus the new element number

ARRAY[ND_num]=$New_string                    # Place the new string into the array

done
done


Last edited by Franklin52; 06-28-2012 at 04:52 AM.. Reason: Please use code tags for data and code samples
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need more efficient log file grep

I'm writing a script that at one point needs to check the contents of another script's log file to determine how to proceed. An example record from the log file is: "mcref04152006","060417","ANTH0415","282","272","476,983.37","465,268.44","loaded" I want my script to return this record if: ... (3 Replies)
Discussion started by: Glenn Arndt
3 Replies

2. Shell Programming and Scripting

input text from file into 2d array

Hi all I have a little brainscratcher here. I want to draw a pie chart from data in a text file. The drawing of the graph works fine, if I insert the data manually into a 2d array. Now I want to pull the data from a text file (which was created using a uniq -c command) see sample below.... (2 Replies)
Discussion started by: pietie
2 Replies

3. Shell Programming and Scripting

Pipe text from a file into an array

Hi Guys I have a question about filling up an array I have a file called USER_FILE.txt it contains the following: Real Name:Thomas A Username:THOMAS_A Real Name:Thomas B Username:THOMAS_B Real Name:Thomas C Username:THOMAS_C Real Name:Thomas D Username:THOMAS_D Real Name:Thomas E... (8 Replies)
Discussion started by: grahambo2005
8 Replies

4. Shell Programming and Scripting

File transformation - what is most efficient method

I've done quite a bit of searching on this but cannot seem to find exactly what I'm looking for. Say I have a | delimited input file with 6 columns and I need to change the value of a few columns and create an output file. With my limited knowledge I can do this with many lines of code but want... (5 Replies)
Discussion started by: 1superdork
5 Replies

5. Homework & Coursework Questions

Efficient Text File Writing

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write a template main.c file via shell script to make it easier for yourself later. The issue here isn't writing... (2 Replies)
Discussion started by: george3isme
2 Replies

6. Shell Programming and Scripting

Reading columns from a text file and to make an array for each column

Hi, I am not so familiar with bash scripting and would appreciate your help here. I have a text file 'input.txt' like this: 2 3 4 5 6 7 8 9 10 I want to store each column in an array like this a ={2 5 8}, b={3 6 9}, c={4 7 10} so that i can access any element, e.g b=6 for the later use. (1 Reply)
Discussion started by: Asif Siddique
1 Replies

7. Shell Programming and Scripting

Array & text file

Hi all, i have a text file such as: 10 17:54:47,213 10 17:54:47,214 10 17:54:49,338 10 17:54:49,399 10 17:54:50,402 10 17:54:50,403 11 17:54:47,213 11 17:54:47,213 11 17:54:49,362 11 17:54:49,422 11 17:54:50,429 11 17:54:50,429 11 17:54:50,429 11 17:54:50,429 11 17:54:51,510 12... (10 Replies)
Discussion started by: sbamap
10 Replies

8. Shell Programming and Scripting

Most efficient method to extract values from text files

I have a list of files defined in a single file , one on each line.(No.of files may wary each time) eg. content of ETL_LOOKUP.dat /data/project/randomname /data/project/ramname /data/project/raname /data/project/radomname /data/project/raame /data/project/andomname size of these... (5 Replies)
Discussion started by: h0x0r21
5 Replies

9. Shell Programming and Scripting

Efficient way to search array in text file by awk

I have one array SPLNO with approx 10k numbers.Now i want to search the subscriber number from MDN.TXT file (containing approx 1.5 lac record)from the array.if subscriber number found in array it will perform below operation.my issue is that it's taking more time because for one number it's search... (6 Replies)
Discussion started by: siramitsharma
6 Replies

10. Shell Programming and Scripting

Portable and efficient way to add text after pattern

Shell: sh/bash OS: Linux (all unix flavors) Suppose i have a variable with this content: ArgZ=' import os import sys MySpecialpath = os.path.abspath(sys.argv) # ' ArgZB='#REGEN #REGEN #REGEN ' I want to add this text to a file/script, only under the following conditions: 1. ... (1 Reply)
Discussion started by: SkySmart
1 Replies
Tree::Simple::Visitor::PathToRoot(3pm)			User Contributed Perl Documentation		    Tree::Simple::Visitor::PathToRoot(3pm)

NAME
Tree::Simple::Visitor::PathToRoot - A Visitor for finding the path back a Tree::Simple object's root SYNOPSIS
use Tree::Simple::Visitor::PathToRoot; # create an instance of our visitor my $visitor = Tree::Simple::Visitor::PathToRoot->new(); # pass the visitor to a Tree::Simple object $tree->accept($visitor); # now get the accumulated path as a string # with the '/' character as the delimiter print $visitor->getPathAsString("/"); # include the tree's trunk in your # output as well $visitor->includeTrunk(); # for more complex node objects, you can specify # a node filter which will be used to extract the # information desired from each node $visitor->setNodeFilter(sub { my ($t) = @_; return $t->getNodeValue()->description(); }); # you can also get the path back as an array my @path = $visitor->getPath(); DESCRIPTION
Given a Tree::Simple object, this Visitor will find the path back to the tree's root node. METHODS
new There are no arguments to the constructor the object will be in its default state. You can use the "includeTrunk" and "setNodeFilter" methods to customize its behavior. includeTrunk ($boolean) Based upon the value of $boolean, this will tell the visitor to collect the trunk of the tree as well. setNodeFilter ($filter_function) This method accepts a CODE reference as its $filter_function argument and throws an exception if it is not a code reference. This code reference is used to filter the tree nodes as they are collected. This can be used to customize output, or to gather specific information from a more complex tree node. The filter function should accept a single argument, which is the current Tree::Simple object. visit ($tree) This is the method that is used by Tree::Simple's "accept" method. It can also be used on its own, it requires the $tree argument to be a Tree::Simple object (or derived from a Tree::Simple object), and will throw and exception otherwise. getPath This will return the collected path as an array, or in scalar context, as an array reference. getPathAsString ($delimiter) This will return the collected path as a string with the path elements joined by a $delimiter. If no $delimiter is specified, the default (', ') will be used. BUGS
None that I am aware of. Of course, if you find a bug, let me know, and I will be sure to fix it. CODE COVERAGE
See the CODE COVERAGE section in Tree::Simple::VisitorFactory for more inforamtion. SEE ALSO
These Visitor classes are all subclasses of Tree::Simple::Visitor, which can be found in the Tree::Simple module, you should refer to that module for more information. AUTHOR
stevan little, <stevan@iinteractive.com> COPYRIGHT AND LICENSE
Copyright 2004, 2005 by Infinity Interactive, Inc. <http://www.iinteractive.com> This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.10.1 2005-07-14 Tree::Simple::Visitor::PathToRoot(3pm)
All times are GMT -4. The time now is 08:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy