Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with Efficient Looping

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Efficient Looping
# 1  
Old 12-26-2011
Help with Efficient Looping
Hello guys

My requirement is to read a file with parent-child relationship
we need to iterate through each row to find its latest child.
for eg. parent child
ABC PQR
PQR DEF
DEF XYZ

Expected Output
ABC XYZ
PQR XYZ
DEF XYZ

Script Logic :
read parent from file
seach child =parent in file if match found replace child with parent
else
go to next line
I have created a bash script to achive this and its working fine
My issue is I need to process a file with more than 2 million records.
My script is taking one and half hrs for 25000 records
Can anyone suggest more effecient approach
# 2  
Old 12-26-2011
here is your code :-)
Code:
i=0
while read line
do
        if [ $i -eq 0 ]; then
                first1=$( echo $line | awk -F' ' '{print $1}' )
                i=1
        fi
        tmp1=$( echo $line | awk -F' ' '{print $1}' )
        if [[ "$second" == "$tmp1" ]] ; then
                tmp2=$( echo $line | awk -F' ' '{print $2}' )
        fi
        first=$( echo $line | awk -F' ' '{print $1}' )
        second=$( echo $line | awk -F' ' '{print $2}' )
done < tmp.sql
while read line
do
        first=$( echo $line | awk -F' ' '{print $1}' )
        echo "$first    $tmp2"
done < tmp.sql

this is acutally time consuming.. wait till some one replies with one line code... the code which i pasted above is inefficient...
# 3  
Old 12-26-2011
See if this works faster:
Code:
awk '{for(i in A)if($1==A[i])A[i]=$2; A[$1]=$2} END{for(i in A) print i, A[i]}' infile

# 4  
Old 12-26-2011
Quote:
Originally Posted by Scrutinizer
See
Code:
awk '{for(i in A)if($1==A[i])A[i]=$2; A[$1]=$2} END{for(i in A) print i, A[i]}' infile

Ok Will try this one and let you know only thing is I am reading from fixed width file I'm not sure abt how this will work with awk ...

if I will try one more option like

awk '{ while read in A ...and similar logic }'infile

Please let me know if I am on correct path
# 5  
Old 12-26-2011
If you sort the file into reverse order then the latest child is located first.
Code:
nl input |sort -r -n

This User Gave Thanks to jgt For This Post:
sanjaydubey2006
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Efficient way to grep

Hi Experts, I've been trying simple grep to search for a string in a huge number of files in a directory. grep <pattern> * this gives the search results as well as the following - grep: <filename>: Permission denied grep: <filename>: Permission denied for files which I don't have... (4 Replies)
Discussion started by: sumoka
4 Replies

2. UNIX for Dummies Questions & Answers

Is this regex efficient?

I want to match the red portion: 9784323456787-Unknown Phrase with punctuation "Some other PhrASE." Is this the best regex to match this? '978\{10\}-*' (4 Replies)
Discussion started by: glev2005
4 Replies

3. UNIX for Advanced & Expert Users

efficient repace

some of the data i receive has been typed in manually due to which there are often places where i find 8 instead of ( and the incorrect use of case what according to you is the best way to correct such data. The data has around 20,000 records. The value i want to change is in the 4th field.... (2 Replies)
Discussion started by: VGR
2 Replies

4. Shell Programming and Scripting

efficient search

Hi, i have 2 files each with 200K lines. Each line contains a number. Now, i need to get the list of numbers existing in one fine and NOT in other file. I'm doing this by reading each number from 1 file and grepping on other file. But this taking LOT of time. Is there any efficient way of doing... (14 Replies)
Discussion started by: prvnrk
14 Replies

5. Shell Programming and Scripting

Is there a way to make this more efficient

I have the following code. printf "Test Message Report" > report.txt while read line do msgid=$(printf "%n" "$line" | cut -c1-6000| sed -e 's///g' -e 's|.*ex:Msg\(.*\)ex:Msg.*|\1|') putdate=$(printf "%n" "$line" | cut -c1-6000| sed -e 's///g' -e 's|.*PutDate\(.*\)PutTime.*|\1|')... (9 Replies)
Discussion started by: gugs
9 Replies

6. Shell Programming and Scripting

help on most efficient search

Hello, We have a directory with 15 sub-directories where each sub-directory contains 1.5 to 2 lakhs of files in it. Daily, around 300-500 files will be uploaded to each sub-directory. Now, i need to get the list of files received today in most efficient way. I tried using "find with newer... (16 Replies)
Discussion started by: prvnrk
16 Replies

7. Shell Programming and Scripting

Can you suggest a more efficient way for this?

Hi I have the following at the end of a service shutdown script used in part of an active-passive failover setup: ### # Shutdown all primary Network Interfaces # associated with failover ### # get interface names based on IP's # and shut them down to simulate loss of # heartbeatd ... (1 Reply)
Discussion started by: mikie
1 Replies

8. Shell Programming and Scripting

Efficient way of Awk

Hi, Can someone let me know if the below AWK can be made much simpler / efficient ? I have 200 fields, I need to substr only the last fields. So i'm printing awk -F~ 'print {$1, $2, $3....................................$196,$197 , susbstr($198,1,3999), substr($199,1,3999)..}' Is there a... (4 Replies)
Discussion started by: braindrain
4 Replies

9. Shell Programming and Scripting

Is there a more efficient way?

I'm using korn shell to connect to oracle, retrieve certain values, put them in a list, and iterate through them. While this method works, I can't help but think there is an easier method. If you know of one, please suggest a shorter, more efficient method. ############### FUNCTIONS ... (6 Replies)
Discussion started by: SelectSplat
6 Replies

10. UNIX for Advanced & Expert Users

Efficient Dispatching

Does anyone know what's new with Efficient dispatching in the Solaris 2.8 release (vs Solaris 2.6) release? Specifically, does anyone know of a good website to get detailed information on thread dispatching using efficient dispatching in solaris 2.8? Thank you. (1 Reply)
Discussion started by: uchachra
1 Replies
Login or Register to Ask a Question