Complex awk problem Post: 302770504

Sponsored Content

Top Forums Shell Programming and Scripting Complex awk problem Post 302770504 by dietmar13 on Saturday 16th of February 2013 03:47:14 PM

02-16-2013

Registered User

thank you rudiC

I do not understand you approach completely, but the while loop is run every new value from file2 (?), and as I have hundreds of millions of values, this approach has to be slower than the array approach, where the array is load once and only used every value...

the script looks now like this and works perfect (the additional p-value translation increases the time only by 1 minute from 15 to 16 minutes):

Code:

dir='/home/ws/R_workspace/OVCAD'
fn='miR.MIC'
pv='pvalues_220.txt'
cd $dir
fname=${fn%.*}
echo $fname
echo -e "N1\tN2\tMIC\tMAS\tMEV\tMCN\tMICR2\tpearson\tnon-linearity\tp-value" > ${fname}.mine
 
gawk 'BEGIN { FS = "\t"; OFS="\t" } ; NR==FNR{p[$1]=$2;next} { NR==1;{ for (i = 1; i <= NF; i++) name[i]=$i } }; \
 { split(FILENAME,fname,"."); fn=fname[1]; getline < (fn".MAS"); getline < (fn".MEV"); getline < (fn".MCN"); getline < (fn".MICR2"); getline < (fn".cor"); getline < (fn".nl") } \
 { for(k=1; k <= NF; k++) { getline; split($0, MIC, "\t") ; \
 getline mas < (fn".MAS"); split(mas, MAS, "\t") ; \
 getline mev < (fn".MEV"); split(mev, MEV, "\t") ; \
 getline mcn < (fn".MCN"); split(mcn, MCN, "\t") ; \
 getline micr2 < (fn".MICR2"); split(micr2, MICR2, "\t") ; \
 getline cor < (fn".cor"); split(cor, COR, "\t") ; \
 getline nl < (fn".nl"); split(nl, NL, "\t") ; \
 for(j=k+1; j <= NF; j++)  \
 print name[k],name[j],MIC[j],MAS[j],MEV[j],MCN[j],MICR2[j],COR[j],NL[j],p[int(MIC[j]*10000)] } }' $pv ${fname}.MIC >> ${fname}.mine

dietmar13

View Public Profile for dietmar13

Find all posts by dietmar13

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Complex Pipeline/Redirection/Regular Expression problem

This is a very hard unix command which I could not perform..... :( ......here it is: The file ~unx122/public/data.txt contains over 18,000 lines of text. Here are the first ten lines of the file: pagination 20657 2740 28416 31090 18145 shiner 1695 2507 9964 14512 13122 cool 29210...

2. Shell Programming and Scripting

Complex problem about nested for loops

Hey, I'm writing this bash script that will test print me many copies of the same program but with different combos of 4 variables being between 1 and 100. Here's the code: #! /bin/bash x=0 for ((a=1; a < 101; a++)) do for ((b=1; b < 101; b++)) do for ((c=1; c < 101; c++)) do for...

3. Shell Programming and Scripting

Complex file count problem

Hi all! I have a question regarding possibilities to do line counts. SEARCH_VAR=TEX rsh $REM_HOST -l $REM_USER "cd $REM_DIR; ls *$SEARCH_VAR* 2> /dev/null" | sort -n | awk 'BEGIN { FS = "-" } ; { print $1"\t"$0 }' Will produce an output on the screen like this: 483 483-SOME-TEXT-1...

4. IP Networking

Need to solve complex network problem

I have a Red Hat linux server X on a x.x.0.0 network. This machine also has to communicate with another server Y on a network called y.y.0.0 Server X has two network interfaces. eth0 is configured on the x.x.0.0 network and has a default gateway on the x.x.0.0 network. In order to...

5. Shell Programming and Scripting

Complex use with awk

Hi , I have file named docs.txt The content of the file look like this: DOC disk location Size ======= ===== ============= ========= TXT A /dev/dm-1 10 TXT B /dev/dm-2 10 BIN C ...

6. Shell Programming and Scripting

complex Awk Question

Hi, I have a file look likes this : --->start hir Trace file: pudwh_ora_9998.trc Sort options: fchela exeela ***************************************************************count = number of times OCI procedure was executed cpu = cpu time in seconds executing elapsed = elapsed...

7. Shell Programming and Scripting

Get values from 2 files - Complex "for loop and if" awk problem

Hi everyone, I've been thinking and trying/changing all day long the below code, maybe some awk expert could help me to fix the for loop I've thought, I think I'm very close to the correct output. file1 is: <boxes content="Grapes and Apples"> <box No.="Box MT. 53"> <quantity...

8. Shell Programming and Scripting

awk script (complex)

picked this up from another thread. echo 1st_file.csv; nawk -F, 'NR==FNR{a++;next} a{b++} END{for(i in b){if(b-1&&a!=b){print i";\t\t"b}else{print "NEW:"i";\t\t"b} } }' OFS=, 1st_file.csv *.csv | sort -r i need to use the above but with a slight modification.. 1.compare against 3 month...

9. Shell Programming and Scripting

Help with Complex Awk.

Hi, I have a file. In this file when ever the word "ABC" occurs at position from 25 and 34 I would like to replace the value at postion 100 to 5 for the first 1000 rows only. I have written the following Awk command. nawk 'substr($0,25,9)=="ABC" {print $0}' filename The above command...

10. Shell Programming and Scripting

Complex text parsing with speed/performance problem (awk solution?)

I have 1.6 GB (and growing) of files with needed data between the 11th and 34th line (inclusive) of the second column of comma delimited files. There is also a lot of stray white space in the file that needs to be trimmed. They have DOS-like end of lines. I need to transpose the 11th through...

LEARN ABOUT DEBIAN

yaz-url

YAZ-URL(1)							     Commands								YAZ-URL(1)

NAME

       yaz-url - YAZ URL fetch utility

SYNOPSIS

       yaz-url [-H name:value] [-m method] [-O fname] [-p fname] [-u user/password] [-x proxy] [url...]

DESCRIPTION

       yaz-url is utility to get web content. It is very limited in functionality compared to programs such as curl, wget.

       The options must be precede the URL given on the command line to take effect.

       Fetched HTTP content is written to stdout, unless option -O is given.

OPTIONS

       -H name:value
	   Specifies HTTP header content with name and value. This option can be given multiple times (for different names, of course).

       -m method
	   Specifies the HTTP method to be used for the next URL. Default is method "GET". However, option -p sets it to "POST".

       -O fname
	   Sets output filename for HTTP content.

       -p fname
	   Sets a file to be POSTed in the folloing URL.

       -u user/password
	   Specifies a user and a password to be uesd in HTTP basic authentication in the following URL fetch. The user and password must be
	   separated by a slash (this it is not possible to specify a user with a slash in it).

       -x proxy
	   Specifies a proxy to be used for URL fetch.

SEE ALSO

       yaz(7)

YAZ 4.2.30							    04/16/2012								YAZ-URL(1)