Memory exhausted in awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Memory exhausted in awk
# 1  
Old 08-14-2007
Memory exhausted in awk

Dear All,

I have executed a awk script in linux box which consists of 21 Million records.And i have two mapping files of 500 and 5200 records.To my surprise i found an error

Code:
awk: cmd. line:19: (FILENAME=/home/FILE FNR=21031272) fatal: Memory exhausted.

Is there any limitation for records in awk?

regards,
cskumar.
# 2  
Old 08-14-2007
show your code
# 3  
Old 08-14-2007
Also if possible post sample input and output Smilie

Possibly there could be a better logic for that
# 4  
Old 09-04-2007
Awk help needed.

Hi gurus,
Code:
awk -F"|" 'BEGIN {

	while(getline<"'${LOOKUPDIR}'/EAST_N_EXPANSION.lst")
		{
	gsub(" ",""); N[$1] = $2;
		}

		while(getline<"'${LOOKUPDIR}'/C_E_US.lst")
		{
	gsub(" ",""); U[$1] = $1;
		}
}
function trim(str)
{
sub("^[ ]*", "", str);
sub("[ ]*$", "", str);
return str;
}
{
    gsub ("\r","");
	if (trim($5) == "")
         {
		W=substr($1,1,10)
         }
         else
         {
		    W=N[substr($5,1,1)]substr($5,2,7)
         }

        HASH_W=W;
        while ( length (HASH_W) < 20 )
        {
	        HASH_W=("#"HASH_W)
        }

        HASH_B=substr($1,1,10);
        while ( length (HASH_B) < 20 )
        {
	        HASH_B=("#"HASH_B)
        }

         if ( W==substr($1,1,10))
        {

                LINESTATUS="PRIMARY"
        }
        else
        {

                LINESTATUS="ADDITIONAL"
        }
	if (trim($12) != "")
	 {
		SID=$12
	 }
	 if (trim($11) != "")
	 {
		SID=$11
	 }
	if ((trim($12) == "") && (trim($11) == ""))
	{
		SID="   "
	}
	if ( substr(W,4,3) < 200 )
	{
		print $0 > "'${BASEPATH}'/'${REGION}'/READY/'${FILENAME}'_'${FDATE}'_'${REGION}'_ALL_SVO_ALL_INXX"
	}
	else
	{
		if ( trim($3) == "L" && trim($13) == U[trim($13)] )
			{
			   printf ( "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%020s|%s|%s|%s|ESVO|1\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,HASH_W,substr($1,1,14),substr($1,15,3),LINESTATUS,SID ) > "ACTRECORDS"
			}

		if ( trim($3) == "L" && ( trim($13) == "NMO6A" ||  trim($13) == "NMO6P" ||  trim($13) == "NMO6Q" ||  trim($13) == "NMO6Z" ))
		{

			     printf ( "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|ESVO|1\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,HASH_W,HASH_B,substr($1,15,3),LINESTATUS ) > "SNRECORDS"
		}
		if ( trim($3) == "L" && ( $16 == "0236" ||  $17 == "0236" ) && !(trim($13) == "NPL" || trim($13) == "RTY" || trim($13) == "SEA" || trim($13) == "SCG3X" ))
		{
			printf ( "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|ESVO|1\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,HASH_W,HASH_B,substr($1,15,3),LINESTATUS ) > "LDRECORDS"
		}
		if ( trim($3) == "L" && trim($13) ~/^XRE/ && ( trim($19) != "" || trim($19) != "?" ) )
		{
		match ($19,/[1-9]/)
		{
				printf ( "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%020s|%s|%s|ESVO|1|%020s\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,W,substr($1,1,10),substr($1,15,3),LINESTATUS,substr(trim($19),RSTART) ) > "TVRECORDS"
		    }
		}
		if ( trim($3) == "L" &&  ( trim($13) == "WAL" ||  trim($13) == "WRL" ||  trim($13) == "WAB" ||  trim($13) == "WBL" ) )
		{
			printf ( "%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%020s|%s|%s|ESVO|1\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,HASH_W,substr($1,1,10)substr($1,15,3),substr($1,15,3),LINESTATUS ) > "2KRECORDS"
		}
	}
}' ${INPUTDIR}/${FILENAME}

Input(sample)
---------
701A180008    008|               |L  |I8006426261|16426261  |01C|900822|WCT1S|           |W|   |   |8L9S2|010|20060701|0288|5123| |                              |
701A180032    067|               |L  |           |FA180032  |01C|990519|XDV6X|           |)|   |   |NP3  |010|20070118|    |    | |                              |
701A180032    067|               |L  |           |FA180032  |01C|990519|XDV6X|           |)|   |   |MOM5G|010|19990519|    |    | |                              |
701A180548    857|               |L  |I8009328869|09328869  |01C|801204|WCTCS|           |W|   |   |TWASK|010|20060701|    |    | |                              |
701A182760    184|               |L  |I8004722649|34722649  |01C|000000|WCT1S|           |W|   |   |8L9SQ|010|20060701|0288|5123| |				 |
EAST_N_EXPANSION.lst(sample)
-----------------------------
A|641
B|563
C|319
D|515
E|712
F|701
R|320
S|651
T|763
U|952

C_E_US.lst(sample)
--------------------
AAO
AB3
ACB3X
ADV
AD53D
AD83B
AD83C
AD83D
AD93B
AD93C

Please help...


regards,
cskumar.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk to capture memory difference

Hi, I have a file which consists of the following information in repeating blocks. ************First iteration*************** xr_lab#show memory compare start Thu Sep 19 14:38:06.400 WIB <<<<<<<<<< START TIME Successfully stored memory snapshot in... (3 Replies)
Discussion started by: sand1234
3 Replies

2. UNIX for Beginners Questions & Answers

Memory leak with awk on MacOs

Dear all, I use awk quite a bit for data wrangling ... today I find weird behavior that I cannot wrap my head around. if I execute the following command (simplified to illustrate the behavior ... nothing to do with the real command) bash-3.2$ awk... (3 Replies)
Discussion started by: comm|getline
3 Replies

3. Shell Programming and Scripting

Math count %memory using awk

Hi expert, i have log this: Memory: 74410384 Memory: 75831176 Memory: 77961232 Memory: 77074656 Memory: 76086160 Memory: 77128592 Memory: 78045384 Memory: 76696040 Memory: 72401176 Memory: 72520016 Memory: 72137016 Memory: 73175832 Memory: 73034528 Memory: 71770736 Memory:... (4 Replies)
Discussion started by: justbow
4 Replies

4. Shell Programming and Scripting

awk eating too much memory?

Hi all, using AWK iam sorting auniq data from a file the file size is 8GB, while running that script , the over all cpu usage will be nearly 8 how to avoid this ?? any other alternate is available for awk? Thanks in Advance Anish kumar.V (13 Replies)
Discussion started by: anishkumarv
13 Replies

5. Shell Programming and Scripting

Error PHP Fatal error: Allowed memory size of 67108864 bytes exhausted(tried to allocate 401 bytes)

While running script I am getting an error like Few lines in data are not being processed. After googling it I came to know that adding such line would give some memory to it ini_set("memory_limit","64M"); my input file size is 1 GB. Is that memory limit is based on RAM we have on... (1 Reply)
Discussion started by: elamurugu
1 Replies

6. Web Development

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted

Any clues on how to get rid of this PHP error? PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /website/www/includes/functions_manpages.php on line 58 PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71... (4 Replies)
Discussion started by: Neo
4 Replies

7. Shell Programming and Scripting

AWK Memory Limit ?

Is there an input file memory limit for awk? I have a 38Mb text file that I am trying to print out certatin lines and add a string to the end of that line. When I excute the script on the 38Mb file the string I am adding is put on a new line. If I do the same with a smaller file the... (3 Replies)
Discussion started by: cold_Que
3 Replies

8. Shell Programming and Scripting

Compare lines in two files (Memory getting exhausted)

Hi, Could someone please help me with the best approach to compare lines from one file to another? Here is how I have entries - File 1 a1 a2 a3 a4 a9 a10 a15 File2 a5 a6 a15 (5 Replies)
Discussion started by: sncoupons
5 Replies

9. UNIX for Dummies Questions & Answers

'memory corruption' error when using Awk

Hello, everyone. I got the following error when I am using awk to analysis some text file: *** glibc detected *** awk: malloc(): memory corruption: 0x080c67f8 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6 /lib/tls/i686/cmov/libc.so.6... (5 Replies)
Discussion started by: kooyee
5 Replies
Login or Register to Ask a Question