Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

What is solution for this error?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting What is solution for this error?
# 8  
Old 04-26-2013
Quote:
Originally Posted by Corona688
mawk's main claim to fame is that it's very fast. I rarely see it used but sometimes people bring it in to speed up big problems.
It's used by quite a lot of people, even if they don't realize it. mawk is the default AWK on debian and its derivatives (ubuntu, mint, etc).

Quote:
Originally Posted by Corona688
In the end, though, my brain divides awk into two realms: "GNU Awk", and "Where'd mktime() go". Smilie
mawk changelog:
Code:
20121129
...
	+ add systime and mktime functions

I am not evangelizing; I am simply mentioning some facts that may be of interest.

Personally, I have a strong preference for *BSD userlands, where the default implementation is invariably the "One True AWK" aka nawk.

Regards,
Alister
This User Gave Thanks to alister For This Post:
Corona688
# 9  
Old 04-28-2013
Getting one more new error

file size is about 890MB

Code:
gawk: cmd. line:23: (FILENAME=Check.dat FNR=1633435) fatal: more_nodes: nextfree: can't allocate 4000 bytes of memory (Cannot allocate memory)

# 10  
Old 04-28-2013
Any chance your code is adding a huge number of elements to an array? That might possibly explain why running out of memory.
# 11  
Old 04-28-2013
I am using 3 dimensional array in which 11 functions are there...total 3 files I am using of size 20kb, 890MB,50kb

after the execution of each function one field will be increase, and finally I will write to file

here is my code...

pls corona and hanson give me some solution ..otherwise I have to split file and then I have to use, splitting file is time consuming

I have planned to store msg in separate array and then print at the end, if works then I will be very happy

Code:
awk 'FNR==1{i++}

                                   {LNC[i]=NF}
                                   {for(k=1;k<=NF;k++)A[i,FNR,k]=$k}
                                   {LC[i]=FNR}

function Step1(){   
                            w=1
                            for(i=1;i<=LC[2];i++)
                
                            if(A[2,i,2]<A[2,i+1,2])
                              {
                               PT[i,1]=w
                              }
             else
                             {
                              PT[i,1]=w++
                             }
                 }

function Dup_C(){

                               for(i=1;i<=LC[2];i++)
            
                                        if(PC[2,i,3]==PT[2,i+1,3] && PT[2,i,2]==PT[2,i+1,2])
                                                  PT[i,2]="DUPLICATE"
                                        else
                                                   PT[i,2]="00"

                             }

# Write to file finally here
function out(){
                                    for(i=1;i<=LC[2];i++)
                                     {                
                                          for(m=1;m<=2;m++)
                                                   printf "%s%s",PT[i,m],OFS >>"QC.dat"
                                                   
                                     }
                                     {    
                                              for(m=1;m<=LNC[2];m++)        
                                              printf "%s%s",A[2,i,m],OFS >>"QC.dat"
                                      }
                                              printf "\n" >>"QC.dat"
                   }

                      END{
                                          Step1()
       
                                   Dup_C()

                                     out()
   
                                        # Some more functions
                        }' OFS=\\t file1 file2 file3 ..

Last edited by Akshay Hegde; 04-29-2013 at 02:44 AM..
# 12  
Old 04-28-2013
You obviously are carrying out complex operations. I'm not going to be able to simplify that code. Someone else might be able to. But I would bet it's those arrays causing the problem, running out of memory. I would suggest mawk might be innocent.

Just as a diagnostic, maybe use length function on the arrays at some point, to see how big they are. You could set it to print before it crashes, since the error message tells you where it runs out. If you find out the sizes, it would be interesting if you posted.

There are also ways of monitoring program memory usage, external to awk. That would be worth doing.

Quote:
splitting file is time consuming
How long does it take to run? From my experience, doing these kind of industrial-strength operations in one pass is often problematic. I typically break it into a few steps. It could well take longer, as you suggest, and that's a negative. But I think it's usually a better approach, and the time difference does not have to be that large. Also, it's easier to verify, since each step can be validated, instead of trying to do everything in one pass.
# 13  
Old 04-28-2013
Thanks hanson44...

What you said is correct it's because of array, and validation will be easier to verify, if split into parts, but my approach is whether its possible to overcome from this problem ? one way is in each function instead of writing msg to array being used to new array, but still I think I will get error.. is there any provision to set memory for running script ?
# 14  
Old 04-28-2013
Quote:
any provision to set memory for running script
My understanding is there is not, except that you could install more RAM or increase swap size. Even if huge amount of memory needed, programs know how to swap memory to disk. I think it would help to monitor memory usage while the process is running, and see what is going on. free and top come to mind, there are many programs to monitor memory usage.

Perhaps someone else can suggest how to modify the script. It's understandable you would want to modify the existing script instead of rewriting the sequence of operations.
Login or Register to Ask a Question

Previous Thread | Next Thread

2 More Discussions You Might Find Interesting

1. Ubuntu

What is solution for this error "tar: Exiting with failure status due to previous errors"?

Does anyone know what is solution for this error ?tar: Exiting with failure status due to previous errors from last 3 days I am trying to take backup of home/user directory getting again and again same error please anyone give me solution (8 Replies)
Discussion started by: Akshay Hegde
8 Replies

2. Shell Programming and Scripting

need the solution

Write a program to print all prime numbers from 1 to 300. (3 Replies)
Discussion started by: paniruddha
3 Replies
Login or Register to Ask a Question