Question: Help need to remove blank line & sed: Couldn't re-allocate memory error.

senthil.ak · #1 (**permalink**) 08-18-2009

I've shell script where i used the below command to take the line which contains patterns.

Code:

sed -n "/$year 05:/,/$year 17:/p" trace.log | grep -f patterns.txt > output.log

This was working fine for long time, but now a days this script is not working with and throwing error like

sed: Couldn't re-allocate memory

when I looked for the file size its huge now a days, usually it will be less than or equal to 1.5 GB. but now a days its

Code:

$lt trace.log
 6.9G Aug 18 00:31 trace.log

trace.log is 6.9GB in size - so i conculded that the huge file size is the root cause of the issues. when i look in to the trace.log which 7 GB in size, its contains huge blank lines. i tired to remove the blank line using below commands but failed to perform and got the below error

Code:

 
$grep -av "^$" trace.log > ~/temp.tmp
$cat temp.tmp
Binary file trace.log matches
$sed '/^$/d' /logs/gopm/trace.log > ~/temp.tmp
sed: Couldn't re-allocate memory
$

Could you please suggest some other solution to over come this issue, appericaite your help
thanks
Senthil

TonyFullerMalv · #2 (**permalink**) 08-18-2009

Removed the empty lines is only going to remove 1 byte per line so it is not going to help much.

Could you do some of the:

Code:

grep -f patterns.txt"

into a temporary file and then do the:

Code:

sed -n "/$year 05:/,/$year 17:/p"

?

senthil.ak · #3 (**permalink**) 08-19-2009

Even I tired the sed command alone which gives me memory error, i grepped for only empty line to a temp directory the file size is coming around 740 MB, so i think we can remove the blank lines whihc will eliminate the issue, but i dont know how to do this.

Thanks
Senthil.

thanhdat · #4 (**permalink**) 08-19-2009

So you can try :

Code:

 grep -ve "^[ \t]*$" glpi.conf.bak | grep -f pattern.txt > temp.log

and then sed :

Code:

sed -n "/$year 05:/,/$year 17:/p" temp.log

If that doesn't work, why don't you split your huge file into smaller ones ^___^

senthil.ak · #5 (**permalink**) 08-19-2009

Spliting the file is a very good option, but the problem is the sed where i'm taking 5:00 to 17:00 time log wont work at all. Becuase i dont know what time string will be come in the first and last line.

Also the main concern here is the space, i wont have enough space in my /home directory where i work for the temp.log file

bash-2.05$ grep -ve "^[ \t]*$" /logs/trace.log | grep -f ~/patterns.txt >~/temp.log
bash-2.05$ cat temp.log

but this command is generating a empty file

bash-2.05$ cat temp.log
bash-2.05$

Franklin52 · #6 (**permalink**) 08-19-2009

You can try this assuming year is a shell variable:

Code:

awk -v Y=$year '
$0 ~ Y " 05:"{p=1}
$0 ~ Y " 17:"{print;exit}
p' trace.log | grep -f patterns.txt > output.log

Regards

senthil.ak · #7 (**permalink**) 08-19-2009

Franklin, could you please explain me the awk statement, cause the command is taking almost 99 % of CPU when i executed

Code:

$  ps -aux| grep 9919
sayyavu    9919 99.9  0.5 34768 22672 pts/5   R    03:22   3:07 awk -v Y= ?$0 ~ Y

also here is the sample of trace.log which will help you in finding the solution.

Code:

7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser 
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.getCookieValue entry
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.getCookieValue exit
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser 
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser Current Session is Not New
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser Session Id from SSO null
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser Before sm_universalid
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser After sm_universalid
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser exit
6332  PrePr: 08/19/2009 03:30:54   ValidateUserRequestProcessor.processPreprocess entry
6332  PrePr: 08/19/2009 03:30:54   ValidateUserRequestProcessor.getGuid

The bold letter which i want to grep so i am using the sed with year then hour.

Code:

sed -n "/$year 05:/,/$year 17:/p" trace.log

Regards,
senthil.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Remove last blank line of file	ruchimca	Shell Programming and Scripting	2	04-08-2009 10:30 PM
Remove blank line - SED	pcwiz	Shell Programming and Scripting	10	11-11-2008 11:19 PM
How to allocate memory	ravi3553	UNIX for Advanced & Expert Users	5	08-19-2008 10:09 AM
how allocate virtual memory	munnu	High Level Programming	2	04-20-2007 04:07 PM
Unix Help - allocate more memory to /tmp	cmackin	UNIX for Dummies Questions & Answers	1	06-01-2006 07:25 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: