Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Question: Help need to remove blank line & sed: Couldn't re-allocate memory error.

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Question: Help need to remove blank line & sed: Couldn't re-allocate memory error.
# 1  
Old 08-18-2009
Lightbulb Question: Help need to remove blank line & sed: Couldn't re-allocate memory error.
I've shell script where i used the below command to take the line which contains patterns.

Code:
sed -n "/$year 05:/,/$year 17:/p" trace.log | grep -f patterns.txt > output.log

This was working fine for long time, but now a days this script is not working with and throwing error like

sed: Couldn't re-allocate memory

when I looked for the file size its huge now a days, usually it will be less than or equal to 1.5 GB. but now a days its

Code:
$lt trace.log
 6.9G Aug 18 00:31 trace.log

trace.log is 6.9GB in size - so i conculded that the huge file size is the root cause of the issues. when i look in to the trace.log which 7 GB in size, its contains huge blank lines. i tired to remove the blank line using below commands but failed to perform and got the below error

Code:
 
$grep -av "^$" trace.log > ~/temp.tmp
$cat temp.tmp
Binary file trace.log matches
$sed '/^$/d' /logs/gopm/trace.log > ~/temp.tmp
sed: Couldn't re-allocate memory
$

Could you please suggest some other solution to over come this issue, appericaite your help
thanks
Senthil
# 2  
Old 08-18-2009
Removed the empty lines is only going to remove 1 byte per line so it is not going to help much.

Could you do some of the:
Code:
grep -f patterns.txt"

into a temporary file and then do the:
Code:
sed -n "/$year 05:/,/$year 17:/p"

?
# 3  
Old 08-19-2009
Even I tired the sed command alone which gives me memory error, i grepped for only empty line to a temp directory the file size is coming around 740 MB, so i think we can remove the blank lines whihc will eliminate the issue, but i dont know how to do this.

Thanks
Senthil.
# 4  
Old 08-19-2009
So you can try :
Code:
 grep -ve "^[ \t]*$" glpi.conf.bak | grep -f pattern.txt > temp.log

and then sed :
Code:
sed -n "/$year 05:/,/$year 17:/p" temp.log

If that doesn't work, why don't you split your huge file into smaller ones ^___^
# 5  
Old 08-19-2009
Spliting the file is a very good option, but the problem is the sed where i'm taking 5:00 to 17:00 time log wont work at all. Becuase i dont know what time string will be come in the first and last line.

Also the main concern here is the space, i wont have enough space in my /home directory where i work for the temp.log file

bash-2.05$ grep -ve "^[ \t]*$" /logs/trace.log | grep -f ~/patterns.txt >~/temp.log
bash-2.05$ cat temp.log

but this command is generating a empty file

bash-2.05$ cat temp.log
bash-2.05$
# 6  
Old 08-19-2009
You can try this assuming year is a shell variable:

Code:
awk -v Y=$year '
$0 ~ Y " 05:"{p=1}
$0 ~ Y " 17:"{print;exit}
p' trace.log | grep -f patterns.txt > output.log

Regards
# 7  
Old 08-19-2009
Franklin, could you please explain me the awk statement, cause the command is taking almost 99 % of CPU when i executed

Code:
$  ps -aux| grep 9919
sayyavu    9919 99.9  0.5 34768 22672 pts/5   R    03:22   3:07 awk -v Y= ?$0 ~ Y

also here is the sample of trace.log which will help you in finding the solution.
Code:
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser 
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.getCookieValue entry
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.getCookieValue exit
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser 
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser Current Session is Not New
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser Session Id from SSO null
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser Before sm_universalid
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser After sm_universalid
7811  CMA: 08/19/2009 03:30:54   UserValidationFilter.validateUser exit
6332  PrePr: 08/19/2009 03:30:54   ValidateUserRequestProcessor.processPreprocess entry
6332  PrePr: 08/19/2009 03:30:54   ValidateUserRequestProcessor.getGuid

The bold letter which i want to grep so i am using the sed with year then hour.

Code:
sed -n "/$year 05:/,/$year 17:/p" trace.log

Regards,
senthil.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash - sed - Remove first word from line which can begin eventually with blank

hello. How to remove first word from line. The line may or may not start with blank. NEW_PARAM1=$(magic-command " -t --protocol=TCP -P 12345-u root -h localhost ") NEW_PARAM2=$(magic-command "-t --protocol=TCP -P 12345 -u root -h localhost ") I want NEW_PARAM1 equal to NEW_PARAM2 equal ... (2 Replies)
Discussion started by: jcdole
2 Replies

2. Shell Programming and Scripting

'Couldn't read file' error in bash script with expect, sed and awk!

Ok, so I have a bash script with an embedded expect statement. Inside of the expect statement, i'm trying to pull all of the non-comment lines from the /etc/oratab file one at a time. Here's my command: cat /etc/oratab |sed /^s*#/d\ | awk 'NR==1'|awk -F: '{print \"$1\"}'|. oraenv Now,... (0 Replies)
Discussion started by: alexdglover
0 Replies

3. Shell Programming and Scripting

How to remove blank line from a text file?

Hi All, I am creating a text file using perl. The first record I am writing as "$line" and all the other as "\n$line". At the end the file is having N number of lines. I am using this file for MLOAD (Teradata), which is reading N+1 lines in the file and failing.I am not able to find new line... (2 Replies)
Discussion started by: unankix
2 Replies

4. Shell Programming and Scripting

Print lines between a regExp & a blank line

Hi, I have a file, say files_list, as below (o/p of ls -R cmd) $ cat files_list /remote/dir/path/to/file: sub-dir1 sub-dir2 sub-dir3 ... /remote/dir/path/to/file/sub-dir1: remote_file1.csv.tgz <blank line 1> /remote/dir/path/to/file/sub-dir2: remote_file2.csv.tgz <blank... (3 Replies)
Discussion started by: dips_ag
3 Replies

5. Shell Programming and Scripting

"Cannot allocate memory" error when run from script

hi in my application there is a line with open(/dev/mydevice,0); it work good when it run manually, but when try to run it within script /////////////////////////////////////////// #!/bin/sh ./device_test 0 0 0 1 1 1 0 0 1 ///////////////////////////////////////// I receive 'Failed... (3 Replies)
Discussion started by: majeed
3 Replies

6. Shell Programming and Scripting

Error PHP Fatal error: Allowed memory size of 67108864 bytes exhausted(tried to allocate 401 bytes)

While running script I am getting an error like Few lines in data are not being processed. After googling it I came to know that adding such line would give some memory to it ini_set("memory_limit","64M"); my input file size is 1 GB. Is that memory limit is based on RAM we have on... (1 Reply)
Discussion started by: elamurugu
1 Replies

7. Shell Programming and Scripting

Perl & Sed command -- Out of Memory Error

Experts, We used to receive our source files with '~^' as row delimiter. This file contains 2500K records and two of the columns having value in HTML formats within the file. While running the below commands against the file, we are encountering out of memory, could you please help to... (3 Replies)
Discussion started by: srivijay81
3 Replies

8. Shell Programming and Scripting

Remove last blank line of file

I have a number of files (arranged in directories) which have last line blank, I am trying to synchronize my code with other env and due to this blank lines, all files error out as different although only difference is that of balnk line at end of file. Is there a way I can recursively... (2 Replies)
Discussion started by: ruchimca
2 Replies

9. UNIX for Dummies Questions & Answers

Newbie help with New Line & Blank Line

I have this in my script: usercount=`ldapsearch -L -b"ou=people,ou=$ou,dc=paisleyhosting,dc=com" -h"$server" -p"1391" cn=* nothing | grep -c dn` admincount=`ldapsearch -b"ou=groups,ou=$ou,dc=paisleyhosting,dc=com" -h"$server" -p"1391" cn=USERADMIN* uniquemember | grep... (5 Replies)
Discussion started by: kthatch
5 Replies

10. Shell Programming and Scripting

Remove blank line - SED

HI, I have this list of apps like so: DivX Products.app DivX Support.app Uninstall DivX for Mac.app Build Applet.app SpringBoard.app Interface.app MobileAddressBook.app MobileSafari.app MobileSlideShow.app Preferences.app Install Flash Player 8 OSX.app Yap.app check_afp.app ... (10 Replies)
Discussion started by: pcwiz
10 Replies
Login or Register to Ask a Question