Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Question: Help need to remove blank line & sed: Couldn't re-allocate memory error. Post 302344916 by senthil.ak on Tuesday 18th of August 2009 03:43:02 AM
Old 08-18-2009
Lightbulb Question: Help need to remove blank line & sed: Couldn't re-allocate memory error.

I've shell script where i used the below command to take the line which contains patterns.

Code:
sed -n "/$year 05:/,/$year 17:/p" trace.log | grep -f patterns.txt > output.log

This was working fine for long time, but now a days this script is not working with and throwing error like

sed: Couldn't re-allocate memory

when I looked for the file size its huge now a days, usually it will be less than or equal to 1.5 GB. but now a days its

Code:
$lt trace.log
 6.9G Aug 18 00:31 trace.log

trace.log is 6.9GB in size - so i conculded that the huge file size is the root cause of the issues. when i look in to the trace.log which 7 GB in size, its contains huge blank lines. i tired to remove the blank line using below commands but failed to perform and got the below error

Code:
 
$grep -av "^$" trace.log > ~/temp.tmp
$cat temp.tmp
Binary file trace.log matches
$sed '/^$/d' /logs/gopm/trace.log > ~/temp.tmp
sed: Couldn't re-allocate memory
$

Could you please suggest some other solution to over come this issue, appericaite your help
thanks
Senthil
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove blank line - SED

HI, I have this list of apps like so: DivX Products.app DivX Support.app Uninstall DivX for Mac.app Build Applet.app SpringBoard.app Interface.app MobileAddressBook.app MobileSafari.app MobileSlideShow.app Preferences.app Install Flash Player 8 OSX.app Yap.app check_afp.app ... (10 Replies)
Discussion started by: pcwiz
10 Replies

2. UNIX for Dummies Questions & Answers

Newbie help with New Line & Blank Line

I have this in my script: usercount=`ldapsearch -L -b"ou=people,ou=$ou,dc=paisleyhosting,dc=com" -h"$server" -p"1391" cn=* nothing | grep -c dn` admincount=`ldapsearch -b"ou=groups,ou=$ou,dc=paisleyhosting,dc=com" -h"$server" -p"1391" cn=USERADMIN* uniquemember | grep... (5 Replies)
Discussion started by: kthatch
5 Replies

3. Shell Programming and Scripting

Remove last blank line of file

I have a number of files (arranged in directories) which have last line blank, I am trying to synchronize my code with other env and due to this blank lines, all files error out as different although only difference is that of balnk line at end of file. Is there a way I can recursively... (2 Replies)
Discussion started by: ruchimca
2 Replies

4. Shell Programming and Scripting

Perl & Sed command -- Out of Memory Error

Experts, We used to receive our source files with '~^' as row delimiter. This file contains 2500K records and two of the columns having value in HTML formats within the file. While running the below commands against the file, we are encountering out of memory, could you please help to... (3 Replies)
Discussion started by: srivijay81
3 Replies

5. Shell Programming and Scripting

Error PHP Fatal error: Allowed memory size of 67108864 bytes exhausted(tried to allocate 401 bytes)

While running script I am getting an error like Few lines in data are not being processed. After googling it I came to know that adding such line would give some memory to it ini_set("memory_limit","64M"); my input file size is 1 GB. Is that memory limit is based on RAM we have on... (1 Reply)
Discussion started by: elamurugu
1 Replies

6. Shell Programming and Scripting

"Cannot allocate memory" error when run from script

hi in my application there is a line with open(/dev/mydevice,0); it work good when it run manually, but when try to run it within script /////////////////////////////////////////// #!/bin/sh ./device_test 0 0 0 1 1 1 0 0 1 ///////////////////////////////////////// I receive 'Failed... (3 Replies)
Discussion started by: majeed
3 Replies

7. Shell Programming and Scripting

Print lines between a regExp & a blank line

Hi, I have a file, say files_list, as below (o/p of ls -R cmd) $ cat files_list /remote/dir/path/to/file: sub-dir1 sub-dir2 sub-dir3 ... /remote/dir/path/to/file/sub-dir1: remote_file1.csv.tgz <blank line 1> /remote/dir/path/to/file/sub-dir2: remote_file2.csv.tgz <blank... (3 Replies)
Discussion started by: dips_ag
3 Replies

8. Shell Programming and Scripting

How to remove blank line from a text file?

Hi All, I am creating a text file using perl. The first record I am writing as "$line" and all the other as "\n$line". At the end the file is having N number of lines. I am using this file for MLOAD (Teradata), which is reading N+1 lines in the file and failing.I am not able to find new line... (2 Replies)
Discussion started by: unankix
2 Replies

9. Shell Programming and Scripting

'Couldn't read file' error in bash script with expect, sed and awk!

Ok, so I have a bash script with an embedded expect statement. Inside of the expect statement, i'm trying to pull all of the non-comment lines from the /etc/oratab file one at a time. Here's my command: cat /etc/oratab |sed /^s*#/d\ | awk 'NR==1'|awk -F: '{print \"$1\"}'|. oraenv Now,... (0 Replies)
Discussion started by: alexdglover
0 Replies

10. Shell Programming and Scripting

Bash - sed - Remove first word from line which can begin eventually with blank

hello. How to remove first word from line. The line may or may not start with blank. NEW_PARAM1=$(magic-command " -t --protocol=TCP -P 12345-u root -h localhost ") NEW_PARAM2=$(magic-command "-t --protocol=TCP -P 12345 -u root -h localhost ") I want NEW_PARAM1 equal to NEW_PARAM2 equal ... (2 Replies)
Discussion started by: jcdole
2 Replies
PYP(1)							      General Commands Manual							    PYP(1)

NAME
pyp - The Pyed Piper: A Modern Python Alternative to awk, sed and Other Unix Text Manipulation Utilities SYNOPSIS
pyp [options] files ... DESCRIPTION
pyp, the Pyed Piper, is a command line tool for text manipulation. It is similar to awk and sed in functionality, but its subcommands are Python based, and thus more familiar to many programmers. It can operate both on a per-line base and on the complete input stream. Different features can be pipelined in a single command by using the pipe character familiar from shell commands. pyp backs up its input for reruns with modified commands, and can save commands as macros. On the downside, the rerun feature makes it unsuitable for continuous pipe operation. OPTIONS
These programs follow the usual GNU command line syntax, with long options starting with two dashes (`-'). A summary of options is included below. For a complete description, use --manual. -h, --help Show this help message and exit. -m, --manual Prints out extended help. -l, --macro_list Lists all available macros. -s MACRO_SAVE_NAME, --macro_save=MACRO_SAVE_NAME Saves current command as macro. use "#" for adding comments EXAMPLE: pyp -s "great_macro # prints first letter" "p[1]". -f MACRO_FIND_NAME, --macro_find=MACRO_FIND_NAME Searches for macros with keyword or user name. -d MACRO_DELETE_NAME, --macro_delete=MACRO_DELETE_NAME Deletes specified public macro. -g, --macro_group Specify group macros for save and delete; default is user. -t TEXT_FILE, --text_file=TEXT_FILE Specify text file to load. For advanced users, you should typically cat a file into pyp. -x, --execute Execute all commands. -c, --turn_off_color Prints raw, uncolored output. -u, --unmodified_config Prints out generic PypCustom.py config file. -b BLANK_INPUTS, --blank_inputs=BLANK_INPUTS Generate this number of blank input lines; useful for generating numbered lists with variable 'n'. -n, --no_input Use with command that generates output with no input; same as --dummy_input 1. -k, --keep_false Print blank lines for lines that test as False. default is to filter out False lines from the output. -r, --rerun Rerun based on automatically cached data from the last run. Use this after executing "pyp", pasting input into the shell, and hitting CTRL-D. SEE ALSO
awk(1), grep(1), sed(1). AUTHOR
pyp was written by Toby Rosen <tobyrosen@gmail.com>. This manual page was written by Khalid El Fathi <khalid@elfathi.fr>, for the Debian project (and may be used by others). March 19, 2012 PYP(1)
All times are GMT -4. The time now is 06:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy