Awk limitation

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk limitation
# 15  
Old 09-23-2007
You could try nawk, it may have an extended limit, I can't remember off hand. Otherwise:

1. Install gawk from
2. If 1 is not an option, rewite the affected part in perl.
# 16  
Old 09-23-2007
Hi reborg,

I am not familiar with perl.
Can you help to create a similar code in perl ?
# 17  
Old 09-23-2007
Lightbulb How to work around tool limitations ...

Originally Posted by Raynon
Hi All,

I have an awk code below.
I have an input file which has a line which has a last field with about 4000 characters and it pop up an error stated below. It is too much for awk to take ?

$nawk -f getfull_line_awk myinputfile
nawk: input record `0017c C12efusecntl0 ...' too long
input record number 2795, file myinputfile source line number 1
If you're going to be a UNIX/Linux scripting professional you're going to encounter lots of tools which were written decades ago with hard-coded limits that, by today's standards, are pathetic.

Your choices often boil down to: find a work-around or install more modern and less limited tools (such as the GNU utilities). Sometimes the work-around may entail writing a small utility of your own, in C.

As noted elsewhere in this thread you could install the GNU version of awk (called gawk) or you could use Perl. Both of these are written to avoid hard-coded limits wherever possible. They dynamically allocate memory and re-size their allocations and use the data types which are suitable for the underlying OS for numeric limits (so some things may be limited to 32-bit values on a 32-bit platform, etc).

Other work-arounds depend quite a bit on the problem at hand.

For example, in your code:

Awk Code:
{if( $NF == "2007" && $1 == "**" ) LINE = $0;} 
END{printf("%20s\n",LINE); }

It might be possible for you to use grep filter out all of the lines that start with ** (a literal pair of asterisk characters) and feed just those through your (limited version of) awk. Obviously that will only work if the lines beginning with ** are smaller than this 4000 character limitation; and assuming that your version of grep can cope with longer lines.

Similar workarounds might be possible by prefiltering with sed or cut (if the stuff you cut out of the long lines isn't needed in the results), etc.

(former Linux Gazette AnswerGuy)
# 18  
Old 09-24-2007

First option of installation of newer version of awk is out because i cannot administer/control the whole system.
I tried to use grep "**" myinputfile but it seems to produce the whole input file again which is very strange

Can anybody provide me with a perl code which as a similar function as my awk code ?

Last edited by Raynon; 09-24-2007 at 03:20 AM..
# 19  
Old 09-25-2007
Hi ,

Can any experts provide me with some advice on this perl code?
# 20  
Old 09-26-2007
Hi ,

Anybody can help me with this by providing me a perl code?
# 21  
Old 09-27-2007
Use a2p to convert awk to perl, ...
$[ = 1;                 # set array base to 1

while (<>) {
    chomp;      # strip record separator
    @Fld = split(' ', $_, 9999);
    if ($Fld[$#Fld] eq '2007' && $Fld[1] eq '**') {
        $LINE = $_;

printf "%20s\n", $LINE;

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

What is the limitation in AIX?

Hi All, i got few questions... 1) What is the maximum number of files that we can save under a single directory in AIX ? (* we have enough storage/disk space) 2) And what is the maximum number of sub - directories in side a directory? I know that...every directory is a (special)... (11 Replies)
Discussion started by: System Admin 77
11 Replies

2. UNIX for Dummies Questions & Answers

Limitation in addition

whats wrong with this addition? Whats the maximum number of digits can be handled? pandeeswaran@ubuntu:~/Downloads$ const=201234454654768979799999 pandeeswaran@ubuntu:~/Downloads$ let new+=const pandeeswaran@ubuntu:~/Downloads$ echo $new -2152890657037557890 pandeeswaran@ubuntu:~/Downloads$ (4 Replies)
Discussion started by: pandeesh
4 Replies

3. Shell Programming and Scripting

Limitation on rm command

Hi all, does any one know ,if there is any limitation on rm command limitation referes here as a size . Ex:when my script try to rum rm command which have size of nearly 20-22 GB ..CPU load gets high ? if anyone know the relation of CPU load and limitation of rm command . (8 Replies)
Discussion started by: niteshagrawal06
8 Replies

4. UNIX for Advanced & Expert Users

size for sum variable limitation on awk

Hello first, truth been told, I'm not even close to be advanced user. I'm posting here because maybe my question is complicated enough to need your expert help I need to use awk (or nawk - I don't have gawk) to validate some files by computing the total sum for a large numeric variable. It... (1 Reply)
Discussion started by: cwitarsa
1 Replies

5. HP-UX

Limitation on *.ext

Is there a size limit when passing an argument using wildcards? I.E. when I pass an argument in the form (like) "ftp_auto *.txt" - is there a limitation on the size of UNIX expanding "*.txt" ? (1 Reply)
Discussion started by: vslewis
1 Replies

6. UNIX for Dummies Questions & Answers

grep limitation

Hello, I am looking for a way to get around an issue, as I am using the grep command in a very common situation: grep ^50 File.*.txt | "some awk process" My problem is that bash throws me an error on the grep command if the directory in question contains several thousands files. ... (6 Replies)
Discussion started by: Indalecio
6 Replies

7. Shell Programming and Scripting

Is this a bug or a limitation?

Hi, I'm having a problem with a while loop syntax that doesn't seem to loop correctly. TODAY=`date +%d%m%Y` while read hostname #for hostname in $(cat $CONFIG) do OUTFILE=/tmp/health_check.$hostname.$TODAY if then touch $OUTFILE func_header else rm $OUTFILE ... (2 Replies)
Discussion started by: gilberteu
2 Replies

8. Shell Programming and Scripting

Limitation of ls command

Hi, Iam using an alias to get the file count from one directory using normal ls command like ls file*|wc -l.If my file increases more than 35,000 ,my alias is not working.It shows that arg list too long. is that can be limitation of ls or problem in alias? I would appreciate if anyone can... (2 Replies)
Discussion started by: cskumar
2 Replies

9. HP-UX

HP-UX 11i - File Size Limitation And Number Of Folders Limitation

Hi All, Can anyone please clarify me the following questions: 1. Is there any file size limitation in HP-UX 11i, that I can able to create upto certain size of file (say 2 GB) and not more then that???? 2. At max. how many files we can able to keep inside a folder???? 3. How many... (2 Replies)
Discussion started by: sundeep_mohanty
2 Replies

10. Shell Programming and Scripting

find limitation

Hi , i'm trying to use "find "command with "-size "option but i encounter 2gb file limitation. Can you confirm this limitation ? Is there a simple way to do the same thing ? My command is : <clazz01g-notes01>/base/base01 # find /base/base01 -name '*.nsf' -size +5242880000c -exec ls... (2 Replies)
Discussion started by: Nicol
2 Replies
Login or Register to Ask a Question