Sponsored Content
Full Discussion: Out of memory message
Top Forums UNIX for Beginners Questions & Answers Out of memory message Post 303044756 by stomp on Tuesday 3rd of March 2020 09:35:57 AM
Old 03-03-2020
Hi,

I just checked your script on a linux system without any output with a file with 1/3 million words in it (filesize 2700 KB: I used this file: wordlist.xz).

Takes < 0.5 second and uses 100 MB of ram.

Code:
{
   gsub(/[.,:;!?"<>\[\]#(){}]/,"")
    for(i=1; i<=NF; i++){
       trigram = word1 " " word2" " $i
       word1 = word2
       word2 = $i
       count[trigram]++
      }
  }

Code:
/usr/bin/time awk -f t.awk wordlist
0.39user 0.02system 0:00.41elapsed 99%CPU (0avgtext+0avgdata 101800maxresident)k
0inputs+0outputs (0major+25315minor)pagefaults 0swaps

System Info:

Code:
CPU: Quad Core Intel Core i7-2600 (-MT MCP-) speed/max: 3051/3800 MHz Kernel: 4.19.0-8-amd64 x86_64 Up: 4h 08m 
Mem: 9670.3/15995.6 MiB (60.5%) Storage: 931.51 GiB (89.1% used) Procs: 284 Shell: bash 5.0.3 inxi: 3.0.32

In a windows 10 virtual machine with 4 GB of RAM and with GNU awk 3.1.6 - downloaded as compiled binary it takes only little more time.

So it boils down to the question Peasant asked already...

Last edited by stomp; 03-03-2020 at 10:51 AM..
These 3 Users Gave Thanks to stomp For This Post:
 

10 More Discussions You Might Find Interesting

1. AIX

Printer Memory Message

Hello Everyone, I received the following (root) email. Does anyone know what causes this and how I can find the offending printer? Thanks in advance. Jim Message 2: From daemon Wed Nov 30 09:51:07 2005 Date: Wed, 30 Nov 2005 09:51:07 -0800 From: daemon To: root (2 Replies)
Discussion started by: jlslhills
2 Replies

2. Programming

shared memory and message queues

Hi, According to my understanding.. When message queues are used, when a process post a message in the queue and if another process reads it from the queue then the queue will be empty unlike shared memory where n number of processess can access the shared memory and still the contents remain... (2 Replies)
Discussion started by: rvan
2 Replies

3. Programming

How to limit max no of message in a posix message queue

Hii can anyone pls tell how to limit the max no of message in a posix message queue. I have made changes in proc/sys/fs/mqueue/msg_max But still whenever i try to read the value of max. message in the queue using attr.mq_curmsgs (where struct mq_attr attr) its giving the default value as 10.... (0 Replies)
Discussion started by: mohit3884
0 Replies

4. Shell Programming and Scripting

memory full warning message

Hi I wrote a script #!/usr/bin/ksh #set -x for fs in `df -k|awk '{print $1}'|sed -n "3,14 p"` do x=`df -kl | grep $fs | awk '{ print $5 }'` y=50% if then message="File System `df -k |grep $fs |awk '{print $6\", \"$5}'`... (1 Reply)
Discussion started by: namishtiwari
1 Replies

5. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

6. Programming

How to deal with lots of data in memory in order not to run out of memory

Hi, I'm trying to learn how to manage memory when I have to deal with lots of data. Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at once (yep!) but of course the... (1 Reply)
Discussion started by: emitrax
1 Replies

7. Solaris

restrcit physical memory with zone.max-locked-memory

Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Discussion started by: fugitive
1 Replies

8. Programming

POSIX Message Queue Memory Allocation

Hi, I wanted to know whether the POSIX message queues are statically allocated memory by the kernel based on the parameters specified in the open or as and when we send messages, memory are allocated? Does the kernel reserve the specified memory for the message queue irrespective of whether... (1 Reply)
Discussion started by: sumtata
1 Replies

9. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

10. UNIX and Linux Applications

Ssmtp -t < /path/to/the/message.txt (How to format message.txt for html email)

ssmtp has been running well under Kubuntu 12.04.1 for plain text messages. I would like to send html messages with ssmtp -t < /path/to/the/message.txt, but I cannot seem to get the message.txt file properly formatted. I have tried various charsets, Content-Transfer-Encoding, rearranging the... (0 Replies)
Discussion started by: Ronald B
0 Replies
PREZIP-BIN(1)						 Aspell Abbreviated User's Manual					     PREZIP-BIN(1)

NAME
prezip-bin - prefix zip delta word list compressor/decompressor SYNOPSIS
prezip-bin [ -V | -d | -z ] DESCRIPTION
prezip-bin compresses/decompresses sorted word lists from standard input to standard output. Prezip-bin is similar to word-list-compress(1) but it allows a larger character set of {0x00...0x09, 0x0B, 0x0C, 0x0E...0xFF} and multi-words larger than 255 characters in length. It can also decompress word-list-compress(1) compatible files. COMMANDS
Prezip-bin accepts only one of these commands. -V Display prezip-bin version number to standard output. -d Read a compressed word list from standard input and decompress it to standard output. This can be a word-list-compress(1) or a prezip-bin compressed file. -z Read a binary word list from standard input and compress it to standard output. EXAMPLES
prezip-bin -d <wordlist.cwl >wordlist.txt Decompress file wordlist.cwl to text file wordlist.txt prezip-bin -z <wordlist.txt >wordlist.pz 2>errors.txt Compress wordlist.txt to binary file wordlist.pz and send any error messages to a text file named errors.txt LC_COLLATE=C sort -u <wordlist.txt | prezip-bin -z >wordlist.pz Sort a word list, then pipe it to prezip-bin to create a compressed binary wordlist.pz file. prezip-bin -d <words.pz | aspell create master ./words.rws Decompress a wordlist, then pipe it to aspell(1) to create a spelling list. Please check the aspell(1) info manual for proper usage and options. TIPS
Prezip-bin is best used with sorted word list type files. It is not a general purpose compression program since resulting files may actu- ally increase in size. Unlike word-list-compress(1) if your word list has leading or trailing blank spaces for formatting purposes, you should remove them first before you compress your list using prezip-bin -z , otherwise those spaces will be included in the compressed binary output. DIAGNOSTICS
Prezip-bin normally exits with a return code of 0. If it encounters an error, a message is sent to standard error output (stderr), and prezip-bin exits with a non-zero return value. Error messages are listed below: (display help/usage message) Unknown command given on the command line so prezip-bin displays a usage message to standard error output. unknown format The input file appears not to be an expected format, or may possibly be a more advanced format. The output file will be empty. corrupt input This is only for the decompression command -d. The input file appeared to be of a correct format, but something appears wrong now. There may be some valid data in output, but due to input corruption, the rest of the file can not be completed. unexpected EOF The input file appeared okay but ended sooner than expected, therefore the output file is not complete. SEE ALSO
aspell(1), aspell-import(1), run-with-aspell(1), word-list-compress(1) Aspell is fully documented in its Texinfo manual. See the `aspell' entry in info for more complete documentation. REPORTING BUGS
For help, see the Aspell homepage at <http://aspell.net>. Send bug reports/comments to the Aspell user list at the above address. AUTHOR
This info page was written by Jose Da Silva <digital@joescat.com>. prezip-bin-0.1.2 2005-09-30 PREZIP-BIN(1)
All times are GMT -4. The time now is 05:51 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy