Sponsored Content
Top Forums Shell Programming and Scripting Perl : Large amount of data put into an array Post 302819311 by DGPickett on Monday 10th of June 2013 03:21:36 PM
Old 06-10-2013
The only thing faster than the scalar loop is to memory map the file in PERL so it is a big string in memory, and not by much, as many OS read flat files via mmap() (automatic buffering in RAM via VM using no swap).

It's a classic case of advanced tools doing expensive favors for you! But there is a base cost below which perl level code cannot go.

Compressing the file might speed the flow out a pipe, as CPUs are so much faster than disks. Old compress is faster than gzip -1. However, if it gets referenced on the same host often, it may be in RAM.
 

10 More Discussions You Might Find Interesting

1. Linux

shmat() Failure While Using a Large Amount of Shared Memory

Hi, I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size... (2 Replies)
Discussion started by: theicarusagenda
2 Replies

2. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

3. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

4. Shell Programming and Scripting

Perl Array / pattern match large CPU usage

Hi, I have one file in this format 20 value1 33 value2 56 value3 I have another file in this format: 34,30-SEP-09,57,100237775,33614510126,2,34 34,30-SEP-09,57,100237775,33620766654,2,34 34,30-SEP-09,108,100237775,33628458122,2,34 34,30-SEP-09,130,100237775,33635266741,2,254... (6 Replies)
Discussion started by: Donkey25
6 Replies

5. Shell Programming and Scripting

perl, put one array into many array when field is equal to sth

Hi Everyone, #!/usr/bin/perl use strict; use warnings; my @test=("a;b;qqq;c;d","a;b;ggg;c;d","a;b;qqq;c;d"); would like to split the @test array into two array: @test1=(("a;b;qqq;c;d","a;b;qqq;c;d"); and @test2=("a;b;ggg;c;d"); means search for 3rd filed. Thanks find the... (0 Replies)
Discussion started by: jimmy_y
0 Replies

6. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

7. Shell Programming and Scripting

storing large data in unix array variable

Hi, I have table in sql ..from this table im storing the first coloumn values in shell array variable ... after this passing this variable as an arugument in SQL procedure. But the proc. is running fine only for 1024 values in array ... How to store more than 1024 values in the array... (5 Replies)
Discussion started by: ankitknit
5 Replies

8. Shell Programming and Scripting

How do I put data piped to my script into an array for repeated processing

Hi folks, Sorry for something I'm sure was answered already, I just could not find it in a search of the forums. I am trying to build a script that eats a config file as: cat file.cnf | ConProcess.shWhat I want to do inside the script is: !#/usr/bin/bash # captured piped cat into an... (6 Replies)
Discussion started by: Marc G
6 Replies

9. Shell Programming and Scripting

Read file and get the data then put it in array

Hi, I have a file called "readfile" it contains below parameters #cat readfile word=/abc=225,/abc/cba=150 three=12 four=45 five=/xyz/yza likewise multiple line. From the above file, I have to read "word" output should be like, /abc /abc/cba these values need to be put in... (3 Replies)
Discussion started by: munna_dude
3 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
DPHYS-SWAPFILE(8)					      System Manager's Manual						 DPHYS-SWAPFILE(8)

NAME
dphys-swapfile - set up, mount/unmount, and delete an swap file SYNOPSIS
dphys-swapfile setup|swapon|swapoff|uninstall DESCRIPTION
dphys-swapfile computes the size for an optimal swap file (and resizes an existing swap file if necessary), mounts an swap file, unmounts it, and and delete it if not wanted any more. OPTIONS
There is only one parameter, an command, which can be either of these: setup Tells dphys-swapfile to compute the optimal swap file size and (re-)generate an fitting swap file. Default it 2 times RAM size. This can be called at boot time, so the file allways stays the right size for current RAM, or run by hand whenever RAM size has changed. swapon and swapoff These run the swapon and swapoff commands on the swapfile. Note that direct swapon/off from /etc/fstab is not possible, as that is (at least on Debian) done in the same script that mounts /var (which is where the swap file most likely resides). And we need to do our setup between those actions. So pass up /etc/fstab, and do our own swapon/off. uninstall Gets rid of an unwanted swap file, reclaiming the disk space. CONFIG
The config file /etc/dphys-swapfile allows the user to set up the working environment for dphys-swapfile. This config file is a sh script fragment full of assignments, which is sourced. Standard sh syntax rules apply. Assignments are: CONF_SWAPFILE Set where the swap file should be placed. Defaults to /var/swap. It is unlikely that you will need to change this, unless you have very strange partitioning, and then you will most likely be using an swap partition anyway. CONF_SWAPSIZE Force file size to this. Default is 2*RAM size. This is unlikely to be needed, unless in strange diskspace situations. Note that swap enabled and smaller than RAM causes kernal-internal VM trouble on random systems. CONF_SWAPFACTOR Set the relation between RAM and swap size. Must be an integer. Defaults to 2 which means swap size = 2 * RAM size CONF_MAXSWAP Set maximum size of the swap file in MBytes. Defaults to 2048 which was the former kernel limit for the swapfile size and is now a limit to prevent unusual big swap files on systems with a lot of RAM. FILES
/etc/dphys-swapfile user config $CONF_SWAPFILE the swap file, target of the whole action (defaults to /var/swap) EXAMPLES
dphys-swapfile is usually run at system startup and shutdown from an /etc/init.d (or /etc/rc.d) script, such as this (minimal) one: #!/bin/sh # /etc/init.d/dphys-swapfile - automatically set up an swapfile # author franklin, last modification 2004.06.04 # This script is copyright ETH Zuerich Physics Departement, # use under either modified/non-advertising BSD or GPL license case "$1" in start) /sbin/dphys-swapfile setup /sbin/dphys-swapfile swapon ;; stop) /sbin/dphys-swapfile swapoff ;; esac exit 0 If an sysadmin wants to have his swapfile in annother place, say /var/run/swap, he can use: In /etc/dphys-swapfile: CONF_SWAPFILE=/var/run/swap AUTHOR
franklin@phys.ethz.ch, http://www.phys.ethz.ch/~franklin/ D-PHYS Swapfile Tools 2006.09.15 DPHYS-SWAPFILE(8)
All times are GMT -4. The time now is 12:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy