Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

GZIP memory constraints

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users GZIP memory constraints
# 1  
Old 06-26-2007
GZIP memory constraints
Currently I am using the ZLIB_VERSION "1.2.3" . The memory requirement for Zlib/GZIP compression is stated as


/* The memory requirements for deflate are (in bytes):
(1 << (windowBits+2)) + (1 << (memLevel+9))
that is: 128K for windowBits=15 + 128K for memLevel = 8 (default values)
plus a few kilobytes for small objects. For example, if you want to reduce
the default memory requirements from 256K to 128K, compile with
make CFLAGS="-O -DMAX_WBITS=14 -DMAX_MEM_LEVEL=7"
Of course this will generally degrade compression */

If I can make changes to these macro to reduce memory consumption,
MAX_WBITS=14
MAX_MEM_LEVEL = 7

How it will effect the GZIP compression method ?
What would be the memory requirement for window size, the size of Heads of the hash chains,
and prev pointer which is basically linking to older string with the same hash index?

Also, what are the various ways to reduce memory usage?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Solaris

Need help in configuring Password Constraints for Solaris 11.3

Hello Friends I was running Solaris 11.3 x86. Below is my configuration to set password Constraints. more /etc/default/passwd MAXWEEKS=4 MINWEEKS=1 WARNWEEKS=1 PASSLENGTH=8 NAMECHECK=YES HISTORY=4 MINDIFF=3 MINALPHA=2 (4 Replies)
Discussion started by: jebby123
4 Replies

2. UNIX for Dummies Questions & Answers

Grep with time constraints

Hello Friends - I am trying to grep certain messages that have a time slot like this: MyRate=33FC|SystemDEF=445DE|Calc=33W2|Time=15:50:24 I am trying to grep everything after Time=15:50:26 including SystemDEF=E2S and Calc=33W2 into a file called myrate.dat Not able to... (7 Replies)
Discussion started by: DallasT
7 Replies

3. Shell Programming and Scripting

Parsing a file based on positional constraints

I have a list file1 like dog cow fox cat fish duck crowI want to classify the elements of file1 based on constrains applied on file2. Additionally the number of elements (words) in the each line of file2 is not fixed. This is my file2 cow cat fox dog cow fox dog fish crow fox dog cat ... (5 Replies)
Discussion started by: sammy777
5 Replies

4. Solaris

Pkg update: No solution was found to satisfy constraints

I have an x86 Solaris box running 11.2 and have run into the following issue when attempting to run a package update. Has anyone else come across this issue and resolved it successfully, or am I waiting on Oracle to release other updated packages? uname -a SunOS <hostname> 5.11 11.2... (13 Replies)
Discussion started by: nova_cyclist
13 Replies

5. UNIX for Advanced & Expert Users

Teradata and Informatica Load constraints

HI Team , I have interesting issue observed when using teradata sql assistant(14.1) and Informatica tool (9.5) versions. I created SQL code in teradata where source count is 5000 records . I am using source and target database as teradata and trying to load using informatica tool . Its straight... (0 Replies)
Discussion started by: Perlbaby
0 Replies

6. UNIX for Advanced & Expert Users

gzip vs pipe gzip: produce different file size

Hi All, I have a random test file: test.txt, size: 146 $ ll test.txt $ 146 test.txt Take 1: $ cat test.txt | gzip > test.txt.gz $ ll test.txt.gz $ 124 test.txt.gz Take 2: $ gzip test.txt $ ll test.txt.gz $ 133 test.txt.gz As you can see, gzipping a file and piping into gzip... (1 Reply)
Discussion started by: hanfresco
1 Replies

7. UNIX for Advanced & Expert Users

How to reduce GZIP memory usage

I am using the ZLIB_VERSION "1.2.3" . The memory requirement for Zlib/GZIP compression is stated as /* The memory requirements for deflate are (in bytes): (1 << (windowBits+2)) + (1 << (memLevel+9)) that is: 128K for windowBits=15 + 128K for memLevel = 8 (default values) ... (0 Replies)
Discussion started by: Parmod Garg
0 Replies

8. Programming

problem about using zlib to uncompress gzip in memory

I wrote a function which for uncompressing data for gzip or deflate format using zlib,see followed code; source param is pointed to the compressed data,len param is the size of compressed data, dest param is for returning the address which pointed to the uncompressed data;the last gzip param tell... (0 Replies)
Discussion started by: iwishfine
0 Replies

9. Programming

About template constraints

Hi, i have class template, 1)can i override the copy constructor 2)can we have virtual function in class template if not plz tel why? I tried , compile error comes for me... Thanks Sarwan (0 Replies)
Discussion started by: sarwan
0 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

h5repack

h5repack(1)						      General Commands Manual						       h5repack(1)

NAME

       h5repack - Copy an HDF5 file to a new file with or without compression/chunking

SYNOPSIS

       h5repack -i file1 -o file2 [-h] [-v] [-f 'filter'] [-l 'layout'] [-m number] [-e file]

DESCRIPTION

       h5repack is a command line tool that applies HDF5 filters to a input file file1, saving the output in a new file, file2.

       'filter' is a string with the format	 <list of objects> : <name of filter> = <filter parameters>.

       <list  of objects> is a comma separated list of object names meaning apply compression only to those objects. If no object names are speci-
       fied, the filter is applied to all objects.

       <name of filter> can be:
	   GZIP, to apply the HDF5 GZIP filter (GZIP compression)
	   SZIP, to apply the HDF5 SZIP filter (SZIP compression)
	   SHUF, to apply the HDF5 shuffle filter
	   FLET, to apply the HDF5 checksum filter
	   NONE, to remove the filter

       <filter parameters> contains the optional compression information:
	   SHUF (no parameter)
	   FLET (no parameter)
	   GZIP=<deflation level> from 1-9
	   SZIP=<pixels per block,coding> (pixels per block is a even number in 2-32 and coding method is 'EC' or 'NN')

       'layout' is a string with the format
	   <list of objects> : <layout type>

       <list of objects> is a comma separated list of object names, meaning that layout information is supplied for those objects.  If	no  object
       names are specified, the layout is applied to all objects.

       <layout type> can be:
	   CHUNK, to apply chunking layout
	   COMPA, to apply compact layout
	   CONTI, to apply continuous layout

       <layout parameters> is present for the chunk case only it is the chunk size of each dimension: <dim_1 x dim_2 x ... dim_n>

OPTIONS

       file1,file2
	      The input and output HDF5 files

       -h     Print a help message

       -f filter
	      Filter type

       -l layout
	      Layout type

       -v     Verbose mode. Print output (list of objects in the file, filters and layout applied).

       -e file
	      File with the -f and -l options (only filter and layout flags)

       -d delta
	      Print  only differences that are greater than the limit delta.  delta must be a positive number. The comparison criterion is whether
	      the absolute value of the difference of two corresponding values is greater than delta (e.g., |a-b| > delta, where a is a  value	in
	      file1 and b is a value in file2).

       -m number
	      Do  not  apply  the filter to objects which size in bytes is smaller than number. If no size is specified a minimum of 1024 bytes is
	      assumed.

EXAMPLES

       Apply GZIP compression to all objects in file1 and save the output in file2:

	    h5repack -i file1 -o file2 -f GZIP=1 -v

       Apply SZIP compression only to object 'dset1':

	    h5repack -i file1 -o file2 -f dset1:SZIP=8,NN -v

       Apply a chunked layout to objects 'dset1' and 'dset2':

	    h5repack -i file1 -o file2 -l dset1,dset2:CHUNK=20x10 -v

SEE ALSO

       h5dump(1), h5ls(1), h5diff(1), h5import(1), gif2h5(1), h52gif(1), h5perf(1), h5repart(1).

																       h5repack(1)