Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Maximum size of sed file...
Top Forums UNIX for Dummies Questions & Answers Maximum size of sed file... Post 17731 by peter.herlihy on Tuesday 19th of March 2002 06:52:52 PM
Old 03-19-2002
Yes - sed was used in creating the sed file.....

The purpose for this is to apply an encryption algorithm against a field within each record of a large file.

1. I first extract all of the values in a certain field from each record.
2. I then apply an encryption algorithim to each value in this new file and that creates another file.
3. I then 'paste' the two files together - which leaves me with 2 columns - tab telimited - first column of raw data - second column encrypted data.
4. I use a series of sed commands to change each line to 's/old/new/g'
5. I then use the output of this as a sed file - which I split into a number of smaller sed files (due to this limitation) and apply the substitutions in each file against the original file - effectively substituting all the raw data in each record for its encrypted equivalent.
6. I then do each of these steps (1-5) against a number of different files. (there are a couple of spurious steps such as removing header records etc when creating the sed files - but they're not that relevant).

The original files are flat (text) files. And when the big sed file with all the substitutions iscreated- there are 1,000s of rows. The limitation seems to actually be 199 commands in a sedfile (as 200 stops with the too many arguements error).

Extreme programming - I assure you it's far from it.

I know there are other ways to do this - i.e. 1 record at a time.... but that seems fairly inefficient.
Last edited by peter.herlihy; 03-19-2002 at 07:58 PM..
 

10 More Discussions You Might Find Interesting

1. Programming

Maximum File Size

Hi, When i checked for the maximum file size on solaris 5.9 the max file size obtained was only 2147483647 and all the further writes to the file which had reached that max size is not added to that file. even i had registered the signal SIGXFSZ, but the signal was not delivered to the... (5 Replies)
Discussion started by: matrixmadhan
5 Replies

2. UNIX for Dummies Questions & Answers

Maximum input file size in "Diff" Command

Hello, Can anyone let me know what is the maximum file size that can be given as input for the "Diff" Command in Unix? I have a file size as large as 28MB and which can also increase. Will I face any issues with such a file size. If yes, What is the other alternative. Thanks in advance for... (1 Reply)
Discussion started by: Neeraja
1 Replies

3. HP-UX

Mailx Maximum attachment size

Hi friends, what is the maximum allowable attachment size, we can send using mailx command. what is an alternate option if my file is > than that size? thanks sreeji (0 Replies)
Discussion started by: sreejithau
0 Replies

4. UNIX for Dummies Questions & Answers

Maximum size of a file in unix

What's the maximum file size supported by unix. (3 Replies)
Discussion started by: nagalenoj
3 Replies

5. UNIX for Dummies Questions & Answers

maximum tar file size?

Is there a reasonable maximum limit for tar file sizes? I want to transfer a pile of files from one server to another but have restricted means, so tarring them first will probably be best... but how big can I go - both for the file format itself and for the operating system (linux) to handle? ... (7 Replies)
Discussion started by: Bobby
7 Replies

6. UNIX for Dummies Questions & Answers

Maximum file size ????

Hi All, - block size of 512KB & every address requires 4 bits - The inode structure contains 10 direct pointers, 1 single indirect, 1 double indirect & 1 triple indirect pointer What could be the possible maximum file size for this system Any guess? I am unable to understand the question... (0 Replies)
Discussion started by: preethgideon
0 Replies

7. AIX

maximum size of a ramdisk on AIX 5.3

Hi, Do you know what is the maximum size I can use to create a ramdisk on AIX 5.3? I m pretty sure i've seen somewhere i can use more than 2 Gb but I can't remember where. I need to do some recommandations for one of my customer and they'll need to create a ramdisk of 20 Gb. Can this be done? ... (1 Reply)
Discussion started by: cedric hanquez
1 Replies

8. UNIX for Advanced & Expert Users

How to identify maximum stack size?

Hi All, I have set max stack size as 4KB for my thread, but it always using very less. So I like to know what is the maximum stack size is used by my thread. I tried with gcc -fstack-usage command line option, but its not supported by mips. Kindly suggest me the way to find the max stack... (6 Replies)
Discussion started by: rajamohan
6 Replies

9. Programming

Maximum buffer size for read()

Hi friends, Hope everybody is fine. First have a look at my code, then we will talk about it. $ cat copy.c #include <stdio.h> #define PERMS 0644 /* RW for owner, R for group, others */ #define BUFSIZE 1 char *progname; int main(int argc,char * argv) { int f1, f2, n; ... (4 Replies)
Discussion started by: gabam
4 Replies

10. Shell Programming and Scripting

Maximum size of attachment in mail

Hi Friends, My requirment is to Query the oracle database , generate the file change the extension to .csv and send to clients automatically everyday. However i am able to perform the task. But sometimes when the file size is getting increased more than 1 MB then the mail is... (5 Replies)
Discussion started by: Showdown
5 Replies

LEARN ABOUT LINUX

shtool-install

SHTOOL-INSTALL.TMP(1)					      GNU Portable Shell Tool					     SHTOOL-INSTALL.TMP(1)

NAME

       shtool-install - GNU shtool install(1) command

SYNOPSIS

       shtool install [-v|--verbose] [-t|--trace] [-d|--mkdir] [-c|--copy] [-C|--compare-copy] [-s|--strip] [-m|--mode mode] [-o|--owner owner]
       [-g|--group group] [-e|--exec sed-cmd] file [file ...]  path

DESCRIPTION

       This command installs a one or more files to a given target path providing all important options of the BSD install(1) command.	The trick
       is that the functionality is provided in a portable way.

OPTIONS

       The following command line options are available.

       -v, --verbose
	   Display some processing information.

       -t, --trace
	   Enable the output of the essential shell commands which are executed.

       -d, --mkdir
	   To maximize BSD compatiblity, the BSD "shtool "install -d"" usage is internally mapped to the "shtool "mkdir -f -p -m 755"" command.

       -c, --copy
	   Copy the file to the target path. Default is to move.

       -C, --compare-copy
	   Same as -c except if the destination file already exists and is identical to the source file, no installation is done and the target
	   remains untouched.

       -s, --strip
	   This option strips program executables during the installation, see strip(1). Default is to install verbatim.

       -m, --mode mode
	   The file mode applied to the target, see chmod(1). Setting mode to ""-"" skips this step and leaves the operating system default which
	   is usually based on umask(1). Some file modes require superuser privileges to be set. Default is 0755.

       -o, --owner owner
	   The file owner name or id applied to the target, see chown(1). This option requires superuser privileges to execute. Default is to skip
	   this step and leave the operating system default which is usually based on the executing uid or the parent setuid directory.

       -g, --group group
	   The file group name or id applied to the target, see chgrp(1). This option requires superuser privileges to execute to the fullest
	   extend, otherwise the choice of group is limited on most operating systems.	Default is to skip this step and leave the operating
	   system default which is usually based on the executing gid or the parent setgid directory.

       -e, --exec sed-cmd
	   This option can be used one or multiple times to apply one or more sed(1) commands to the file contents during installation.

EXAMPLE

	#   Makefile
	install:
	     :
	    shtool install -c -s -m 4755 foo $(bindir)/
	    shtool install -c -m 644 foo.man $(mandir)/man1/foo.1
	    shtool install -c -m 644 -e "s/@p@/$prefix/g" foo.conf $(etcdir)/

HISTORY

       The GNU shtool install command was originally written by Ralf S.  Engelschall <rse@engelschall.com> in 1997 for GNU shtool. It was prompted
       by portability issues in the installation procedures of OSSP libraries.

SEE ALSO

       shtool(1), umask(1), chmod(1), chown(1), chgrp(1), strip(1), sed(1).

18-Jul-2008							   shtool 2.0.8 					     SHTOOL-INSTALL.TMP(1)
All times are GMT -4. The time now is 04:53 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy