Sponsored Content
Top Forums Shell Programming and Scripting Improve script - slow process with big files Post 302990410 by RudiC on Thursday 26th of January 2017 06:25:36 AM
Old 01-26-2017
To expand on what stomp already said, your script seems to read the input file n+1 times, once to detect the count of reports ( = n) and their respective location in it (one sed and one awk invocation), then using a shell loop to "extract" and analyse the respective single reports, invoking awk 13 times, and sed 4 times per loop (i.e. 130 awks and 40 seds for the sample file with 10 reports).

No surprise this will be somewhat lengthy on large files with many reports...
This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

looking for solution to improve process replicate files to remote loc.

looking for solution to replicate 1.5GB files to a remote location... Currently, this process looks like the following: move 1.5GB files into a staging area. compress files. rsync files to remote server. remove compressed files. I have performed some timings, and compress seems more... (5 Replies)
Discussion started by: mr_manny
5 Replies

2. Shell Programming and Scripting

bash script working for small size files but not for big size files.

Hi, I have one file stat. Stat file contents are as follows: for example. H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000 H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000 Now i... (1 Reply)
Discussion started by: davidpreml
1 Replies

3. AIX

How to send big files over slow network?

Hi, I am trying to send oracle archives over WAN and it is taking hell a lot of time. To reduce the time, I tried to gzip the files and send over to the other side. That seems to reduce the time. Does anybody have experienced this kind of problem and any possible ways to reduce the time. ... (1 Reply)
Discussion started by: giribt
1 Replies

4. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
Discussion started by: hidnana
7 Replies

5. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

6. Shell Programming and Scripting

Very big text file - Too slow!

Hello everyone, suppose there is a very big text file (>800 mb) that each line contains an article from wikipedia. Each article begins with a tag (<..>) containing its url. Currently there are 10^6 articles in the file. I want to take random N articles, eliminate all non-alpharithmetic... (14 Replies)
Discussion started by: fedonMan
14 Replies

7. UNIX for Dummies Questions & Answers

How do I slow down a process?

Hello, I've been searching for something that slows down a process for some time now. Slow down as in make time pass by slower. I have rarely turned to asking a forum in the past but at this point I've given up. For example: if I made a program that would print "Hello" in 5 seconds, I would use... (18 Replies)
Discussion started by: Nathan1
18 Replies

8. HP-UX

Script execution is very slow when trying to find all files and their owners on HP-UX box

Hi, I have a HP-UX server were I need to list all the files in the entire file system, their directory path, last modified date, owner and group. I do not need to search the file contents. I created the script given below and I am excluding directories and files of type tmp, temp and log. The... (4 Replies)
Discussion started by: Adyan Faruqi
4 Replies

9. Solaris

Rsync quite slow (using very little cpu): how to improve its speed?

I have "inherited" a OmniOS (illumos based) server. I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware. Using same hardware, same command with same source and target disks, OmniOS r151026 gives: test@omniosce:~# time... (11 Replies)
Discussion started by: priyadarshan
11 Replies

10. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
DH_INSTALLINIT(1)						     Debhelper							 DH_INSTALLINIT(1)

NAME
dh_installinit - install init scripts and/or upstart jobs into package build directories SYNOPSIS
dh_installinit [debhelperoptions] [--name=name] [-n] [-R] [-r] [-d] [--params] DESCRIPTION
dh_installinit is a debhelper program that is responsible for installing init scripts with associated defaults files, as well as upstart job files into package build directories. It also automatically generates the postinst and postrm and prerm commands needed to set up the symlinks in /etc/rc*.d/ to start and stop the init scripts. FILES
debian/package.init If this exists, it is installed into etc/init.d/package in the package build directory. debian/package.default If this exists, it is installed into etc/default/package in the package build directory. debian/package.upstart If this exists, it is installed into etc/init/package.conf in the package build directory. OPTIONS
-n, --noscripts Do not modify postinst/postrm/prerm scripts. -o, --onlyscripts Only modify postinst/postrm/prerm scripts, do not actually install any init script, default files, or upstart job. May be useful if the init script or upstart job is shipped and/or installed by upstream in a way that doesn't make it easy to let dh_installinit find it. -R, --restart-after-upgrade Do not stop the init script until after the package upgrade has been completed. This is different than the default behavior, which stops the script in the prerm, and starts it again in the postinst. This can be useful for daemons that should not have a possibly long downtime during upgrade. But you should make sure that the daemon will not get confused by the package being upgraded while it's running before using this option. -r, --no-restart-on-upgrade Do not stop init script on upgrade. --no-start Do not start the init script on install or upgrade, or stop it on removal. Only call update-rc.d. Useful for rcS scripts. -d, --remove-d Remove trailing d from the name of the package, and use the result for the filename the upstart job file is installed as in etc/init/ , and for the filename the init script is installed as in etc/init.d and the default file is installed as in etc/default/ . This may be useful for daemons with names ending in d. (Note: this takes precedence over the --init-script parameter described below.) -uparams --update-rcd-params=params -- params Pass params to update-rc.d(8). If not specified, defaults will be passed to update-rc.d(8). --name=name Install the init script (and default file) as well as upstart job file using the filename name instead of the default filename, which is the package name. When this parameter is used, dh_installinit looks for and installs files named debian/package.name.init, debian/package.name.default and debian/package.name.upstart instead of the usual debian/package.init, debian/package.default and debian/package.upstart. --init-script=scriptname Use scriptname as the filename the init script is installed as in etc/init.d/ (and also use it as the filename for the defaults file, if it is installed). If you use this parameter, dh_installinit will look to see if a file in the debian/ directory exists that looks like package.scriptname and if so will install it as the init script in preference to the files it normally installs. This parameter is deprecated, use the --name parameter instead. This parameter is incompatible with the use of upstart jobs. --error-handler=function Call the named shell function if running the init script fails. The function should be provided in the prerm and postinst scripts, before the #DEBHELPER# token. NOTES
Note that this command is not idempotent. dh_prep(1) should be called between invocations of this command. Otherwise, it may cause multiple instances of the same text to be added to maintainer scripts. SEE ALSO
debhelper(7) This program is a part of debhelper. AUTHORS
Joey Hess <joeyh@debian.org> Steve Langasek <steve.langasek@canonical.com> 9.20120909 2012-04-10 DH_INSTALLINIT(1)
All times are GMT -4. The time now is 02:48 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy