I have a bash script that will take approx. 130 days to complete. I am trying to grep a list of 1,144 user ID's out of 41 (1 GB each) files. The 41 files were originally one 41 G file, but that was horrendously too slow.
This is my current file:
Currently, I am processing one of the 41 files every 4 minutes. 4 x 41 = 164 min.
164 / 60 (min/hour) = 2.73 hours per user_id. I have 1,144 user_id's multiplied by 2.73 = 3123.12 hours. 3123.12 / 24 (hours in a day) = 130.13 days.
As you can see, that is way too long to process this task. I don't know PERL but I've heard its faster. If anyone has any suggestions please let me know.
Last edited by vbe; 01-29-2010 at 10:54 AM..
Reason: code tags please
Hello experts,
I have a series issue in script that result with bad peformence and I wonder if you can assist me.
For example
I have two files:
File-New, size 15Mb.
File-Old, size 1Mb.
File-New content:
a
b
c
k
File-Old content:
d
f
a
b (0 Replies)
Hi,
I am running a BASH shell with the following script. The script works and gives me correct output but is very slow with large files. The more rows and columns (width and height) the slower as you can probably see.
How can I do what I want more efficiently? Any ideas welcome. It has been... (10 Replies)
I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file
STRUCTURE OF THE TWO FILES
The first file is a series of names
ramesh
sushil
jonga
sudesh
lugdi
whereas the second file (could be... (4 Replies)
My script builds a lot of these array lists, then compares their sizes which solves my problem, but runs very slow. :(
set -A comboSorted -- $(
for x in ${IDs}
do
nawk -v s=$x '
BEGIN { testPattern="^" s "$" }
{
if ( $2 ~ testPattern ) {
getline;getline;
if ($1 == "IMAGE_SIZE")
print... (1 Reply)
Basically my requirement is to know the total number of free anonymous ports.
anonymous port range is 32768- 65535.
i wrote a script for that
**********************************************
for i in {32768..65535}
do
netstat -an | grep $i > /dev/null
if
... (21 Replies)
Hi
I know the basic about script and sleep processes.
However this is more tricky:
I would like to run
sh -x /sbin/dhclient-script and slow down the output of the script as a whole.
How would you do it?
I would like to delay output on the screen with 1 second for every line for the output... (3 Replies)
I have shell program as below
#!/bin/sh
echo ======= LogManageri start ==========
#This directory is getting the raw data from remote server
Raw_data=/opt/ftplogs
# This directory is ready for process the data
Processing_dir=/opt/processing_dir
# This directory is prcoessed files and... (4 Replies)
Hello,
I am showing the start of my script.
I am finding that 'xmessage' is taking about 12-15 seconds to show.
This in a terminal is very quick '/opt/vc/bin/vcgencmd get_camera'.
Is there any way to get 'camera not detected' to show faster.
Regards
#!/bin/bash
s=$(/opt/vc/bin/vcgencmd... (4 Replies)
I'm new to utilities like socat and netcat and I'm not clear if they will do what I need.
I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes.
I'm using Cygwin bash on Windows 10.
My... (3 Replies)
Discussion started by: siegfried
3 Replies
LEARN ABOUT DEBIAN
httpindex
httpindex(1) General Commands Manual httpindex(1)NAME
httpindex - HTTP front-end for SWISH++ indexer
SYNOPSIS
wget [ options ] URL... 2>&1 | httpindex [ options ]
DESCRIPTION
httpindex is a front-end for index++(1) to index files copied from remote servers using wget(1). The files (in a copy of the remote direc-
tory structure) can be kept, deleted, or replaced with their descriptions after indexing.
OPTIONS
wget Options
The wget(1) options that are required are: -A, -nv, -r, and -x; the ones that are highly recommended are: -l, -nh, -t, and -w. (See the
EXAMPLE.)
httpindex Options
httpindex accepts the same short options as index++(1) except for -H, -I, -l, -r, -S, and -V.
The following options are unique to httpindex:
-d Replace the text of local copies of retrieved files with their descriptions after they have been indexed. This is useful to display
file descriptions in search results without having to have complete copies of the remote files thus saving filesystem space. (See
the extract_description() function in WWW(3) for details about how descriptions are extracted.)
-D Delete the local copies of retrieved files after they have been indexed. This prevents your local filesystem from filling up with
copies of remote files.
EXAMPLE
To index all HTML and text files on a remote web server keeping descriptions locally:
wget -A html,txt -linf -t2 -rxnv -nh -w2 http://www.foo.com 2>&1 |
httpindex -d -e'html:*.html,text:*.txt'
Note that you need to redirect wget(1)'s output from standard error to standard output in order to pipe it to httpindex.
EXIT STATUS
Exits with a value of zero only if indexing completed sucessfully; non-zero otherwise.
CAVEATS
In addition to those for index++(1), httpindex does not correctly handle the use of multiple -e, -E, -m, or -M options (because the Perl
script uses the standard GetOpt::Std package for processing command-line options that doesn't). The last of any of those options ``wins.''
The work-around is to use multiple values for those options seperated by commas to a single one of those options. For example, if you want
to do:
httpindex -e'html:*.html' -e'text:*.txt'
do this instead:
httpindex -e'html:*.html,text:*.txt'
SEE ALSO
index++(1), wget(1), WWW(3)AUTHOR
Paul J. Lucas <pauljlucas@mac.com>
SWISH++ August 2, 2005 httpindex(1)