I have a bash script that will take approx. 130 days to complete. I am trying to grep a list of 1,144 user ID's out of 41 (1 GB each) files. The 41 files were originally one 41 G file, but that was horrendously too slow.
This is my current file:
Code:
#!/bin/bash
for i in `cat WashFD.txt` # 1,144 files
do
for b in `cat xfiles` # 41 "x??" files
do
echo "looking for " $i "in " $b
cat $b | grep -i $i >> SEID.searches
done
done
Currently, I am processing one of the 41 files every 4 minutes. 4 x 41 = 164 min.
164 / 60 (min/hour) = 2.73 hours per user_id. I have 1,144 user_id's multiplied by 2.73 = 3123.12 hours. 3123.12 / 24 (hours in a day) = 130.13 days.
As you can see, that is way too long to process this task. I don't know PERL but I've heard its faster. If anyone has any suggestions please let me know.
Last edited by vbe; 01-29-2010 at 10:54 AM..
Reason: code tags please
Hello experts,
I have a series issue in script that result with bad peformence and I wonder if you can assist me.
For example
I have two files:
File-New, size 15Mb.
File-Old, size 1Mb.
File-New content:
a
b
c
k
File-Old content:
d
f
a
b (0 Replies)
Hi,
I am running a BASH shell with the following script. The script works and gives me correct output but is very slow with large files. The more rows and columns (width and height) the slower as you can probably see.
How can I do what I want more efficiently? Any ideas welcome. It has been... (10 Replies)
I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file
STRUCTURE OF THE TWO FILES
The first file is a series of names
ramesh
sushil
jonga
sudesh
lugdi
whereas the second file (could be... (4 Replies)
My script builds a lot of these array lists, then compares their sizes which solves my problem, but runs very slow. :(
set -A comboSorted -- $(
for x in ${IDs}
do
nawk -v s=$x '
BEGIN { testPattern="^" s "$" }
{
if ( $2 ~ testPattern ) {
getline;getline;
if ($1 == "IMAGE_SIZE")
print... (1 Reply)
Basically my requirement is to know the total number of free anonymous ports.
anonymous port range is 32768- 65535.
i wrote a script for that
**********************************************
for i in {32768..65535}
do
netstat -an | grep $i > /dev/null
if
... (21 Replies)
Hi
I know the basic about script and sleep processes.
However this is more tricky:
I would like to run
sh -x /sbin/dhclient-script and slow down the output of the script as a whole.
How would you do it?
I would like to delay output on the screen with 1 second for every line for the output... (3 Replies)
I have shell program as below
#!/bin/sh
echo ======= LogManageri start ==========
#This directory is getting the raw data from remote server
Raw_data=/opt/ftplogs
# This directory is ready for process the data
Processing_dir=/opt/processing_dir
# This directory is prcoessed files and... (4 Replies)
Hello,
I am showing the start of my script.
I am finding that 'xmessage' is taking about 12-15 seconds to show.
This in a terminal is very quick '/opt/vc/bin/vcgencmd get_camera'.
Is there any way to get 'camera not detected' to show faster.
Regards
#!/bin/bash
s=$(/opt/vc/bin/vcgencmd... (4 Replies)
I'm new to utilities like socat and netcat and I'm not clear if they will do what I need.
I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes.
I'm using Cygwin bash on Windows 10.
My... (3 Replies)
Discussion started by: siegfried
3 Replies
LEARN ABOUT V7
at
AT(1) General Commands Manual AT(1)NAME
at - execute commands at a later time
SYNOPSIS
at time [ day ] [ file ]
DESCRIPTION
At squirrels away a copy of the named file (standard input default) to be used as input to sh(1) at a specified later time. A cd(1) com-
mand to the current directory is inserted at the beginning, followed by assignments to all environment variables. When the script is run,
it uses the user and group ID of the creator of the copy file.
The time is 1 to 4 digits, with an optional following `A', `P', `N' or `M' for AM, PM, noon or midnight. One and two digit numbers are
taken to be hours, three and four digits to be hours and minutes. If no letters follow the digits, a 24 hour clock time is understood.
The optional day is either (1) a month name followed by a day number, or (2) a day of the week; if the word `week' follows invocation is
moved seven days further off. Names of months and days may be recognizably truncated. Examples of legitimate commands are
at 8am jan 24
at 1530 fr week
At programs are executed by periodic execution of the command /usr/lib/atrun from cron(8). The granularity of at depends upon how often
atrun is executed.
Standard output or error output is lost unless redirected.
FILES
/usr/spool/at/yy.ddd.hhhh.uu
activity to be performed at hour hhhh of year day ddd of year yy. uu is a unique number.
/usr/spool/at/lasttimedone contains hhhh for last hour of activity.
/usr/spool/at/past directory of activities now in progress
/usr/lib/atrun program that executes activities that are due
pwd(1)SEE ALSO calendar(1), cron(8)DIAGNOSTICS
Complains about various syntax errors and times out of range.
BUGS
Due to the granularity of the execution of /usr/lib/atrun, there may be bugs in scheduling things almost exactly 24 hours into the future.
AT(1)
01-29-2010
