I have a bash script that will take approx. 130 days to complete. I am trying to grep a list of 1,144 user ID's out of 41 (1 GB each) files. The 41 files were originally one 41 G file, but that was horrendously too slow.
This is my current file:
Code:
#!/bin/bash
for i in `cat WashFD.txt` # 1,144 files
do
for b in `cat xfiles` # 41 "x??" files
do
echo "looking for " $i "in " $b
cat $b | grep -i $i >> SEID.searches
done
done
Currently, I am processing one of the 41 files every 4 minutes. 4 x 41 = 164 min.
164 / 60 (min/hour) = 2.73 hours per user_id. I have 1,144 user_id's multiplied by 2.73 = 3123.12 hours. 3123.12 / 24 (hours in a day) = 130.13 days.
As you can see, that is way too long to process this task. I don't know PERL but I've heard its faster. If anyone has any suggestions please let me know.
Last edited by vbe; 01-29-2010 at 10:54 AM..
Reason: code tags please
Hello experts,
I have a series issue in script that result with bad peformence and I wonder if you can assist me.
For example
I have two files:
File-New, size 15Mb.
File-Old, size 1Mb.
File-New content:
a
b
c
k
File-Old content:
d
f
a
b (0 Replies)
Hi,
I am running a BASH shell with the following script. The script works and gives me correct output but is very slow with large files. The more rows and columns (width and height) the slower as you can probably see.
How can I do what I want more efficiently? Any ideas welcome. It has been... (10 Replies)
I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file
STRUCTURE OF THE TWO FILES
The first file is a series of names
ramesh
sushil
jonga
sudesh
lugdi
whereas the second file (could be... (4 Replies)
My script builds a lot of these array lists, then compares their sizes which solves my problem, but runs very slow. :(
set -A comboSorted -- $(
for x in ${IDs}
do
nawk -v s=$x '
BEGIN { testPattern="^" s "$" }
{
if ( $2 ~ testPattern ) {
getline;getline;
if ($1 == "IMAGE_SIZE")
print... (1 Reply)
Basically my requirement is to know the total number of free anonymous ports.
anonymous port range is 32768- 65535.
i wrote a script for that
**********************************************
for i in {32768..65535}
do
netstat -an | grep $i > /dev/null
if
... (21 Replies)
Hi
I know the basic about script and sleep processes.
However this is more tricky:
I would like to run
sh -x /sbin/dhclient-script and slow down the output of the script as a whole.
How would you do it?
I would like to delay output on the screen with 1 second for every line for the output... (3 Replies)
I have shell program as below
#!/bin/sh
echo ======= LogManageri start ==========
#This directory is getting the raw data from remote server
Raw_data=/opt/ftplogs
# This directory is ready for process the data
Processing_dir=/opt/processing_dir
# This directory is prcoessed files and... (4 Replies)
Hello,
I am showing the start of my script.
I am finding that 'xmessage' is taking about 12-15 seconds to show.
This in a terminal is very quick '/opt/vc/bin/vcgencmd get_camera'.
Is there any way to get 'camera not detected' to show faster.
Regards
#!/bin/bash
s=$(/opt/vc/bin/vcgencmd... (4 Replies)
I'm new to utilities like socat and netcat and I'm not clear if they will do what I need.
I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes.
I'm using Cygwin bash on Windows 10.
My... (3 Replies)
Discussion started by: siegfried
3 Replies
LEARN ABOUT DEBIAN
condor_userlog
condor_userlog(1) General Commands Manual condor_userlog(1)Name
condor_userlog Display - and summarize job statistics from job log files.
Synopsis
condor_userlog [-help] [-total | -raw] [-debug] [-evict] [-j cluster | cluster.proc] [-all] [-hostname] logfile ...
Description
condor_userlog parses the information in job log files and displays summaries for each workstation allocation and for each job. See the
manual page for condor_submiton page for instructions for specifying that Condor write a log file for your jobs.
If -totalis not specified, condor_userlogwill first display a record for each workstation allocation, which includes the following informa-
tion:
Job
The cluster/process id of the Condor job.
Host
The host where the job ran. By default, the host's IP address is displayed. If -hostnameis specified, the host name will be displayed
instead.
Start Time
The time (month/day hour:minute) when the job began running on the host.
Evict Time
The time (month/day hour:minute) when the job was evicted from the host.
Wall Time
The time (days+hours:minutes) for which this workstation was allocated to the job.
Good Time
The allocated time (days+hours:min) which contributed to the completion of this job. If the job exited during the allocation, then this
value will equal ``Wall Time.'' If the job performed a checkpoint, then the value equals the work saved in the checkpoint during this
allocation. If the job did not exit or perform a checkpoint during this allocation, the value will be 0+00:00. This value can be greater
than 0 and less than ``Wall Time'' if the application completed a periodic checkpoint during the allocation but failed to checkpoint
when evicted.
CPU Usage
The CPU time (days+hours:min) which contributed to the completion of this job.
condor_userlogwill then display summary statistics per host:
Host/Job
The IP address or host name for the host.
Wall Time
The workstation time (days+hours:minutes) allocated by this host to the jobs specified in the query. By default, all jobs in the log are
included in the query.
Good Time
The time (days+hours:minutes) allocated on this host which contributed to the completion of the jobs specified in the query.
CPU Usage
The CPU time (days+hours:minutes) obtained from this host which contributed to the completion of the jobs specified in the query.
Avg Alloc
The average length of an allocation on this host (days+hours:minutes).
Avg Lost
The average amount of work lost (days+hours:minutes) when a job was evicted from this host without successfully performing a checkpoint.
Goodput
This percentage is computed as Good Time divided by Wall Time.
Util.
This percentage is computed as CPU Usage divided by Good Time.
condor_userlogwill then display summary statistics per job:
Host/Job
The cluster/process id of the Condor job.
Wall Time
The total workstation time (days+hours:minutes) allocated to this job.
Good Time
The total time (days+hours:minutes) allocated to this job which contributed to the job's completion.
CPU Usage
The total CPU time (days+hours:minutes) which contributed to this job's completion.
Avg Alloc
The average length of a workstation allocation obtained by this job in minutes (days+hours:minutes).
Avg Lost
The average amount of work lost (days+hours:minutes) when this job was evicted from a host without successfully performing a checkpoint.
Goodput
This percentage is computed as Good Time divided by Wall Time.
Util.
This percentage is computed as CPU Usage divided by Good Time.
Finally, condor_userlogwill display a summary for all hosts and jobs.
Options-help
Get a brief description of the supported options
-total
Only display job totals
-raw
Display raw data only
-debug
Debug mode
-j
Select a specific cluster or cluster.proc
-evict
Select only allocations which ended due to eviction
-all
Select all clusters and all allocations
-hostname
Display host name instead of IP address
General Remarks
Since the Condor job log file format does not contain a year field in the timestamp, all entries are assumed to occur in the current year.
Allocations which begin in one year and end in the next will be silently ignored.
Exit Status
condor_userlogwill exit with a status value of 0 (zero) upon success, and it will exit with the value 1 (one) upon failure.
Author
Condor Team, University of Wisconsin-Madison
Copyright
Copyright (C) 1990-2012 Condor Team, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI. All Rights Reserved.
Licensed under the Apache License, Version 2.0.
See the Condor Version 7.8.2 Manualor http://www.condorproject.org/licensefor additional notices. condor-admin@cs.wisc.edu
September 2012 condor_userlog(1)
01-29-2010
