Sponsored Content
Top Forums Shell Programming and Scripting Incredibly inefficient cat | grep script Post 302913044 by achenle on Wednesday 13th of August 2014 01:01:35 PM
Old 08-13-2014
What's the exact contents of each file?

Right now, you're literally running millions of greps, comparing each ID to every other ID. That's a huge waste of time, obviously. You only really need to compare each ID to itself.

Because it seems to me that you're just looking for IDs that are in both files. If each ID only occurs once in each file, pull the IDs out of each file, combine them into one file, sort them, and only count IDs that are duplicated in the combined file.

Or you can figure out how to use the "join" utility, though that may not be any faster than what you're already doing, although it does have the huge advantage of not having to fork() and exec() a new process millions of times.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

cat and grep not working

I am trying to cat a file and then grep that file for a number. I can do it fine on other files but this particular file will not do anything. I tried running it on an older file from the same device but it is just not working. The file is nothing more than a flat file on a unix box. Here is just a... (3 Replies)
Discussion started by: jphess
3 Replies

2. Shell Programming and Scripting

cat in the command line doesn't match cat in the script

Hello, So I sorted my file as I was supposed to: sort -n -r -k 2 -k 1 file1 | uniq > file2 and when I wrote > cat file2 in the command line, I got what I was expecting, but in the script itself ... sort -n -r -k 2 -k 1 averages | uniq > temp cat file2 It wrote a whole... (21 Replies)
Discussion started by: shira
21 Replies

3. Shell Programming and Scripting

Perl sum really inefficient!!

Hi all, I have a file like the following: ID, 2,Andrew,0,1,2,3,4,2,5,6,7,7,9,3,4,5,34,3,2,1,5,6,78,89,8,7,6...................... 4,James,0,6,7,0,5,6,4,7,8,9,6,46,6,3,2,5,6,87,0,341,0,5,2,5,6.................... END, (there are more entires on each line but to keep it simple I've left... (10 Replies)
Discussion started by: Donkey25
10 Replies

4. Shell Programming and Scripting

Problem with IF - CAT - GREP in simple shell script

Hi all, Here is my requirement I have to search 'ORA' word in out.log file,if it is present then i need to send that file (out.log) content to some mail id.If 'ORA' word is not in that file then i need to send 'load succesful' message to some mail id. The below the shell script is not... (5 Replies)
Discussion started by: mak_boop
5 Replies

5. Shell Programming and Scripting

cat /etc/passwd and grep -v on /etc/shells

Hi All, I'd like to do this cat /etc/passwd and grep -v on the /etc/shells list I'd like to find all shell that doesn't exist on the /etc/passwd. Is there an easy way without doing a egrep -v "/bin/sh|/bin/bash................"? How do I use a file /etc/shells as my list for... (4 Replies)
Discussion started by: itik
4 Replies

6. Shell Programming and Scripting

cat -n and grep

I am not sure if using cat -n is the most efficient way to split a file into multiple files, one file per line in the source file. I thought using cat -n would make it easy to process the file because it produces an output that numbers each line that I could then grep for with the regex "^ *$i".... (3 Replies)
Discussion started by: kapu
3 Replies

7. Shell Programming and Scripting

grep or cat using sed

Is there a way using grep or cat a file to create a new file based on whether the first 9 positions of each record is less than 399999999? This is a fixed file format. (3 Replies)
Discussion started by: ski
3 Replies

8. UNIX for Dummies Questions & Answers

Grep and cat combined

Hello, i need to search one word (snp1) from many files and copy the content of the columns of this word in new file. example: file 1: SNP BP CHR P snp1 1 3 0.01 snp2 2 2 0.05 . . file 2: SNP BP CHR P snp1 1 3 0.06 snp2 2 2 0.3 output... (6 Replies)
Discussion started by: biopsy
6 Replies

9. Shell Programming and Scripting

Replace cat and grep with <

Hello someone told me to use OS=`awk '{print int($3)}' < /etc/redhat-release` instead of OS=cat /etc/redhat-release | `awk '{print int($3)}'` any idea for the reason ? (5 Replies)
Discussion started by: nimafire
5 Replies

10. UNIX for Dummies Questions & Answers

Grep or cat The Whole Directory PROBLEMS :(

Hi Guys This is my first post so I am not sure how things go here. I'm sorry if I'm breaking the rule or something. Feel free to correct me about that :) So as I was saying... I'd been trying to grep this folder containing 900,000 txt files but seems no luck. I get either "No such file... (6 Replies)
Discussion started by: Nexeu
6 Replies
wait(1)                                                            User Commands                                                           wait(1)

NAME
wait - await process completion SYNOPSIS
/bin/sh wait [pid...] /bin/jsh /bin/ksh /usr/xpg4/bin/sh wait [pid...] wait [ % jobid...] /bin/csh wait DESCRIPTION
The shell itself executes wait, without creating a new process. If you get the error message cannot fork,too many processes, try using the wait command to clean up your background processes. If this doesn't help, the system process table is probably full or you have too many active foreground processes. There is a limit to the number of process IDs associated with your login, and to the number the system can keep track of. Not all the processes of a pipeline with three or more stages are children of the shell, and thus cannot be waited for. /bin/sh, /bin/jsh Wait for your background process whose process ID is pid and report its termination status. If pid is omitted, all your shell's currently active background processes are waited for and the return code will be 0. The wait utility accepts a job identifier, when Job Control is enabled (jsh), and the argument, jobid, is preceded by a percent sign (%). If pid is not an active process ID, the wait utility will return immediately and the return code will be 0. csh Wait for your background processes. ksh When an asynchronous list is started by the shell, the process ID of the last command in each element of the asynchronous list becomes known in the current shell execution environment. If the wait utility is invoked with no operands, it will wait until all process IDs known to the invoking shell have terminated and exit with an exit status of 0. If one or more pid or jobid operands are specified that represent known process IDs (or jobids), the wait utility will wait until all of them have terminated. If one or more pid or jobid operands are specified that represent unknown process IDs (or jobids), wait will treat them as if they were known process IDs (or jobids) that exited with exit status 127. The exit status returned by the wait utility will be the exit status of the process requested by the last pid or jobid operand. The known process IDs are applicable only for invocations of wait in the current shell execution environment. OPERANDS
The following operands are supported: One of the following: pid The unsigned decimal integer process ID of a command, for which the utility is to wait for the termination. jobid A job control job ID that identifies a background process group to be waited for. The job control job ID notation is applicable only for invocations of wait in the current shell execution environment, and only on systems supporting the job control option. USAGE
On most implementations, wait is a shell built-in. If it is called in a subshell or separate utility execution environment, such as one of the following, (wait) nohup wait ... find . -exec wait ... ; it will return immediately because there will be no known process IDs to wait for in those environments. EXAMPLES
Example 1: Using A Script To Identify The Termination Signal Although the exact value used when a process is terminated by a signal is unspecified, if it is known that a signal terminated a process, a script can still reliably figure out which signal is using kill, as shown by the following (/bin/ksh and /usr/xpg4/bin/sh): sleep 1000& pid=$! kill -kill $pid wait $pid echo $pid was terminated by a SIG$(kill -l $(($?-128))) signal. Example 2: Returning The Exit Status Of A Process If the following sequence of commands is run in less than 31 seconds (/bin/ksh and /usr/xpg4/bin/sh): sleep 257 | sleep 31 & jobs -l %% then either of the following commands will return the exit status of the second sleep in the pipeline: wait <pid of sleep 31> wait %% ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of wait: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
csh(1), jobs(1), ksh(1), sh(1), attributes(5), environ(5), standards(5) SunOS 5.10 12 Dec 1997 wait(1)
All times are GMT -4. The time now is 05:22 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy