Sponsored Content
Top Forums Shell Programming and Scripting Performance issue in Grepping large files Post 302819983 by drl on Tuesday 11th of June 2013 08:05:32 PM
Old 06-11-2013
Hi.

For multiple CPU hardware, introducing some parallelism might be useful in decreasing real time.

I think I would split the list of files to be examined into several pieces and then run whatever scanning program is desired on each piece.

There are several utilities to help with the control of simultaneous processes: xargs, parallel, etc.

Best wishes ... cheers, drl
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Unix File System performance with large directories

Hi, how does the Unix File System perform with large directories (containing ~30.000 files)? What kind of structure is used for the organization of a directory's content, linear lists, (binary) trees? I hope the description 'Unix File System' is exact enough, I don't know more about the file... (3 Replies)
Discussion started by: dive
3 Replies

2. Shell Programming and Scripting

Grepping issue..

I found another problem with my disk-adding script today. When looking for disks, I use grep. When I grep for the following disk sizes: 5242880 I also pick up these as well: 524288000 How do I specifically pick out one or the other, using grep, without resorting to the -v option? ... (9 Replies)
Discussion started by: LinuxRacr
9 Replies

3. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

4. Shell Programming and Scripting

replace issue with large files

I have the following problem: I have two files: S containing sentences (one in each row) and W containing files (one in each row). It might look like this: S: a b c apple d. e f orange g. h banana i j. W: orange banana apple My task is to replace in S all words that appear in W... (2 Replies)
Discussion started by: tootles564
2 Replies

5. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

6. Red Hat

Empty directory, large size and performance

Hi, I've some directory that I used as working directory for a program. At the end of the procedure, the content is deleted. This directory, when I do a ls -l, appears to still take up some space. After a little research, I've seen on a another board of this forum that it's not really taking... (5 Replies)
Discussion started by: bdx
5 Replies

7. Shell Programming and Scripting

Grepping large list of files

Hi All, I need help to know the exact command when I grep large list of files. Either using ls or find command. However I do not want to find in the subdirectories as the number of subdirectories are not fixed. How do I achieve that. I want something like this: find ./ -name "MYFILE*.txt"... (2 Replies)
Discussion started by: angshuman
2 Replies

8. Shell Programming and Scripting

Grepping verbal forms from a large corpus

I want to extract verbal forms from a large corpus of English. I have identified a certain number of patterns. Each pattern has the following structure SPACE word_CATEGORY where word refers to the verbal form and CATEGORY refers to the class of the verb The categories are identified as per the... (4 Replies)
Discussion started by: gimley
4 Replies

9. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
STARTPAR(8)						      System Manager's Manual						       STARTPAR(8)

NAME
startpar - start runlevel scripts in parallel SYNOPSIS
startpar [-p par] [-i iorate] [-t timeout] [-T global_timeout] [-a arg] prg1 prg2 ... startpar [-p par] [-i iorate] [-t timeout] [-T global_timeout] -M [ boot|start|stop] DESCRIPTION
startpar is used to run multiple run-level scripts in parallel. The degree of parallelism on one CPU can be set with the -p option, the default is full parallelism. An argument to all of the scripts can be provided with the -a option. Processes blocked by pending I/O will cause new process creation to be weighted by the iorate factor 800. To change this factor the option -i can be used to specify another value. The amount weight=(nblockedxiorate)/1000 will be subtracted from the total number of processes which could be started, where nblocked is the number of processes currently blocked by pending I/O. The output of each script is buffered and written when the script exits, so output lines of different scripts won't mix. You can modify this behaviour by setting a timeout. The timeout set with the -t option is used as buffer timeout. If the output buffer of a script is not empty and the last output was timeout seconds ago, startpar will flush the buffer. The -T option timeout works more globally. If no output is printed for more than global_timeout seconds, startpar will flush the buffer of the script with the oldest output. Afterwards it will only print output of this script until it is finished. The -M option switches startpar into a make(1) like behaviour. This option takes three different arguments: boot, start, and stop for reading .depend.boot or .depend.start or .depend.stop respectively in the directory /etc/init.d/. By scanning the boot and runlevel direc- tories in /etc/init.d/ it then executes the appropriate scripts in parallel. FILES
/etc/init.d/.depend.boot /etc/init.d/.depend.start /etc/init.d/.depend.stop SEE ALSO
init(8) insserv(8). COPYRIGHT
2003,2004 SuSE Linux AG, Nuernberg, Germany. 2007 SuSE LINUX Products GmbH, Nuernberg, Germany. AUTHOR
Michael Schroeder <mls@suse.de> Takashi Iwai <tiwai@suse.de> Werner Fink <werner@suse.de> Jun 2003 STARTPAR(8)
All times are GMT -4. The time now is 12:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy