Sponsored Content
Top Forums Shell Programming and Scripting Improve script - slow process with big files Post 302990411 by jiam912 on Thursday 26th of January 2017 06:33:58 AM
Old 01-26-2017
Hi RudiC,

Sometimes I have files with more of 20000 lines in this case it takes long time to end the job.
Kindly, could you please help me to improve it.
Appreciate your help.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

looking for solution to improve process replicate files to remote loc.

looking for solution to replicate 1.5GB files to a remote location... Currently, this process looks like the following: move 1.5GB files into a staging area. compress files. rsync files to remote server. remove compressed files. I have performed some timings, and compress seems more... (5 Replies)
Discussion started by: mr_manny
5 Replies

2. Shell Programming and Scripting

bash script working for small size files but not for big size files.

Hi, I have one file stat. Stat file contents are as follows: for example. H50768020040913,00260100,507680,13,0000000643,0000000643,00000,0000 H50769520040808,00260100,507695,13,0000000000,0000000000,00000,0000 H50770620040611,00260100,507706,13,0000000000,0000000000,00000,0000 Now i... (1 Reply)
Discussion started by: davidpreml
1 Replies

3. AIX

How to send big files over slow network?

Hi, I am trying to send oracle archives over WAN and it is taking hell a lot of time. To reduce the time, I tried to gzip the files and send over to the other side. That seems to reduce the time. Does anybody have experienced this kind of problem and any possible ways to reduce the time. ... (1 Reply)
Discussion started by: giribt
1 Replies

4. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
Discussion started by: hidnana
7 Replies

5. UNIX for Advanced & Expert Users

sed working slow on big files

HI Experts , I'm using the following code to remove spaces appearing at the end of the file. sed "s/*$//g" <filename> > <new_filename> mv <new_filename> <filename> this is working fine for volumes upto 20-25 GB. for the bigger files it is taking more time that it is required... (5 Replies)
Discussion started by: sumoka
5 Replies

6. Shell Programming and Scripting

Very big text file - Too slow!

Hello everyone, suppose there is a very big text file (>800 mb) that each line contains an article from wikipedia. Each article begins with a tag (<..>) containing its url. Currently there are 10^6 articles in the file. I want to take random N articles, eliminate all non-alpharithmetic... (14 Replies)
Discussion started by: fedonMan
14 Replies

7. UNIX for Dummies Questions & Answers

How do I slow down a process?

Hello, I've been searching for something that slows down a process for some time now. Slow down as in make time pass by slower. I have rarely turned to asking a forum in the past but at this point I've given up. For example: if I made a program that would print "Hello" in 5 seconds, I would use... (18 Replies)
Discussion started by: Nathan1
18 Replies

8. HP-UX

Script execution is very slow when trying to find all files and their owners on HP-UX box

Hi, I have a HP-UX server were I need to list all the files in the entire file system, their directory path, last modified date, owner and group. I do not need to search the file contents. I created the script given below and I am excluding directories and files of type tmp, temp and log. The... (4 Replies)
Discussion started by: Adyan Faruqi
4 Replies

9. Solaris

Rsync quite slow (using very little cpu): how to improve its speed?

I have "inherited" a OmniOS (illumos based) server. I noticed rsync is significantly slower in respect to my reference, FreeBSD 12-CURRENT, running on exactly same hardware. Using same hardware, same command with same source and target disks, OmniOS r151026 gives: test@omniosce:~# time... (11 Replies)
Discussion started by: priyadarshan
11 Replies

10. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
SGE_REQUEST(5)						   Sun Grid Engine File Formats 					    SGE_REQUEST(5)

NAME
sge_request - Sun Grid Engine default request definition file format DESCRIPTION
sge_request reflects the format of the files to define default request profiles. If available, default request files are read and processed during job submission before any submit options embedded in the job script and before any options in the qsub(1) or qsh(1) command-line are considered. Thus, the command-line and embedded script options may overwrite the settings in the default request files (see qsub(1) or qsh(1) for details). There is a cluster global, a user private and a working directory local default request definition file. The working directory local default request file has the highest precedence and is followed by the user private and then the cluster global default request file. Note, that the -clear option to qsub(1) or qsh(1) can be used to discard any previous settings at any time in a default request file, in the embedded script flags or in a qsub(1) or qsh(1) command-line option. The format of the default request definition files is: o The default request files may contain an arbitrary number of lines. Blank lines and lines with a '#' sign in the first column are skipped. o Each line not to be skipped may contain any qsub(1) option as described in the Sun Grid Engine Reference Manual. More than one option per line is allowed. The batch script file and argument options to the batch script are not considered as qsub(1) options and thus are not allowed in a default request file. EXAMPLES
The following is a simple example of a default request definition file: ===================================================== # Default Requests File # request group to be sun4 and a CPU-time of 5hr -l arch=sun4,s_cpu=5:0:0 # don't restart the job in case of system crashes -r n ===================================================== Having defined a default request definition file like this and submitting a job as follows: qsub test.sh would have precisely the same effect as if the job was submitted with: qsub -l arch=sun4,s_cpu=5:0:0 -r n test.sh FILES
<sge_root>/<cell>/common/sge_request global defaults file $HOME/.sge_request user private defaults file $cwd/.sge_request cwd directory defaults file SEE ALSO
sge_intro(1), qsh(1), qsub(1), Sun Grid Engine Installation and Administration Guide COPYRIGHT
See sge_intro(1) for a full statement of rights and permissions. SGE 6.2u5 $Date$ SGE_REQUEST(5)
All times are GMT -4. The time now is 09:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy