06-23-2016
Does performance actually get worse? Or does it just take 100,000x longer to process a 100,000x larger file? About how many matches are you expecting?
There are memory-heavy ways to do it faster, but they're not really applicable to massive files. You could try divide-and-conquer: Run as many simultaneously as your CPU and disks can easily handle, sort their output individually, then merge them in one final step.
This User Gave Thanks to Corona688 For This Post:
8 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Dear unix users,
Any comments on this matter or not? I've been facing this kind of problem for very long. Thinking how to make my script move faster instead of using C to write it. I still prefer using Shell script.
Normally a FOR loop will make the script loop for very long time, what to do... (1 Reply)
Discussion started by: clemeot
1 Replies
2. SCO
I have been a SCO UNIX user, never an administrator...so I am stumbling around looking for information.
I don't know too much about what is onboard in terms of hardware, however; I will try my best.
We have SCO 5.07 and have applied MP5.
We have a quad core processor with 4 250 GB... (1 Reply)
Discussion started by: atpbrownie
1 Replies
3. Shell Programming and Scripting
Hello I have some nested loop to display files and form menu item.
The part of the code is below.
I found that after runnining the script for a while the display becomes very slow.
Does ksh shell provided any easy way to release variables, reinit, etc.
while ];do
script=0
... (2 Replies)
Discussion started by: ekb
2 Replies
4. Shell Programming and Scripting
Dear All,
I am using the following script to find and replace the date format in a file. The field18 in the file has the following format: "01/26/2010 11:55:14 GMT+04:00" which I want to convert into the following format "20100126115514" for this purpose I am using the following lines of codes:... (5 Replies)
Discussion started by: bilalghazi
5 Replies
5. Shell Programming and Scripting
Hello everybody,
I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel.
My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies
6. Shell Programming and Scripting
Hello,
Hope you are doing fine. Let me describe the problem, I have a script that calls another script K2Test.sh, this script K2Test.sh (created by another team) takes date as argument and generates approx 1365 files in localcurves directory for given date.
Out of these 1365 I am only... (11 Replies)
Discussion started by: srattani
11 Replies
7. Shell Programming and Scripting
Hi,
I want to read folders which do not have a file "processing" in a for loop ordered by timestamp.
Currently im doing like this.
Like
cd /home/working
for i in `ls -c1`
do
some command...
done
I want to exclude folders which have that "processing" file.
The directory... (2 Replies)
Discussion started by: chetan.c
2 Replies
8. Programming
Hello,
I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this :
This is the output of ls command : I stored the output in a file filelist
1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies