What about sorting a 5G file? Post: 302578846

Sponsored Content

Operating Systems Solaris What about sorting a 5G file? Post 302578846 by plmachiavel on Friday 2nd of December 2011 01:14:22 PM

12-02-2011

Registered User

Hi Corona,

Thanks for your answer. Before posting I always read the mans, I ask quesitons when I'm looking for a different way to make things and to try to understand a little bit the mechanisms behind some of the bins.

By looking at the process while it was doing the sort I noticed that it divides the file in multiple chunks (11megs in my case) then makes a new file out of the 64 first "chunks" it created and delete those first smaller "chunks". This will then generate 9 files (approximately) then generate a finale one from those last 9 files. In other words, we need approximately 1.50 to 1.75 x File_size of disk space to be able to complete that sort.

Because our swap device is a disk, I don't think it will actually slow down the process if I tell it to do something like :

Code:

sort -S 80% file > destination

So it'll be as fast as just writing to the disk like it is currently doing. Yes, it'll slow other processes but we don't really care at the moment because this job is necessary for any other job to start.

I realize that the question I asked is not really the question I wanted to ask. By asking a more efficient way, I was more thinking "a fastest way".

Any idea on how to "split" or "fork" sort on multiple cores? We have 8 available on the oldish v490 and I was hoping to use maybe 4 to 6 cores to do it.

If not, I guess that redirecting the sort to our SAN storage will be the most efficient way.

Thanks again Corona for your time, really appreciated.

plmachiavel

View Public Profile for plmachiavel

Find all posts by plmachiavel

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help sorting file.

Hi, I have this file (filex) 07-11-2003 10:11:12!cccc!ddd!eeeeeeee 07-11-2003 09:11:11!dddd!kkkkk!xxxxxx 09-12-2003 14:18:43!aaaa!bbbbb!cccc where I need to sort it by date+time in this order: 09-12-2003 14:18:43!aaaa!bbbbb!cccc 07-11-2003 10:11:12!cccc!ddd!eeeeeeee 07-11-2003...

2. Shell Programming and Scripting

sorting file

hi everyone, i have a document where i have email addresess and names, i need to check if the email addresses are uniq, if they repeat erase one of them, how can i do that? document sample: aD00763357@cucei.udg.mx,ABRAHAM ANTONIO SEVERIANO a199721111@cucei.udg.mx,ABRAHAM GONZALEZ...

3. Shell Programming and Scripting

Sorting a file

hello all, I have a file with two numbers on each line, comma separated. I want to sort the contents of the file in increasing order (smallest to largest) of the numbers on the second line. i.e: 23,3 25,2 27,12 to become 25,2 23,3 27,12 Does anyone know how I can do this? Thanks...

4. Programming

regarding file sorting

i ahve a file like: ************************************* sree 122132 12321 *********************************** phani 21321 3213214 ****************************** dddsds 213213123 23213213 ******************************* i want to sort the file with respect to name how we can do this...

5. Shell Programming and Scripting

file sorting

i have a data in afile like this ************************************** sree sree@yahoo.com 98662323432 ************************************* phani phani@yahoo.com 98662323344 ************************************* i want to sort the file with respect to name. how can i do this. thank...

6. Shell Programming and Scripting

Sorting file

I have the file as follow: A: 60 B: 80 C: 40 D: 11 E: 100 I want to sort the file and get the output to file as follow: E: 100 B: 80 A: 60 C: 40 D: 11 Could any one help me please?

7. Shell Programming and Scripting

Finding & Moving Oldest File by Parsing/Sorting Date Info in File Names

I'm trying to write a script that will look in an /exports folder for the oldest export file and move it to a /staging folder. "Oldest" in this case is actually determined by date information embedded in the file names themselves. Also, the script should only move a file from /exports to...

8. UNIX for Dummies Questions & Answers

sorting s file

how would i sort a file on the third column based on numerical value instead of the ASCII order?

9. UNIX for Dummies Questions & Answers

Sorting data in file based on field in another file

Hi, I have two files, one of which I would like to sort based on the order of the data in the second. I would like to do this using a simple unix statement. My two files as follows: File 1: 12345 1 2 2 2 0 0 12349 0 0 2 2 1 2 12350 1 2 1 2 2 2 . . . File2: 12350...

10. Shell Programming and Scripting

Need help in Sorting a file

Hi Unix Admins, I wanted to sort a file in a specific order, i.e the input file contains two fields and the first column is not unique and had to be sorted. example Input File ------- 2014-10-21:Rand1 2014-11-02:Rand2 2014-11-02:Rand3 2014-11-02:Rand4 2014-11-03:Rand5 2014-11-04:Rand6...

LEARN ABOUT LINUX

gai.conf

GAI.CONF(5)						     Linux Programmer's Manual						       GAI.CONF(5)

NAME

       gai.conf - getaddrinfo(3) configuration file

DESCRIPTION

       A  call	to  getaddrinfo(3)  might return multiple answers.  According to RFC 3484 these answers must be sorted so that the answer with the
       highest success rate is first in the list.  The RFC provides an algorithm for the sorting.  The	static	rules  are  not  always  adequate,
       though.	 For  this reason, the RFC also requires that system administrators should have the possibility to dynamically change the sorting.
       For the glibc implementation, this can be achieved with the /etc/gai.conf file.

       Each line in the configuration file consists of a keyword and its parameters.  White spaces in any place are ignored.  Lines starting  with
       '#' are comments and are ignored.

       The keywords currently recognized are:

       label netmask precedence
	      The  value  is added to the label table used in the RFC 3484 sorting.  If any label definition is present in the configuration file,
	      the default table is not used.  All the label definitions of the default table which are to be maintained  have  to  be  duplicated.
	      Following the keyword, the line has to contain a network mask and a precedence value.

       precedence netmask precedence
	      This  keyword is similar to label, but instead the value is added to the precedence table as specified in RFC 3484.  Once again, the
	      presence of a single precedence line in the configuration file causes the default table to not be used.

       reload <yes|no>
	      This keyword controls whether a process checks whether the configuration file has been changed since the last time it was read.	If
	      the  value  is "yes", the file is reread.  This might cause problems in multithreaded applications and is generally a bad idea.  The
	      default is "no".

       scopev4 mask value
	      Add another rule to the RFC 3484 scope table for IPv4 address.  By default, the scope IDs described in section 3.2 in  RFC 3438  are
	      used.  Changing these defaults should hardly ever be necessary.

FILES

       /etc/gai.conf

VERSIONS

       The gai.conf file is supported by glibc since version 2.5.

EXAMPLE

       The default table according to RFC 3484 would be specified with the following configuration file:

	   label  ::1/128	0
	   label  ::/0		1
	   label  2002::/16	2
	   label ::/96		3
	   label ::ffff:0:0/96	4
	   precedence  ::1/128	     50
	   precedence  ::/0	     40
	   precedence  2002::/16     30
	   precedence ::/96	     20
	   precedence ::ffff:0:0/96  10

SEE ALSO

       getaddrinfo(3), RFC 3484

Linux								    2016-03-15							       GAI.CONF(5)