UNIX for Dummies Questions & Answers

I have a large file to sort on 2 keys but the sort process fails because there is not enough space on disk. I even tried moving the directory for the disk space to somewhere with more space.

How would you sort in memory, using the unix 'sort' command?

Thanks,

- CB

So I guess you have more memory available than disk space?

I have around 64G memory, which I think should be fine for sorting in-memory

- CB

I want to sort on a numeric key.
First step, read-thru (awk command perhaps?) to separate file into ten separate pieces. subfile_0 with numbers starting with 0, subfile_1 with numbers starting with 1, and so on.
Next, sort on those ten files individually.
Finally, combine the files back together again.

If your key is alphabetic, then create 26 sub_files, and follow same path.

There may be other approaches, but without knowledge of individual record sizes, number of records, sort criteria, and actual total file size... that is the first approach that comes to mind.

This is the size of the file:
-r--r----- 1 initiate initiate 184225514544 Sep 10 11:16 INITIATE_FINAL_ALL_810804405.dat

and the line count is 810804405

So... 810 million records
Occupying 184 gigabytes

Perhaps giving problems to a memory sort.
How much free disk space do you have? More than 200gb I hope...

I am still thinking of breaking the file into pieces, and using awk arrays with a sorted file. Does your key structure provide perfectly unique records? Would there be duplicates?

the keys are unique, and the swap space is 64G. Disk space is 1.3TB

- CB