The problem is that this single command is taking more than 12 hours with 10 g of memory and I am looking for a way to speed things up.
I have heard that splitting a large file into subfiles, then sorting each subfile and then merging back together with the sort command can work, but I am imaging this will not work because I'm using the -u flag to keep only unique rows.
maybe split and then sort first WITHOUT the -u flag and then sort -m -u -k2,2 -k3,3n -k4,4n -k5,5n -k6,6n subfiles ?
I'm thinking of ways to test that, but please let me know if you have any ideas for me!
Thank you kindly.
Jonathan
Last edited by radoulov; 08-31-2011 at 06:01 PM..
Reason: Please use code tags for code and data samples, thank you
Hi,
May I know, if a pipe separated File is large, what is the best method to calculate the unique row count of 3rd column and get a list of unique value of the 3rdcolum?
Thanks in advance! (20 Replies)
This may sound like a trivial problem, but I still need some help:
I have a file with ids and I want to split it 'n' ways (could be any number) into files:
1
1
1
2
2
3
3
4
5
5
Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may... (8 Replies)
Input file
---------
12:name1:|host1|host1|host2|host1
13:name2:|host1|host1|host2|host3
14:name3:
......
Required output
---------------
12:name1:host1(2)|host1(1)
13:name2:host1(2)|host2(1)|host3(1)
14:name3:
where (x) - Count how many times field appears in last column
... (3 Replies)
Hi. I am not sure the title gives an optimal description of what I want to do.
I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,... (0 Replies)
I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks. (7 Replies)
Hi,
I have an input file that I have sorted in a previous stage by $1 and $4. I now need something that will take the first record from each group of data based on the key being $1
Input file
1000AAA|"ZZZ"|"Date"|"1"|"Y"|"ABC"|""|AA
1000AAA|"ZZZ"|"Date"|"2"|"Y"|"ABC"|""|AA... (2 Replies)
Dear community, I am facing a problem and I kindly ask your help:
I have 4 different data sets consisted from 3 different types of array.
On each file, column 1 is chromosome position, column 2 is SNP id etc... Lets say I have the following (bim) datasets:
x2014:
1 rs3094315... (4 Replies)
Discussion started by: fondan
4 Replies
LEARN ABOUT HPUX
wrap-and-sort
WRAP-AND-SORT(1) General Commands Manual WRAP-AND-SORT(1)NAME
wrap-and-sort - wrap long lines and sort items in Debian packaging files
SYNOPSIS
wrap-and-sort [options]
DESCRIPTION
wrap-and-sort wraps the package lists in Debian control files. By default the lists will only split into multiple lines if the entries are
longer than 80 characters. wrap-and-sort sorts the package lists in Debian control files and all .install files. Beside that wrap-and-sort
removes trailing spaces in these files.
This script should be run in the root of a Debian package tree. It searches for control, control.in, copyright, copyright.in, install, and
*.install in the debian directory.
OPTIONS -h, --help
Show this help message and exit.
-a, --wrap-always
Wrap all package lists in the Debian control file even if the entries are shorter than 80 characters and could fit in one line line.
-s, --short-indent
Only indent wrapped lines by one space (default is in-line with the field name).
-b, --sort-binary-packages
Sort binary package paragraphs by name.
-k, --keep-first
When sorting binary package paragraphs, leave the first one at the top. Unqualified debhelper(7) configuration files are applied to
the first package.
-n, --no-cleanup
Do not remove trailing whitespaces.
-d path, --debian-directory=path
Location of the debian directory (default: ./debian).
-f file, --file=file
Wrap and sort only the specified file. You can specify this parameter multiple times. All supported files will be processed if no
files are specified.
-v, --verbose
Print all files that are touched.
AUTHORS
wrap-and-sort and this manpage have been written by Benjamin Drung <bdrung@debian.org>.
Both are released under the ISC license.
DEBIAN Debian Utilities WRAP-AND-SORT(1)