Sorting problem: Multiple delimiters, multiple keys Post: 302538042

Sponsored Content

Top Forums Shell Programming and Scripting Sorting problem: Multiple delimiters, multiple keys Post 302538042 by alister on Monday 11th of July 2011 02:42:39 PM

07-11-2011

Registered User

Quote:

Originally Posted by Ryan.

Hello

If you wanted to sort a .csv file that was filled with lines like this:

<Ticker>,<Date as YYYYMMDD>,<Time as H:M:S>,<Volume>,<Corr>

(H : [1, 23], M, S: [0, 59])

by date, does anybody know of a better solution than to turn the 3rd and 4th colons of every line into commas, sorting on four keys, and then turning those two commas in every line back to colons? It seems very inefficient to me. (I would just do it and not bother asking if these files weren't 50+GB.)

---------- Post updated at 09:43 PM ---------- Previous update was at 09:27 PM ----------

Meh, I'll let it run overnight.

Code:

sed 's/:/,/g' big_file.csv | sort -k 2,2 -k 3,3 -k 4,4 -k 5,5 -t',' | sed 's/,/:/3' | sed 's/,/:/3' > big_file.sorted.csv

Are hours, minutes and seconds all zero padded? For example, 01:02:03 instead of 1:2:3 or 1:02:03? If so, you do not need to modify anything. You can use the default lexicographical sort with the date and time fields as the keys.

Also, you mentioned that hours range betwee 1-23. In case it's relevant, that's only a 23 hour day.

If the source file is 50+ GB, you are going to need a lot of ram. You'll probably need to split the file into smaller chunks, sort them individually, and then merge them with sort -m.

Regards,
Alister

alister

View Public Profile for alister

Find all posts by alister

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - treat multiple delimiters as one

Is there anyway to get awk to treat multiple delimiters as one? Particularly spaces...

2. Shell Programming and Scripting

Sorting with multiple numeric keys

Data I want to sort :- 1 10 jj Y 2 100 vv B 19 5 jj A 1 11 hq D 3 8 op X 44 78 ds GG 1 8 hq D and want to sort based on the first 2 columns - which hold numeric values. Am using : cat filename | sort -nk 1,2 But the result is :- 1 10 jj Y 1 11 hq D

3. Shell Programming and Scripting

Cutting a file with multiple delimiters into columns

Hi All I have recently had to start using Unix for work and I have hit brick wall with this prob.... I have a file that goes a little something like this.... EUR;EUR;EUR:USD:USD;USD;;;EUR/USD;XAU/AUD;XAU/EUR;XAU/AUD,GBP/BOB,UAD/XAU;;;1.11;2.22;3.33;4.44;5.55;6.66;;; is it possible to...

4. Shell Programming and Scripting

AWK with multiple delimiters

I have the following string sample: bla bla bla bla bla I would like to extract the "123" using awk. I thought about awk -F"]" '{ print $1 }' but it doesn't work Any ideas ?

5. Shell Programming and Scripting

Sorting based on multiple delimiters

Hello, I have data where words are separated by a delimiter. In this case "=" The number of delimiters in a line can vary from 4to 8. The norm is 4. Is it possible to have a script where the file could be separated starting with highest number of delimiters and ending with the lowest An...

6. Shell Programming and Scripting

treating multiple delimiters[solved]

Hi, I need to display the last column value in the below o/p. sam2 PS 03/10/11 0 441 Unable to get o/p with this awk code awk -F"+" '{ print $4 }' pwdchk.txt I need to display 441(in this eg.) and also accept it as a variable to treat it with if condition and take a decision....

7. Shell Programming and Scripting

awk multiple delimiters

Hi Folks, This is the first time I ever encountered this situation My input file is of this kind cat input.txt 1 PAIXAF 0 1 1 -9 0 0 0 1 2 0 2 1 2 1 7 PAIXEM 0 7 1 -9 1 0 2 0 1 2 2 1 0 2 9 PAKZXY 0 2 1 -9 2 0 1 1 1 0 1 2 0 1 Till the sixth column (which is -9), I want my columns to...

8. Shell Programming and Scripting

Editing phone number with multiple delimiters

Hello all I have a data base of information that is formatted like so: JSD4863 XXX-XX-XXXX DOE, JOHN C JR-II BISS CPSC BS INFO TECH 412/779-9445 I need the last four digits of the phone number. However, many lines contain 'garbage data' that I'm not interested in. So i used a 'for loop'...

9. UNIX for Beginners Questions & Answers

How to append the multiple Delimiters up to requirement?

HI All, How to append the multiple delimiters to at end the file up to 69 fields. FinalDelimiter Count is 69 recrod Delimeter count is 10 so 69-10=59 this script will add upto 59 Delimiters to that records. this script will check each and every record in a file and append the delimiters...

10. Shell Programming and Scripting

Getting fields from a file having multiple delimiters

Hi All, I have a file with a single row having the following text ABC.ABC.ABC,Database,New123,DBNAME,F,ABC.ABC.ABC_APP,"@FUNCTION1("ENT1") ,@FUNCTION2("ENT2")",R, I want an output in the following format ABC.ABC.ABC DBNAME ABC.ABC.ABC_APP '@FUNCTION1("ENT1")...

LEARN ABOUT BSD

sortbib

SORTBIB(1)						      General Commands Manual							SORTBIB(1)

NAME

       sortbib - sort bibliographic database

SYNOPSIS

       sortbib [ -sKEYS ] database  ...

DESCRIPTION

       Sortbib	sorts files of records containing refer key-letters by user-specified keys.  Records may be separated by blank lines, or by .[ and
       .] delimiters, but the two styles may not be mixed together.  This program reads through each database and pulls out key fields, which  are
       sorted  separately.   The  sorted key fields contain the file pointer, byte offset, and length of corresponding records.  These records are
       delivered using disk seeks and reads, so sortbib may not be used in a pipeline to read standard input.

       By default, sortbib alphabetizes by the first %A and the %D fields, which contain the senior author and date.  The -s  option  is  used	to
       specify	new  KEYS.   For  instance, -sATD will sort by author, title, and date, while -sA+D will sort by all authors, and date.  Sort keys
       past the fourth are not meaningful.  No more than 16 databases may be sorted together at one time.  Records  longer  than  4096	characters
       will be truncated.

       Sortbib	sorts  on  the	last  word  on	the %A line, which is assumed to be the author's last name.  A word in the final position, such as
       ``jr.'' or ``ed.'', will be ignored if the name beforehand ends with a comma.  Authors with two-word last names	or  unusual  constructions
       can  be	sorted	correctly by using the nroff convention ``'' in place of a blank.  A %Q field is considered to be the same as %A, except
       sorting begins with the first, not the last, word.  Sortbib sorts on the last word of the %D line, usually the year.  It also ignores lead-
       ing  articles  (like  ``A''  or ``The'') when sorting by titles in the %T or %J fields; it will ignore articles of any modern European lan-
       guage.  If a sort-significant field is absent from a record, sortbib places that record before other records containing that field.

SEE ALSO

       refer(1), addbib(1), roffbib(1), indxbib(1), lookbib(1)

AUTHORS

       Greg Shenaut, Bill Tuthill

BUGS

       Records with missing author fields should probably be sorted by title.

4.2 Berkeley Distribution					  April 29, 1985							SORTBIB(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - treat multiple delimiters as one

Discussion started by: peter.herlihy

2. Shell Programming and Scripting

Sorting with multiple numeric keys

Discussion started by: sinpeak

3. Shell Programming and Scripting

Cutting a file with multiple delimiters into columns

Discussion started by: luckycharm

4. Shell Programming and Scripting

AWK with multiple delimiters

Discussion started by: gdub