Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Sort csv file by duplicated column value Post 302798479 by jl487 on Wednesday 24th of April 2013 02:21:09 PM
Old 04-24-2013
Sort csv file by duplicated column value

hello, I have a large file (about 1gb) that is in a file similar to the following:

Quote:
"Timmy","??","Age 26","1","0"
"Jack","??","Age 21","1","0"
"Troy","??","Age 21","1","0"
"Kim","?","Age 26","1","0"
"Mark","???","Age 24","1","0"
"John","??","Age 27","1","0"
I want to make it so that I can put all the duplicates where column 3 (delimited by the commas) are shown on top. Meaning all people with the same age are listed at the top.

Quote:
"Timmy","??","Age 26","1","0"
"Kim","?","Age 26","1","0"
"Jack","??","Age 21","1","0"
"Troy","??","Age 21","1","0"
"Mark","???","Age 24","1","0"
"John","??","Age 27","1","0"
The command I used was
sort -t, +2 input.csv > output.csv

I assumed that-t would make ',' be the delimiter and "+2" would look at the second column. What am I doing wrong?
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies

2. UNIX for Dummies Questions & Answers

duplicated lines not recognized by sort and uniq

Hello all, I've got a strange behaviour of sort and uniq commands: they do not recognise apparently duplicated lines in a file (already sorted). The lines are identical by eye, but they must differ in smth, because when they are put in two files, those have slightly different size. What can make... (8 Replies)
Discussion started by: roussine
8 Replies

3. Shell Programming and Scripting

Sort, sed, and zero padding date column csv bash scripting

Hello people, I am having problem to sort, sed and zero padding of column in csv file. 7th column only. Input of csv file: 1,2,3,4,5,6,4/1/2010 12:00 AM,8 1,2,3,4,5,6,3/11/2010 9:39 AM,8 1,2,3,4,5,6,5/12/2011 3:43 PM,8 1,2,3,4,5,6,12/20/2009 7:23 PM,8 Output:... (5 Replies)
Discussion started by: sean1357
5 Replies

4. Shell Programming and Scripting

How to remove duplicated column in a text file?

Dear all, How can I remove duplicated column in a text file? Input: LG10_PM_map_19_LEnd 1000560 G AA AA AA AA AA GG LG10_PM_map_19_LEnd 1005621 G GG GG GG AA AA GG LG10_PM_map_19_LEnd 1011214 A AA AA AA AA GG GG LG10_PM_map_19_LEnd 1011673 T TT TT TT TT CC CC... (1 Reply)
Discussion started by: huiyee1
1 Replies

5. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

6. Shell Programming and Scripting

How to delete 'duplicated' column values and make a delimited file too?

Hi, I have the following output from an Oracle SQL statement and I want to remove duplicated column values. I know it is possible using Oracle analytical/statistical functions but unfortunately I don't know how to use any of those. So now, I've gone to PLAN B using awk/sed maybe or any... (5 Replies)
Discussion started by: newbie_01
5 Replies

7. Shell Programming and Scripting

awk script to append suffix to column when column has duplicated values

Please help me to get required output for both scenario 1 and scenario 2 and need separate code for both scenario 1 and scenario 2 Scenario 1 i need to do below changes only when column1 is CR and column3 has duplicates rows/values. This inputfile can contain 100 of this duplicated rows of... (1 Reply)
Discussion started by: as7951
1 Replies

8. UNIX for Beginners Questions & Answers

How to sort a column in excel/csv file?

I have to sort the 4th column of an excel/csv file. I tried the following command sort -u --field-separator=, --numeric-sort -k 2 -n dinesh.csv > test.csv But, it's not working. Moreover, I have to do the same for more than 30 excel/csv file. So please help me to do the same. (6 Replies)
Discussion started by: dineshkumarsrk
6 Replies

9. UNIX for Beginners Questions & Answers

How to align/sort the column pairs of an csv file, based on keyword word specified in another file?

I have a csv file as shown below, xop_thy 80 avr_njk 50 str_nyu 60 avr_irt 70 str_nhj 60 avr_ngt 50 str_tgt 80 xop_nmg 50 xop_nth 40 cyv_gty 40 cop_thl 40 vir_tyk 80 vir_plo 20 vir_thk 40 ijk_yuc 70 cop_thy 70 ijk_yuc 80 irt_hgt 80 I need to align/sort the csv file based... (7 Replies)
Discussion started by: dineshkumarsrk
7 Replies
Boulder::Stream(3pm)					User Contributed Perl Documentation				      Boulder::Stream(3pm)

NAME
Boulder::Stream - Read and write tag/value data from an input stream SYNOPSIS
#!/bin/perl # Read a series of People records from STDIN. # Add an "Eligible" tag to all those whose # Age >= 35 and Friends list includes "Fred" use Boulder::Stream; # filestream way: my $stream = Boulder::Stream->newFh; while ( my $record = <$stream> ) { next unless $record->Age >= 35; my @friends = $record->Friends; next unless grep {$_ eq 'Fred'} @friends; $record->insert(Eligible => 'yes'); print $stream $record; } # object oriented way: my $stream = Boulder::Stream->new; while (my $record = $stream->get ) { next unless $record->Age >= 35; my @friends = $record->Friends; next unless grep {$_ eq 'Fred'} @friends; $record->insert(Eligible => 'yes'); print $stream $record; } DESCRIPTION
Boulder::Stream provides stream-oriented access to Boulder IO hierarchical tag/value data. It can be used in a magic tied filehandle mode, as shown in the synopsis, or in object-oriented mode. Using tied filehandles, Stone objects are read from input using the standard <> operator. Stone objects printed to the tied filehandle appear on the output stream in Boulder format. By default, data is read from the magic ARGV filehandle (STDIN or a list of files provided on the command line) and written to STDOUT. This can be changed to the filehandles of your choice. Pass through behavior When using the object-oriented form of Boulder::Stream, tags which aren't specifically requested by the get() method are passed through to output unchanged. This allows pipes of programs to be constructed easily. Most programs will want to put the tags back into the boulder stream once they're finished, potentially adding their own. Of course some programs will want to behave differently. For example, a database query program will generate but not read a boulderio stream, while a report generator will read but not write the stream. This convention allows the following type of pipe to be set up: query_database | find_vector | find_dups | | blast_sequence | pick_primer | mail_report If all the programs in the pipe follow the conventions, then it will be possible to interpose other programs, such as a repetitive element finder, in the middle of the pipe without disturbing other components. SKELETON BOULDER PROGRAM
Here is a skeleton example. #!/bin/perl use Boulder::Stream; my $stream = Boulder::Stream->newFh; while ( my $record = <$stream> ) { next unless $record->Age >= 35; my @friends = $record->Friends; next unless grep {$_ eq 'Fred'} @friends; $record->insert(Eligible => 'yes'); print $stream $record; } The code starts by creating a Boulder::Stream object to handle the I/O. It reads from the stream one record at a time, returning a Stone object. We recover the Age and Friends tags, and continue looping unless the Age is greater or equal to 35, and the list of Friends contains "Fred". If these criteria match, then we insert a new tag named Eligible and print the record to the stream. The output may look like this: Name=Janice Age=36 Eligible=yes Friends=Susan Friends=Fred Friends=Ralph = Name=Ralph Age=42 Eligible=yes Friends=Janice Friends=Fred = Name=Susan Age=35 Eligible=yes Friends=Susan Friends=Fred = Note that in this case only records that meet the criteria are echoed to standard output. The object-oriented version of the program looks like this: #!/bin/perl use Boulder::Stream; my $stream = Boulder::Stream->new; while ( my $record = $stream->get('Age','Friends') ) { next unless $record->Age >= 35; my @friends = $record->Friends; next unless grep {$_ eq 'Fred'} @friends; $record->insert(Eligible => 'yes'); $stream->put($record); } The get() method is used to fetch Stones containing one or more of the indicated tags. The put() method is used to send the result to standard output. The pass-through behavior might produce a set of records like this one: Name=Janice Age=36 Eligible=yes Friends=Susan Friends=Fred Friends=Ralph = Name=Phillip Age=30 = Name=Ralph Age=42 Eligible=yes Friends=Janice Friends=Fred = Name=Barbara Friends=Agatha Friends=Janice = Name=Susan Age=35 Eligible=yes Friends=Susan Friends=Fred = Notice that there are now two records ("Phillip" and "Barbara") that do not contain the Eligible tag. Boulder::Stream METHODS $stream = Boulder::Stream->new(*IN,*OUT) $stream = Boulder::Stream->new(-in=>*IN,-out=>*OUT) The new() method creates a new Boulder::Stream object. You can provide input and output filehandles. If you leave one or both undefined new() will default to standard input or standard output. You are free to use files, pipes, sockets, and other types of file handles. You may provide the filehandle arguments as bare words, globs, or glob refs. You are also free to use the named argument style shown in the second heading. $fh = Boulder::Stream->newFh(-in=>*IN, -out=>*OUT) Returns a filehandle object tied to a Boulder::Stream object. Reads on the filehandle perform a get(). Writes invoke a put(). To retrieve the underlying Boulder::Stream object, call Perl's built-in tied() function: $stream = tied $fh; $stone = $stream->get(@taglist) @stones = $stream->get(@taglist) Every time get() is called, it will return a new Stone object. The Stone will be created from the input stream, using just the tags provided in the argument list. Pass no tags to receive whatever tags are present in the input stream. If none of the tags that you specify are in the current boulder record, you will receive an empty Stone. At the end of the input stream, you will receive undef. If called in an array context, get() returns a list of all stones from the input stream that contain one or more of the specified tags. $stone = $stream->read_record(@taglist) Identical to get(>, but the name is longer. $stream->put($stone) Write a Stone to the output filehandle. $stream->write_record($stone) Identical to put(), but the name is longer. Useful State Variables in a Boulder::Stream Every Boulder::Stream has several state variables that you can adjust. Fix them in this fashion: $a = new Boulder::Stream; $a->{delim}=':'; $a->{record_start}='['; $a->{record_end}=']'; $a->{passthru}=undef; o delim This is the delimiter character between tags and values, "=" by default. o record_start This is the start of nested record character, "{" by default. o record_end This is the end of nested record character, "}" by default. o passthru This determines whether unrecognized tags should be passed through from the input stream to the output stream. This is 'true' by default. Set it to undef to override this behavior. BUGS
Because the delim, record_start and record_end characters in the Boulder::Stream object are used in optimized (once-compiled) pattern matching, you cannot change these values once get() has once been called. To change the defaults, you must create the Boulder::Stream, set the characters, and only then begin reading from the input stream. For the same reason, different Boulder::Stream objects cannot use different delimiters. AUTHOR
Lincoln D. Stein <lstein@cshl.org>, Cold Spring Harbor Laboratory, Cold Spring Harbor, NY. This module can be used and distributed on the same terms as Perl itself. SEE ALSO
Boulder, Boulder::Blast, Boulder::Genbank, Boulder::Medline, Boulder::Unigene, Boulder::Omim, Boulder::SwissProt perl v5.10.1 2001-06-11 Boulder::Stream(3pm)
All times are GMT -4. The time now is 02:21 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy