05-21-2013
Sort data by date first and then remove duplicates
Hi ,
I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points :
(1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column)
(2)after sorting the file based on latest date , remove the duplicates based on the first column only
15277105||Common Stick|ESHR||Common Stock|CYRO AB|2013-05-14T00:52:31.662-04:00
16111278||Common Stick|ESHR||Common Stock|STANDARD REGISTER CO|2013-05-14T00:52:31.672-04:00
15277105||Common Stick|ESHR||Common Stock|CYRO AB|2013-05-15T00:52:31.672-04:00
39693766||Common Stick|ESHR||Common Stock|HS AG|2013-05-15T00:52:31.672-04:00
Any help with the script is requested .
thanks,
Sam
10 More Discussions You Might Find Interesting
1. Programming
hi all,
could anyone help me?
I need to query output by compare dates from 2 table and i'm using a UNION query..and wanted to sort the output by date..
My query like this:
SELECT
TO_CHAR(DATE) DATE1,
INVOICE
FROM ACCOUNT1
WHERE DATE < (to_date('122003','MMYYYY'))
UNION ... (2 Replies)
Discussion started by: kate katherine
2 Replies
2. Shell Programming and Scripting
Hi,
I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another.
Thanks (6 Replies)
Discussion started by: dvah
6 Replies
3. Shell Programming and Scripting
Hi all,
I have 2 files containing data like this:
so if there is same entry repeated in the column like1,2,3,4
I have to check if there is different entries column like 2,4
but similar entries for duplicatein column 2 like1,3
the output shuld be like this for first file
... (5 Replies)
Discussion started by: manigrover
5 Replies
4. Shell Programming and Scripting
Hi Experts ,
we have a CDC file where we need to get the latest record of the Key columns
Key Columns will be CDC_FLAG and SRC_PMTN_I
and fetch the latest record from the CDC_PRCS_TS
Can we do it with a single awk command.
Please help.... (3 Replies)
Discussion started by: vijaykodukula
3 Replies
5. Shell Programming and Scripting
Hi,
I have a file where data is pipe separated.First i want to sort the file content by date . Then i want to pick up the records based on the first column which should be unique and not have duplicates.
NYSE|yyyrrrddd|toronto|isin|ticker|2013-05-15... (2 Replies)
Discussion started by: samrat dutta
2 Replies
6. Shell Programming and Scripting
I need to use bash to remove duplicates without using sort first.
I can not use:
cat file | sort | uniq
But when I use only
cat file | uniq
some duplicates are not removed. (4 Replies)
Discussion started by: locoroco
4 Replies
7. Shell Programming and Scripting
Hi,
i "tried" to sort data by date. So far, i used sed to take the data from the last and the actual month. Now, after changing the year it is not working properly.
i use:
GNU bash, version 4.2.45(1)-release (x86_64-suse-linux-gnu)
sed -n '/\//p' $Home/../scripte/pd_0.txt
y is a... (6 Replies)
Discussion started by: IMPe
6 Replies
8. Shell Programming and Scripting
Here is my task :
I need to sort two input files and remove duplicates in the output files :
Sort by 13 characters from 97 Ascending
Sort by 1 characters from 96 Ascending
If duplicates are found retain the first value in the file
the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies
9. UNIX for Beginners Questions & Answers
I have /tmp dir with filename as:
010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker
010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker
010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker
010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker
010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies
10. Shell Programming and Scripting
Following is the input. 1st and 3rd block are same(block starts here with '*' and ends before blank line) , 2nd and 4th blocks are also the same:
cat <file>
* Wed Feb 24 2016 Tariq Saeed <tariq.x.saeed@mail.com> 2.0.7-1.0.7
- add vmcore dump support for ocfs2
* Mon Jun 8 2015 Brian Maly... (4 Replies)
Discussion started by: Paras Pandey
4 Replies
LEARN ABOUT MOJAVE
regexp::common::whitespace5.18
Regexp::Common::whitespace(3) User Contributed Perl Documentation Regexp::Common::whitespace(3)
NAME
Regexp::Common::whitespace -- provides a regex for leading or trailing whitescape
SYNOPSIS
use Regexp::Common qw /whitespace/;
while (<>) {
s/$RE{ws}{crop}//g; # Delete surrounding whitespace
}
DESCRIPTION
Please consult the manual of Regexp::Common for a general description of the works of this interface.
Do not use this module directly, but load it via Regexp::Common.
$RE{ws}{crop}
Returns a pattern that identifies leading or trailing whitespace.
For example:
$str =~ s/$RE{ws}{crop}//g; # Delete surrounding whitespace
The call:
$RE{ws}{crop}->subs($str);
is optimized (but probably still slower than doing the s///g explicitly).
This pattern does not capture under "-keep".
SEE ALSO
Regexp::Common for a general description of how to use this interface.
AUTHOR
Damian Conway (damian@conway.org)
MAINTAINANCE
This package is maintained by Abigail (regexp-common@abigail.be).
BUGS AND IRRITATIONS
Bound to be plenty.
For a start, there are many common regexes missing. Send them in to regexp-common@abigail.be.
LICENSE and COPYRIGHT
This software is Copyright (c) 2001 - 2009, Damian Conway and Abigail.
This module is free software, and maybe used under any of the following licenses:
1) The Perl Artistic License. See the file COPYRIGHT.AL.
2) The Perl Artistic License 2.0. See the file COPYRIGHT.AL2.
3) The BSD Licence. See the file COPYRIGHT.BSD.
4) The MIT Licence. See the file COPYRIGHT.MIT.
perl v5.18.2 2013-03-08 Regexp::Common::whitespace(3)