Sponsored Content
Top Forums Shell Programming and Scripting Sort data by date first and then remove duplicates Post 302810225 by samrat dutta on Tuesday 21st of May 2013 12:12:09 PM
Old 05-21-2013
Sort data by date first and then remove duplicates

Hi ,
I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points :
(1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column)
(2)after sorting the file based on latest date , remove the duplicates based on the first column only

15277105||Common Stick|ESHR||Common Stock|CYRO AB|2013-05-14T00:52:31.662-04:00

16111278||Common Stick|ESHR||Common Stock|STANDARD REGISTER CO|2013-05-14T00:52:31.672-04:00

15277105||Common Stick|ESHR||Common Stock|CYRO AB|2013-05-15T00:52:31.672-04:00

39693766||Common Stick|ESHR||Common Stock|HS AG|2013-05-15T00:52:31.672-04:00

Any help with the script is requested .
thanks,
Sam
 

10 More Discussions You Might Find Interesting

1. Programming

sort data by date.. pls help

hi all, could anyone help me? I need to query output by compare dates from 2 table and i'm using a UNION query..and wanted to sort the output by date.. My query like this: SELECT TO_CHAR(DATE) DATE1, INVOICE FROM ACCOUNT1 WHERE DATE < (to_date('122003','MMYYYY')) UNION ... (2 Replies)
Discussion started by: kate katherine
2 Replies

2. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

3. Shell Programming and Scripting

Kindly check:remove duplicates with similar data in front of it

Hi all, I have 2 files containing data like this: so if there is same entry repeated in the column like1,2,3,4 I have to check if there is different entries column like 2,4 but similar entries for duplicatein column 2 like1,3 the output shuld be like this for first file ... (5 Replies)
Discussion started by: manigrover
5 Replies

4. Shell Programming and Scripting

Remove Duplicates on multiple Key Columns and get the Latest Record from Date/Time Column

Hi Experts , we have a CDC file where we need to get the latest record of the Key columns Key Columns will be CDC_FLAG and SRC_PMTN_I and fetch the latest record from the CDC_PRCS_TS Can we do it with a single awk command. Please help.... (3 Replies)
Discussion started by: vijaykodukula
3 Replies

5. Shell Programming and Scripting

Sort data by date and then search by column

Hi, I have a file where data is pipe separated.First i want to sort the file content by date . Then i want to pick up the records based on the first column which should be unique and not have duplicates. NYSE|yyyrrrddd|toronto|isin|ticker|2013-05-15... (2 Replies)
Discussion started by: samrat dutta
2 Replies

6. Shell Programming and Scripting

Bash - remove duplicates without sort

I need to use bash to remove duplicates without using sort first. I can not use: cat file | sort | uniq But when I use only cat file | uniq some duplicates are not removed. (4 Replies)
Discussion started by: locoroco
4 Replies

7. Shell Programming and Scripting

sed --> sort data by date

Hi, i "tried" to sort data by date. So far, i used sed to take the data from the last and the actual month. Now, after changing the year it is not working properly. i use: GNU bash, version 4.2.45(1)-release (x86_64-suse-linux-gnu) sed -n '/\//p' $Home/../scripte/pd_0.txt y is a... (6 Replies)
Discussion started by: IMPe
6 Replies

8. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

9. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies

10. Shell Programming and Scripting

Concatenate and sort to remove duplicates

Following is the input. 1st and 3rd block are same(block starts here with '*' and ends before blank line) , 2nd and 4th blocks are also the same: cat <file> * Wed Feb 24 2016 Tariq Saeed <tariq.x.saeed@mail.com> 2.0.7-1.0.7 - add vmcore dump support for ocfs2 * Mon Jun 8 2015 Brian Maly... (4 Replies)
Discussion started by: Paras Pandey
4 Replies
GIT-FOR-EACH-REF(1)						    Git Manual						       GIT-FOR-EACH-REF(1)

NAME
git-for-each-ref - Output information on each ref SYNOPSIS
git for-each-ref [--count=<count>] [--shell|--perl|--python|--tcl] [(--sort=<key>)...] [--format=<format>] [<pattern>...] DESCRIPTION
Iterate over all refs that match <pattern> and show them according to the given <format>, after sorting them according to the given set of <key>. If <count> is given, stop after showing that many refs. The interpolated values in <format> can optionally be quoted as string literals in the specified host language allowing their direct evaluation in that language. OPTIONS
<count> By default the command shows all refs that match <pattern>. This option makes it stop after showing that many refs. <key> A field name to sort on. Prefix - to sort in descending order of the value. When unspecified, refname is used. You may use the --sort=<key> option multiple times, in which case the last key becomes the primary key. <format> A string that interpolates %(fieldname) from the object pointed at by a ref being shown. If fieldname is prefixed with an asterisk (*) and the ref points at a tag object, the value for the field in the object tag refers is used. When unspecified, defaults to %(objectname) SPC %(objecttype) TAB %(refname). It also interpolates %% to %, and %xx where xx are hex digits interpolates to character with hex code xx; for example %00 interpolates to (NUL), %09 to (TAB) and %0a to (LF). <pattern>... If one or more patterns are given, only refs are shown that match against at least one pattern, either using fnmatch(3) or literally, in the latter case matching completely or from the beginning up to a slash. --shell, --perl, --python, --tcl If given, strings that substitute %(fieldname) placeholders are quoted as string literals suitable for the specified host language. This is meant to produce a scriptlet that can directly be `eval`ed. FIELD NAMES
Various values from structured fields in referenced objects can be used to interpolate into the resulting output, or as sort keys. For all objects, the following names can be used: refname The name of the ref (the part after $GIT_DIR/). For a non-ambiguous short name of the ref append :short. The option core.warnAmbiguousRefs is used to select the strict abbreviation mode. objecttype The type of the object (blob, tree, commit, tag). objectsize The size of the object (the same as git cat-file -s reports). objectname The object name (aka SHA-1). For a non-ambiguous abbreviation of the object name append :short. upstream The name of a local ref which can be considered "upstream" from the displayed ref. Respects :short in the same way as refname above. In addition to the above, for commit and tag objects, the header field names (tree, parent, object, type, and tag) can be used to specify the value in the header field. Fields that have name-email-date tuple as its value (author, committer, and tagger) can be suffixed with name, email, and date to extract the named component. The complete message in a commit and tag object is contents. Its first line is contents:subject, the remaining lines are contents:body and the optional GPG signature is contents:signature. For sorting purposes, fields with numeric values sort in numeric order (objectsize, authordate, committerdate, taggerdate). All other fields are used to sort in their byte-value order. In any case, a field name that refers to a field inapplicable to the object referred by the ref does not cause an error. It returns an empty string instead. As a special case for the date-type fields, you may specify a format for the date by adding one of :default, :relative, :short, :local, :iso8601 or :rfc2822 to the end of the fieldname; e.g. %(taggerdate:relative). EXAMPLES
An example directly producing formatted text. Show the most recent 3 tagged commits: #!/bin/sh git for-each-ref --count=3 --sort='-*authordate' --format='From: %(*authorname) %(*authoremail) Subject: %(*subject) Date: %(*authordate) Ref: %(*refname) %(*body) ' 'refs/tags' A simple example showing the use of shell eval on the output, demonstrating the use of --shell. List the prefixes of all heads: #!/bin/sh git for-each-ref --shell --format="ref=%(refname)" refs/heads | while read entry do eval "$entry" echo `dirname $ref` done A bit more elaborate report on tags, demonstrating that the format may be an entire script: #!/bin/sh fmt=' r=%(refname) t=%(*objecttype) T=${r#refs/tags/} o=%(*objectname) n=%(*authorname) e=%(*authoremail) s=%(*subject) d=%(*authordate) b=%(*body) kind=Tag if test "z$t" = z then # could be a lightweight tag t=%(objecttype) kind="Lightweight tag" o=%(objectname) n=%(authorname) e=%(authoremail) s=%(subject) d=%(authordate) b=%(body) fi echo "$kind $T points at a $t object $o" if test "z$t" = zcommit then echo "The commit was authored by $n $e at $d, and titled $s Its message reads as: " echo "$b" | sed -e "s/^/ /" echo fi ' eval=`git for-each-ref --shell --format="$fmt" --sort='*objecttype' --sort=-taggerdate refs/tags` eval "$eval" AUTHOR
Written by Junio C Hamano <gitster@pobox.com[1]>. DOCUMENTATION
Documentation by Junio C Hamano and the git-list <git@vger.kernel.org[2]>. GIT
Part of the git(1) suite NOTES
1. gitster@pobox.com mailto:gitster@pobox.com 2. git@vger.kernel.org mailto:git@vger.kernel.org Git 1.7.10.4 11/24/2012 GIT-FOR-EACH-REF(1)
All times are GMT -4. The time now is 09:31 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy