Explanation of the sort command


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Explanation of the sort command
# 8  
Old 01-21-2013
Like Don Cragun suggested, you would need to specify a field separator for sort which in this case would probably be an underscore ( _ )
Code:
sort -t_ ...

The number in this case appears to consist of two parts:
8 numbers for date, and the rest of the numbers for sequence.

If that is correct that would mean something like:
Code:
sort -rnt_ -k4.1,4.8 -k4.9

It is probably best to make sure the field number is not influenced by underscores in the path name, so:
Code:
cd "${DEST_LOCATION}"
ls ${FILES} | ...

So then the sort would become:
Code:
sort -rnt_ -k3.1,3.8 -k3.9

One thing to keep in mind is that if there are too many files in $FILES then ls would break because the total line length would be exceeded...
# 9  
Old 01-21-2013
Quote:
Originally Posted by jimbojames
Hi Don,

The output from running
Code:
ls ${DEST_LOCATION}/${FILES} | tee /tmp/pipe1 | sort -r -k 4,4n | tee /tmp/pipe2 | head -1

is
Code:
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PreviousOld.CSV]

.

The contents of pipe1 are:
Code:
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130112235921.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130113001600.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130114002828.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130115144300.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116105000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116122400.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116150101.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116235654.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117113000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117133000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117155000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117235938.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_201302091000.CSV

And the contents of pipe2 are:
Code:
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_201302091000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117235938.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117155000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117133000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130117113000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116235654.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116150101.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116122400.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130116105000.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130115144300.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130114002828.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130113001600.CSV
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PROD_20130112235921.CSV

What I am attempting to do is identify the newest file and delete any other than this, which I think is working using the combination of
Code:
FILENAME=$(ssh ${DEST_SERV} "ls ${DEST_LOCATION}/${FILES} | sort -r -k 4n | head -1")

and
Code:
ssh ${DEST_SERV} "ls ${DEST_LOCATION}/${FILES} | grep -v ${FILENAME} | xargs rm -rf"

.
I'm sorry, but this just is not possible. The head -1 should print the 1st line in pipe2. But:
Code:
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PreviousOld.CSV]

does not appear in pipe2 nor in pipe1!

But, whether it is there or not, the -k option and its option argument (whether it is 4n or 4,4n have absolutely no effect in this pipeline since sort is given input that only contains one field.

The files in pipe1 and pipe2 are in alphabetic and (assuming the last part of the name before the .CSV is the date in YYYYMMDDhhmm[ss] format) in date order with file1 in increasing order and file2 in reverse order.

Note that if the commands:
Code:
FILENAME=$(ls ${DEST_LOCATION}/${FILES} | sort -r -k 4n | head -1)
ls ${DEST_LOCATION}/${FILES} | grep -v ${FILENAME} | xargs rm -rf"

do what you want, the commands:
Code:
last=""
for i in ${DEST_LOCATION}/${FILES}
do      if [ "$last" != "" ]
        then   rm "$last"
        fi
        last="$i"
done

or equivalently:
Code:
last="";for i in ${DEST_LOCATION}/${FILES};do if [ "$last" != "" ];then rm "$last";fi;last="$i";done

will do the same thing in the shell just using rm without needing to call ls twice, sort, head, grep, or xargs.

Note also that if
Code:
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PreviousOld.CSV]

is present with the files listed in file1 and file2, all of the files shown in file1 and file2 will be removed just leaving
Code:
/data/projects/PROD_IPU/inbound/SOVEREIGN/SOVEREIGN_PreviousOld.CSV]

.

Last edited by Don Cragun; 01-21-2013 at 11:01 AM.. Reason: Change *.CSV* to the list actually used in the original scripts
# 10  
Old 01-21-2013
Thank you Don & Scrutinizer, it's has been very valuable. Is there a way to do the above but base the surviving file on the created date, not alphabetically?

Last edited by jimbojames; 01-21-2013 at 05:50 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Explanation of Nawk command

Hi Folks, I am struggling to understand nawk command which was used by another developer. Can you please explain what each character or string is doing here below: if ; then (3 Replies)
Discussion started by: kirans.229
3 Replies

2. UNIX for Beginners Questions & Answers

Explanation for sort utility and its detail

I tried to use sort utility and typed sort --help, read one of lines; its -k option, and part of it: I am really not getting it Anyone do me a useful favor to save me out of my ignorance ? Please use correct tags as required by forum rules! (1 Reply)
Discussion started by: abdulbadii
1 Replies

3. Shell Programming and Scripting

sed command explanation

Will someone give me an explanation on how the sed command below works. sed 's/.*//' Thanks! (3 Replies)
Discussion started by: scj2012
3 Replies

4. Shell Programming and Scripting

Need explanation a of command in linux

Hi All I ran a script in Linux. In the script i have lines like && echo "Failed: Missing ${CM_ENV_FILE} \n" && return 1 . ${CM_ENV_FILE} Where CM_ENV_FILE = /data/ds/dpr_ebicm_uat//etl/cm3_0/entities/BBME/parameters/cm.env But its taking this path... (1 Reply)
Discussion started by: vee_789
1 Replies

5. UNIX for Advanced & Expert Users

command explanation

can anyone please tell me what does this expression means , i am under probation and need some explanation :) $AUDIT_DIR -type f -mtime +$AUDIT_EXPIRE \ -exec rm {} > /dev/null 2>&1 \; AUDIT_DIR="/var/log/" AUDIT_EXPIRE='30' Please use code tags! (4 Replies)
Discussion started by: semaan
4 Replies

6. Shell Programming and Scripting

sed sorting command explanation

sed '$!N; /^\(.*\)\n\1$/!P; D' i found this file which removes duplicates irrespective for sorted or unsorted file. keep first occurance and remove the further occurances. can any1 explain how this is working.. i need to remove duplicates following file. duplicate criteria is not the... (3 Replies)
Discussion started by: mukeshguliao
3 Replies

7. UNIX for Dummies Questions & Answers

SED command explanation

can someone please explain the below sed command.. sed 's/\(*|\)\(.*\)/\2\1/' (6 Replies)
Discussion started by: raghu_shekar
6 Replies

8. Shell Programming and Scripting

How to Sort Floating Numbers Using the Sort Command?

Hi to all. I'm trying to sort this with the Unix command sort. user1:12345678:3.5:2.5:8:1:2:3 user2:12345679:4.5:3.5:8:1:3:2 user3:12345687:5.5:2.5:6:1:3:2 user4:12345670:5.5:2.5:5:3:2:1 user5:12345671:2.5:5.5:7:2:3:1 I need to get this: user3:12345687:5.5:2.5:6:1:3:2... (7 Replies)
Discussion started by: daniel.gbaena
7 Replies

9. Shell Programming and Scripting

command line explanation

Hello everyone, I found this command line in a website: perl -pi.bak -we's/\z/Your new line\n/ if $. == 2;' your_text_file.txt With this command line you can insert a new line anywhere you want in a text without overwriting what's in it. -p causes perl to assume a loop around your... (4 Replies)
Discussion started by: goude
4 Replies

10. Shell Programming and Scripting

sed command explanation needed

Hi, Could you please explain me the below statement -- phrase wise. sed -e :a -e '$q;N;'$cnt',$D;ba' abc.txt > xyz.txt if suppose $cnt contains value: 10 it copies last 9 lines of abc.txt to xyz.txt why it is copying last 9 rather than 10. and also what is ba and $D over there in... (4 Replies)
Discussion started by: subbukns
4 Replies
Login or Register to Ask a Question