Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract unique files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract unique files
# 1  
Old 09-04-2013
Extract unique files
Code:
 
In a incoming folder i have list of files like below,i want to pick the unique files to process the job. if same file  contain more than one then it should pick latest date modified file to process.
 

drwxrwsrwx    2 n308799  infagrp         256 May 20 17:42 Final_Working
drwxrwsrwx    2 n308799  infagrp         256 May 20 17:42 scripts_working
drwxrwsrwx    2 n308799  infagrp         256 May 20 17:42 Scripts_bkp
drwxrwsrwx    2 n308799  infagrp         256 May 22 17:42 Final_Working
output should be as below:
drwxrwsrwx    2 n308799  infagrp         256 May 22 17:42 Final_Working
drwxrwsrwx    2 n308799  infagrp         256 May 20 17:42 scripts_working
drwxrwsrwx    2 n308799  infagrp         256 May 20 17:42 Scripts_bkp

# 2  
Old 09-04-2013
Assuming the lines are in chronological order within the file and there are no spaces in the file names, try
Code:
awk '{A[$NF]=$0} END{for(i in A) print A[i]}' file

This User Gave Thanks to Scrutinizer For This Post:
sdebasis
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract lines with unique value using a specific column

Hi there, I need a help with extracting data from tab delimited file which look like this #CHROM POS ID REF ALT Human Cow Dog Mouse Lizard chr2 3033 . G C 0/0 0/0 0/0 1/1 0/0 chr3 35040 . G T 0/0 0/0 ./. 1/1 0/1 chr4 60584 . T G 1/1 1/1 0/1 1/1 0/0 chr10 7147815 . G A 0/0 1/1 0/0 0/0... (9 Replies)
Discussion started by: houkto
9 Replies

2. Shell Programming and Scripting

Extract UNIque records from File

Hi, I have a file with 20GB Pipe Delimited file where i have too many duplicate records. I need an awk script to extract the unique records from the file and put it into another file. Kindly help. Thanks, Arun (1 Reply)
Discussion started by: Arun Mishra
1 Replies

3. UNIX for Dummies Questions & Answers

Extract unique combination of rows from text files

Hi Gurus, I have 100 tab-delimited text files each with 21 columns. I want to extract only 2nd and 5th column from each text file. However, the values in both 2bd and 5th column contain duplicate values but the combination of these values in a row are not duplicate. I want to extract only those... (3 Replies)
Discussion started by: Unilearn
3 Replies

4. UNIX for Dummies Questions & Answers

getting unique lines from 2 files

hi i have used comm -13 <(sort 1.txt) <(sort 2.txt) option to get the unique lines that are present in file 2 but not in file 1. but some how i am getting the entire file 2. i would expect few but not all uncommon lines fro my dat. is there anything wrong with the way i used the command? my... (1 Reply)
Discussion started by: anurupa777
1 Replies

5. Shell Programming and Scripting

Create shell script to extract unique information from one file to a new file.

Hi to all, I got this content/pattern from file http.log.20110808.gz mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1 mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Discussion started by: Mr_47
16 Replies

6. Shell Programming and Scripting

Extract unique filenames

Hi Unix Gurus, In a script, I am trying to extract unique text from a set of filenames. I have certain files like below in a directory: OPEN_INV_01012011.xls OPEN_INV_01022011.xls OPEN_INV_01032011.xls CLOSE_INV_01012011.xls CLOSE_INV_01022011.xls I need to extract just "OPEN_INV_" ... (4 Replies)
Discussion started by: shankar1dada
4 Replies

7. Shell Programming and Scripting

Unique files in a given directory

I keep all my files on a NAS device and copy files from it to usb or local storage when needed. The bad part about this is that I often have the same file on numerous places. I'd like to write a script to check if the files in a given directory exist in another. An example: say I have a... (7 Replies)
Discussion started by: cue
7 Replies

8. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Hello all, I have a file with following sample data 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26... (5 Replies)
Discussion started by: simonsimon
5 Replies

9. Shell Programming and Scripting

extract unique pattern from large text file

Hi All, I am trying to extract data from a large text file , I want to extract lines which contains a five digit number followed by a hyphen , like 12345- , i tried with egrep ,eg : egrep "+" text.txt but which returns all the lines which contains any number of digits followed by hyhen ,... (19 Replies)
Discussion started by: shijujoe
19 Replies

10. Shell Programming and Scripting

ksh scripting: Extract 1 most recent record for unique key

I'm loading multiple delimited files into an Oracle DB using sqlldr on Unix. I would like to get only the most recent record per each unique key. There may be multiple updates for each key, but I only want the most recent one. There is a date column in my delimited files, so I'm using cat to... (2 Replies)
Discussion started by: OPTIMUS_prime
2 Replies
Login or Register to Ask a Question