Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How do a distinct from a file using sort uniq in bash?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How do a distinct from a file using sort uniq in bash?
# 1  
Old 05-13-2013
How do a distinct from a file using sort uniq in bash?
I have an output file .dat. From this file i have to do a distinct of the ID using the sort uniq command in bash script. How can i do it? i found :
Code:
sort -u ${FILEOUT_DAT}

but i don't think is my solution because the id isn't specified.. is there other solution?
Last edited by Franklin52; 05-13-2013 at 06:34 AM.. Reason: Please use code tags for data and code samples
# 2  
Old 05-13-2013
could you provide some sample data, identifying the ID?
# 3  
Old 05-13-2013
Quote:
Originally Posted by Skrynesaver
could you provide some sample data, identifying the ID?
This is an example of data:

Code:
ID                          COD1        DATA                    COD2                   COD3                    DATA2

;0@13013008021958146502;2@*;3@MGF;4@30-JAN-13 07.57.35.000000 AM;5@0757350024630007;6@DSHDQR;7@30-JAN-13;8@30-JAN-13;9@S;10@
;0@13013008021958146502;2@*;3@MGF;4@30-JAN-13 07.57.35.000000 AM;5@0757350086490008;6@SGCFXT;7@30-JAN-13;8@30-JAN-13;9@S;10@
;0@13013008021958146502;2@*;3@ARC;4@30-JAN-13 07.57.35.000000 AM;5@27864130075735394277000;6@FSC4;7@30-JAN-13;8@30-JAN-13;9@S;10@
;0@13013008021958146502;2@*;3@TER;4@30-JAN-13 07.57.35.000000 AM;5@20130130075735448370000;6@HBG4;7@30-JAN-13;8@30-JAN-13;9@S;10@
;0@13013008021958146502;2@*;3@REB;4@30-JAN-13 07.57.39.000000 AM;5@20130130075741502619000;6@HBG4 ;7@30-JAN-13;8@30-JAN-13;9@S;10@
;0@13013008021958146502;2@*;3@IGI;4@30-JAN-13 08.02.38.000000 AM;5@20154210080238521224000;6@FSC4;7@30-JAN-13;8@30-JAN-13;9@S;10@

Every Col is separated with this ;0@ for the 1st filed ;1@ for the second and so on
# 4  
Old 05-13-2013
you can try below..

Code:
 
awk -F";" '!A[$2]++' OFS=";" filename

# 5  
Old 05-13-2013
Quote:
Originally Posted by vidyadhar85
you can try below..

Code:
 
awk -F";" '!A[$2]++' OFS=";" filename

I have to put the result in an other file sorry.. so.. something like this :

awk -F";" '!A[$2]++' OFS=";" ${FILE_OUT1}> ${FILE_OUT2}?

But nothing change.. Smilie
# 6  
Old 05-13-2013
Code:
cut -d\; -f2 ~/tmp/tmp.dat| sort | uniq

# 7  
Old 05-13-2013
Quote:
Originally Posted by Skrynesaver
Code:
cut -d\; -f2 ~/tmp/tmp.dat| sort | uniq

tmp.dat? i don't understand, sorry i'm new in bash script.. i have one file: the first one is ${FILE_OUT1} and inside there is the data (the example data i posted for example), in the other ${FILE_OUT2} i have to write the output of this command cut -d\; -f2 ~/tmp/tmp.dat| sort | uniq? Really sorry Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort & Uniq -u

Hi All, Below the actual file which i like to sort and Uniq -u /opt/oracle/work/Antony/Shell_Script> cat emp.1st 2233|a.k. shukula |g.m. |sales |12/12/52 |6000 1006|chanchal singhvi |director |sales |03/09/38 |6700... (8 Replies)
Discussion started by: Antony Ankrose
8 Replies

2. UNIX for Dummies Questions & Answers

Uniq and sort -u

Hello all, Need to pick your brains, I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters. So I want to do a sort -u file Will it be considerably faster or slower to use a uniq before piping it to sort... (3 Replies)
Discussion started by: senhia83
3 Replies

3. Shell Programming and Scripting

Trying to find the distinct lines using uniq command

Platform :Oracle Linux 6.4 Shell : bash The below file has 7 lines , some of them are duplicates. There are only 3 distinct lines. But why is the uniq command still showing 7 ? I just want the distinct lines to be returned. $ cat test.txt SELECT FC.COORD_SET_ID FROM OM_ORDER_FLOW F, -... (2 Replies)
Discussion started by: kraljic
2 Replies

4. Shell Programming and Scripting

Sort uniq or awk

Hi again, I have files with the following contents datetime,ip1,port1,ip2,port2,number How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up? Please mind the file may contain 100k lines. (8 Replies)
Discussion started by: LDHB2012
8 Replies

5. Shell Programming and Scripting

sort | uniq question

Hello, I have a large data file: 1234 8888 bbb 2745 8888 bbb 9489 8888 bbb 1234 8888 aaa 4838 8888 aaa 3977 8888 aaa I need to remove duplicate lines (where the first column is the duplicate). I have been using: sort file.txt | uniq -w4 > newfile.txt However, it seems to keep the... (11 Replies)
Discussion started by: palex
11 Replies

6. UNIX for Advanced & Expert Users

$cat file|sort|uniq

I have a file with list of redundant server names and want only unique ones of those. I have used following command but still redudant ones are listing $cat file|sort|uniq where could be the problem. Thanks, Srinivas (3 Replies)
Discussion started by: srinivas Juluri
3 Replies

7. UNIX for Dummies Questions & Answers

Sort and uniq lines of a file while keeping a header line

So, I have a file that has some duplicate lines. The file has a header line that I would like to keep at the top. I could do this by extracting the header from the file, 'sort -u' the remaining lines, and recombine them. But they are quite big, so if there is a way to do it with a single... (1 Reply)
Discussion started by: Digby
1 Replies

8. Shell Programming and Scripting

Help with Uniq and sort

The key is first field i want only uniq record for the first field in file. I want the output as or output as Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies

9. Shell Programming and Scripting

sort and uniq in perl

Does anyone have a quick and dirty way of performing a sort and uniq in perl? How an array with data like: this is bkupArr BOLADVICE_VN this is bkupArr MLT6800PROD2A this is bkupArr MLT6800PROD2A this is bkupArr BOLADVICE_VN_7YR this is bkupArr MLT6800PROD2A I want to sort it... (4 Replies)
Discussion started by: reggiej
4 Replies

10. UNIX for Dummies Questions & Answers

sort/uniq

I have a file: Fred Fred Fred Jim Fred Jim Jim If sort is executed on the listed file, shouldn't the output be?: Fred Fred Fred Fred Jim Jim Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies
Login or Register to Ask a Question