uniq the file content


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting uniq the file content
# 1  
Old 08-13-2009
uniq the file content

Hi
I want to uniq the content of a file based on 4th field

a.txt
Code:
abcd,123,456,234,asd
defg,123,456,235,asd
abcd,123,456,234,asd,lkjl,ijk
ijkl,123,456,234,asd
defg,123,456,235,asd,dfg,klm

o/p will be like

Code:
abcd,123,456,234,asd
defg,123,456,235,asd
ijkl,123,456,234,asd

or

Code:
abcd,123,456,234,asd,lkjl,ijk
ijkl,123,456,234,asd
defg,123,456,235,asd,dfg,klm

pls help

Last edited by vgersh99; 08-13-2009 at 08:45 AM.. Reason: code tags, PLEASE!
# 2  
Old 08-13-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
# 3  
Old 08-13-2009
According to your question , if you need based on 4th field, then

output will be

Code:
ijkl,123,456,234,asd
defg,123,456,235,asd,dfg,klm

# 4  
Old 08-13-2009
use this code:-

Code:
nawk '
!a[$1$4]++
' input_file

Best Regards
# 5  
Old 08-13-2009
Quote:
Originally Posted by ahmad.diab
use this code:-

Code:
nawk '
!a[$1$4]++
' input_file

Best Regards
Pls add, -F",".
# 6  
Old 08-13-2009
Quote:
Originally Posted by panyam
Pls add, -F",".
thanks panyam sorry I forget :-

Code:
nawk -F"," '
!a[$1$4]++
' input_file

# 7  
Old 08-15-2009
Hi when i am giving this comman i m getting error like

Code:
nawk -F"," '!a[$1$4]++' test
a[: Event not found.

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

2. Shell Programming and Scripting

How do a distinct from a file using sort uniq in bash?

I have an output file .dat. From this file i have to do a distinct of the ID using the sort uniq command in bash script. How can i do it? i found : sort -u ${FILEOUT_DAT} but i don't think is my solution because the id isn't specified.. is there other solution? (7 Replies)
Discussion started by: punticci
7 Replies

3. Shell Programming and Scripting

Sed: replace content from file with the content from file

Hi, I am having trouble while using 'sed' with reading files. Please help. I have 3 files. File A, file B and file C. I want to find content of file B in file A and replace it by content in file C. Thanks a lot!! Here is a sample of my question. e.g. (file A: a.txt; file B: b.txt; file... (3 Replies)
Discussion started by: dirkaulo
3 Replies

4. Shell Programming and Scripting

Using dirname on a file before uniq

Hello, I have a list of files generated like this: find dir -type f > file_list I want to get a list of just the unique directories. I can't create a temporary file. So the idea is to do a working equivalent to this: cat file_list | dirname | uniq But of course that doesn't... (4 Replies)
Discussion started by: brsett
4 Replies

5. UNIX for Advanced & Expert Users

$cat file|sort|uniq

I have a file with list of redundant server names and want only unique ones of those. I have used following command but still redudant ones are listing $cat file|sort|uniq where could be the problem. Thanks, Srinivas (3 Replies)
Discussion started by: srinivas Juluri
3 Replies

6. UNIX for Dummies Questions & Answers

Several file comparison not uniq or comm command

When comparing several files is there a way to find values unique to each file? File1 a b c d File2 a b t File 3 a (3 Replies)
Discussion started by: dr_sabz
3 Replies

7. UNIX for Dummies Questions & Answers

find uniq lines in file, using the first field of line

Hello all, new to unix and have just found the forum. I think I will be here quite often, and hope that in time i will be able to provide soem help, role on not being a newbie anymore :) I have a question which iI am hoping someone could help me with. If i have a file with lines in in thus... (8 Replies)
Discussion started by: grom
8 Replies

8. UNIX for Dummies Questions & Answers

How to uniq third field in a file

Hi ; I have a question regarding the uniq command in unix How do I uniq 3rd field in a file ? original file : zoom coord 39 18652 39 18652 zoom coord 39 18653 39 18653 zoom coord 39 18818 39 18818 zoom coord 39 18840 39 18840 zoom coord 41 15096 41 15096 zoom... (1 Reply)
Discussion started by: babycakes
1 Replies

9. Shell Programming and Scripting

compare two col from 2 files, and output uniq from file 1

Hi, I can't find how to achive such thing, please help. I have try with uniq and comm but those command can't compare columns just whole lines, I think awk will be the best but awk is magic for me as of now. file a a1~a2~a3~a4~a6~a7~a8 file b b1~b2~b3~b4~b6~b7~b8 output 1: compare... (2 Replies)
Discussion started by: pp56825
2 Replies

10. UNIX for Dummies Questions & Answers

inserting uniq sequential numbers at the start of the file

Hi Unix gurus, I have a file. I need to insert sequential number at the starting of the file. Fields are delimited by "|". I know the starting number. Example: File is as follows |123|4test|test |121|2test|test |x12|1test|test |vd123|5test|test starting number is : 120 ... (7 Replies)
Discussion started by: jingi1234
7 Replies
Login or Register to Ask a Question