Login or Register to Ask a Question and Join Our Community


Red Hat

filtering the recursive ids in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Linux Red Hat filtering the recursive ids in a file
# 1  
Old 05-07-2009
filtering the recursive ids in a file
i have a file which contains the data as below,

789990
789990
789990
789990
334989
334989
789990
-- here the no 789990 is repeated 5 times and 334989 is 2 times.
i want the output to be,

789990
334989 i.e only one entry of the repeated nos.
# 2  
Old 05-07-2009
check man uniq

alternatively, if you have Python
Code:
# python -c "d=open('file').read().split();print ' '.join((set(d)))"
334989 789990

# 3  
Old 05-07-2009
i checked with python, but it throws the below error

Traceback (most recent call last):
File "<string>", line 1, in ?
NameError: name 'set' is not defined
# 4  
Old 05-07-2009
it means you have older version of Python.
Code:
#!/usr/bin/env python
d={}
data=open("file").read().split()
for i in data: d[i]=""
print d.keys()

# 5  
Old 05-07-2009
sort command may help in this case

Code:
sort -u yourfile.dat

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

List of all ids,groups, privilege ids

I wish to pull out a list of all user ids on the system, including the privileged ids, the groups to which they belong to. Sometimes after deleting an id also, its home dir does not get deleted or an entry is left behind in /etc/passwd. Can someone help me with a script to achieve both. (2 Replies)
Discussion started by: ggayathri
2 Replies

2. Shell Programming and Scripting

Recursive delete in a file using a set of values in other file

Hi, I have got a file with 6K records and I want to delete 500 records from this file which match the values present in another file. Format of the both the files is different. Example : File 1 record CCCCCC 11292562ABCDEF MBR/PSF6/108100502/BEN01XXX XXX Example : File 2 record... (1 Reply)
Discussion started by: Nikhath
1 Replies

3. Shell Programming and Scripting

Inserting IDs from a text file into a sequence alignment file

Hi, I have one file with one column and several hundred entries File1: NA1 NA2 NA3And now I need to run a command within a mapping aligner tool to insert these sample names into a sequence alignment file (SAM) such that they look like this @RG ID:Library1 SM:NA1 PL:Illumina ... (7 Replies)
Discussion started by: nans
7 Replies

4. Shell Programming and Scripting

Masking email ids / phone no's in file along with obscene words

Hi, I would like to know if there is a way to mask obscene words and other contents like email id's/phone numbers in the file. Below is the sample input /output. Sample data : cat smp.txt The service is really bad . My email abc@gmail.com You can contact me at 4078909831 Output... (2 Replies)
Discussion started by: ashwin3086
2 Replies

5. Shell Programming and Scripting

Match ids and print original file

Hello, I have two files Original: ( 5000 entries) Chr Position chr1 879108 chr1 881918 chr1 896874 ... and a file with allele freq ( 2000 entries) Chr Position MAF chr1 881918 0.007 chr1 979748 0.007 chr1... (9 Replies)
Discussion started by: nans
9 Replies

6. Shell Programming and Scripting

Recursive file processing from a path and printing output in a file

Hi All, The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the std... (1 Reply)
Discussion started by: Optimus81
1 Replies

7. UNIX for Dummies Questions & Answers

Recursive file organization?

Does anyone have any idea of how I can make something like the code below run recursively? I'll run it on a tree of directories all with different names and all containing a sequence of .dpx files. I've tried to do it using find and exec but can't get it to work right. What it needs to do is... (4 Replies)
Discussion started by: scribling
4 Replies

8. Shell Programming and Scripting

Send a mail to IDs listed in a text file

I have a list of mail ids in text file and want a ksh script that reads this text file and sends a mail to all mail ids with same subject line and content. I am using UX-HP machine and KSH. Thanks for help in advance! (5 Replies)
Discussion started by: Sriranga
5 Replies

9. Filesystems, Disks and Memory

Recursive file processing

I'm developing a generic script to catenate all files found in a given directory. The problem I've got is that the depth of the given directory is unknown, so I end up with some catenated directories. My unix isn't that hot and I'm at a loss. Heres an extract from the script cd $1 for i in... (4 Replies)
Discussion started by: k_mufasa
4 Replies

10. UNIX for Advanced & Expert Users

Script which can send file to diffrent mail ids.

Hi i am looking for the script which can send file to different mailids, please halp me out. Thanks in advance. (3 Replies)
Discussion started by: vpandey
3 Replies
Login or Register to Ask a Question