Sponsored Content
Top Forums Shell Programming and Scripting remove duplicate words in a line Post 302298706 by ShawnMilo on Wednesday 18th of March 2009 11:10:56 AM
Old 03-18-2009
Code:
#!/usr/bin/env python

for line in open('temp.txt', 'r'):
    seen = []
    words = line.rstrip('\n').split()

    for word in words:
        if not word in seen:
            print word,
            seen.append(word)
    print

Output:

Code:
# cat temp.txt
pink_kite red_pen ball pink_kite ball
yellow_flower white no white no
cloud nine_pen pink cloud pink nine_pen
brown_ball white
red_bear green red_bear
white no

# python temp.py
pink_kite red_pen ball
yellow_flower white no
cloud nine_pen pink
brown_ball white
red_bear green
white no

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies

2. UNIX for Dummies Questions & Answers

Identify duplicate words in a line using command

Hi, Let me explain the problem clearly: Let the entries in my file be: lion,tiger,bear apple,mango,orange,apple,grape unix,windows,solaris,windows,linux red,blue,green,yellow orange,maroon,pink,violet,orange,pink Can we detect the lines in which one of the words(separated by field... (8 Replies)
Discussion started by: srinivasan_85
8 Replies

3. Shell Programming and Scripting

remove first few words from a line

Hi All, Sample: 4051 Oct 4 10:03:36 AM 2008: TEST: end of testcase Checking Interface after reload, result fail I need to remove first 10 words of the above line and output should be like Checking Interface after reload, result fail Please help me in this regard. Thanks, (4 Replies)
Discussion started by: shellscripter
4 Replies

4. Shell Programming and Scripting

Remove all words after first space from each line

My file looks like: asd absjdd sdff vczxs wedssx c dasx ccc I need to keep asd sdff wedssx dasx How do I do that experts?:wall::wall: (1 Reply)
Discussion started by: hakermania
1 Replies

5. Shell Programming and Scripting

Remove very first pair of duplicate words

I have file which is almost look like below MMIT MMIT ... (2 Replies)
Discussion started by: manas_ranjan
2 Replies

6. Shell Programming and Scripting

Remove duplicate line on condition

Hi Ive been scratching over this for some time with no solution. I have a file like this 1 bla bla 1 2 bla bla 2 4 bla bla 3 5 bla bla 1 6 bla bla 1 I want to remove consecutive occurrences of lines like bla bla 1, but the first column may be different. Any ideasss?? (23 Replies)
Discussion started by: jamie_123
23 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Two Line Pairs?

So I have a bunch of files that look like this >gi|33332323 MMKCRGVIMVVEKVMKRDGRIVPFDESRIRWAVQ--- >gi|45235353 MMKCR----VEKMRDVFFDESIRWAVQ They go on...sequences are much longer but all in two line (fasta) format. I want to remove duplicate pairs of ID(GI) number and sequence. I tried... (12 Replies)
Discussion started by: bakere19
12 Replies

8. Shell Programming and Scripting

Remove duplicate entries from the same line

Hello, I have a file which have several duplicate entries on the same line: File ID source 1 GM GF GM 2 GM GF GM GF GM GF GM GF GM GF 3 GM GF GM SF GM GF GM SF 4 FF FF FF FF 5 FF GM FF ... (2 Replies)
Discussion started by: nans
2 Replies

9. Shell Programming and Scripting

Remove last few words from Line

Hi I would like to remove last few words from File Could anybody Help on it. ps -ef | grep mgr.prm | awk '{print $10}' /opt/app/dummyd/xyz/dirprm/mgr.prm /opt/app/dummy/xyz/dirprm/mgr.prm /opt/app/dummy/xyz/dirprm/mgr.prm I want output like /opt/app/dummyd/xyz... (4 Replies)
Discussion started by: tapia
4 Replies

10. UNIX for Dummies Questions & Answers

Remove duplicate words from column 1

Tried using sed and uniq but it's removing the entire line. Can't seem to figure a way to just remove the word. Any help is appreciated. I have a file: dog, text1, text2, text3 dog, text1, text2, text3 dog, text1, text2, text3 cat, text1, text2, text3 Trying to remove all duplicate instances... (6 Replies)
Discussion started by: jimmyf
6 Replies
secolor.conf(5) 						File Formats Manual						   secolor.conf(5)

NAME
secolor.conf - The SELinux color configuration file. DESCRIPTION
This optional file controls the color to be associated to the context components associated to the raw context passed by selinux_raw_con- text_to_color(3), when context related information is to be displayed in color by an SELinux-aware application. selinux_raw_context_to_color(3) obtains this color information from the active policy secolor.conf file as returned by selinux_col- ors_path(3). FILE FORMAT
The file format is as follows: color color_name = #color_mask [...] context_component string = fg_color_name bg_color_name [...] Where: color The color keyword. Each color entry is on a new line. color_name A single word name for the color (e.g. red). color_mask A color mask starting with a hash (#) that describes the hexadecimal RGB colors with black being #000000 and white being #ffffff. context_component The context component name that must be one of the following: user, role, type or range Each context_component string ... entry is on a new line. string This is the context_component string that will be matched with the raw context component passed by selinux_raw_context_to_color(3). A wildcard '*' may be used to match any undefined string for the user, role and type context_component entries only. fg_color_name The color_name string that will be used as the foreground color. A color_mask may also be used. bg_color_name The color_name string that will be used as the background color. A color_mask may also be used. EXAMPLES
Example 1 entries are: color black = #000000 color green = #008000 color yellow = #ffff00 color blue = #0000ff color white = #ffffff color red = #ff0000 color orange = #ffa500 color tan = #D2B48C user * = black white role * = white black type * = tan orange range s0-s0:c0.c1023 = black green range s1-s1:c0.c1023 = white green range s3-s3:c0.c1023 = black tan range s5-s5:c0.c1023 = white blue range s7-s7:c0.c1023 = black red range s9-s9:c0.c1023 = black orange range s15:c0.c1023 = black yellow Example 2 entries are: color black = #000000 color green = #008000 color yellow = #ffff00 color blue = #0000ff color white = #ffffff color red = #ff0000 color orange = #ffa500 color tan = #d2b48c user unconfined_u = #ff0000 green role unconfined_r = red #ffffff type unconfined_t = red orange user user_u = black green role user_r = white black type user_t = tan red user xguest_u = black yellow role xguest_r = black red type xguest_t = black green user sysadm_u = white black range s0:c0.c1023 = black white user * = black white role * = black white type * = black white SEE ALSO
selinux(8), selinux_raw_context_to_color(3), selinux_colors_path(3) SELinux API documentation 08 April 2011 secolor.conf(5)
All times are GMT -4. The time now is 10:33 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy