Sponsored Content
Top Forums Shell Programming and Scripting How to remove duplicated column in a text file? Post 302931576 by huiyee1 on Wednesday 14th of January 2015 02:27:23 AM
Old 01-14-2015
How to remove duplicated column in a text file?

Dear all,

How can I remove duplicated column in a text file?

Input:
Code:
LG10_PM_map_19_LEnd	1000560	G	AA	AA	AA	AA	AA	GG
LG10_PM_map_19_LEnd	1005621	G	GG	GG	GG	AA	AA	GG
LG10_PM_map_19_LEnd	1011214	A	AA	AA	AA	AA	GG	GG
LG10_PM_map_19_LEnd	1011673	T	TT	TT	TT	TT	CC	CC
LG10_PM_map_19_LEnd	1088961	C	TT	TT	TT	TT	TT	TT
LG10_PM_map_19_LEnd	1089024	G	AA	AA	AA	AA	AA	AA
LG10_PM_map_19_LEnd	1108301	C	TT	TT	TT	TT	TT	CC
LG10_PM_map_19_LEnd	11365128	G	AA					
LG10_PM_map_19_LEnd	11365170	T	CC					
LG10_PM_map_19_LEnd	11381744	A	GG	GG	GG	GG	GG	GG
LG10_PM_map_19_LEnd	11381772	T	TT	TT				
LG10_PM_map_19_LEnd	11385851	A	AA	AA	AA	AA	AA	AA
LG10_PM_map_19_LEnd	11386265	A	AA	AA	AA	AA	AA	AA
LG10_PM_map_19_LEnd	1138663	T	AA	TT	AA	AA	TT	TT

Output:
Code:
LG10_PM_map_19_LEnd	1000560	G	AA	GG
LG10_PM_map_19_LEnd	1005621	G	GG	AA
LG10_PM_map_19_LEnd	1011214	A	AA	GG
LG10_PM_map_19_LEnd	1011673	T	TT	CC
LG10_PM_map_19_LEnd	1088961	C	TT
LG10_PM_map_19_LEnd	1089024	G	AA
LG10_PM_map_19_LEnd	1108301	C	TT	CC
LG10_PM_map_19_LEnd	11365128	G	AA			
LG10_PM_map_19_LEnd	11365170	T	CC
LG10_PM_map_19_LEnd	11381744	A	GG
LG10_PM_map_19_LEnd	11381772	T	TT			
LG10_PM_map_19_LEnd	11385851	A	AA
LG10_PM_map_19_LEnd	11386265	A	AA
LG10_PM_map_19_LEnd	1138663	T	AA	TT

---------- Post updated at 02:27 AM ---------- Previous update was at 01:24 AM ----------

Hi all,

I figured out with awk:

Code:
awk '{ while(++i<=NF) printf (!a[$i]++) ? $i FS : ""; i=split("",a); print ""}' input > output

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicated xml record in a file under unix

Hi, If i have a file with xml format, i would like to remove duplicated records and save to a new file. Is it possible...to write script to do it? (8 Replies)
Discussion started by: happyv
8 Replies

2. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies

3. Shell Programming and Scripting

remove duplicated columns

hi all, i have a file contain multicolumns, this file is sorted by col2 and col3. i want to remove the duplicated columns if the col2 and col3 are the same in another line. example fileA AA BB CC DD CC XX CC DD BB CC ZZ FF DD FF HH HH the output is AA BB CC DD BB CC ZZ FF... (6 Replies)
Discussion started by: kamel.seg
6 Replies

4. Shell Programming and Scripting

Remove part of a column in a text file

I have a text file with delimiter "|" and sometimes the zipcode is in "5th" column or "6th" column. I want to scan the file and remove the "-2323" from the zipcode which has zip+4 digits From blah|blah|foo|bar||blah|945523-232|USA blah|blah|foo|bar||blah|foo|94555-2323|USA To... (8 Replies)
Discussion started by: gubbu
8 Replies

5. UNIX for Dummies Questions & Answers

How to remove duplicated based on longest row & largest value in a column

Hii i have a file with data as shown below. Here i need to remove duplicates of the rows in such a way that it just checks for 2,3,4,5 column for duplicates.When deleting duplicates,retain largest row i.e with many columns with values should be selected.Then it must remove duplicates such that by... (11 Replies)
Discussion started by: reva
11 Replies

6. Shell Programming and Scripting

Help with remove duplicated content

Input file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hcmv-UL70-3p hsa-4486 hcms-US25 hsa-360-5 hcms-US25 hsa-4 hcms-US25 hsa-458 hcms-US25 hsa-44812 . . Desired Output file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hsa-4486... (3 Replies)
Discussion started by: perl_beginner
3 Replies

7. UNIX for Dummies Questions & Answers

Sort csv file by duplicated column value

hello, I have a large file (about 1gb) that is in a file similar to the following: I want to make it so that I can put all the duplicates where column 3 (delimited by the commas) are shown on top. Meaning all people with the same age are listed at the top. The command I used was ... (3 Replies)
Discussion started by: jl487
3 Replies

8. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

9. Shell Programming and Scripting

How to delete 'duplicated' column values and make a delimited file too?

Hi, I have the following output from an Oracle SQL statement and I want to remove duplicated column values. I know it is possible using Oracle analytical/statistical functions but unfortunately I don't know how to use any of those. So now, I've gone to PLAN B using awk/sed maybe or any... (5 Replies)
Discussion started by: newbie_01
5 Replies

10. Shell Programming and Scripting

awk script to append suffix to column when column has duplicated values

Please help me to get required output for both scenario 1 and scenario 2 and need separate code for both scenario 1 and scenario 2 Scenario 1 i need to do below changes only when column1 is CR and column3 has duplicates rows/values. This inputfile can contain 100 of this duplicated rows of... (1 Reply)
Discussion started by: as7951
1 Replies
term::interact::pager(n)					 Terminal control					  term::interact::pager(n)

__________________________________________________________________________________________________________________________________________________

NAME
term::interact::pager - Terminal widget, paging SYNOPSIS
package require Tcl 8.4 package require term::interact::pager ?0.1? term::interact::pager object text ?options...? object interact object done object clear object text text object configure object configure option object configure option value... object cget option _________________________________________________________________ DESCRIPTION
This package provides a class for the creation of a simple paging text display. CLASS API
The package exports a single command, the class command, enabling the creation of pager instances. Its API is: term::interact::pager object text ?options...? This command creates a new pager object with the name object, initializes it, and returns the fully qualified name of the object command as its result. The argument is the text to show, possibly followed by configuration options and their values. The options are explained in the sec- tion Configuration. OBJECT API
The objects created by the class command provide the methods listed below: object interact Show the pager in the screen at the configured location and start interacting with it. This opens its own event loop for the pro- cessing of incoming characters. The method returns when the interaction has completed. See section Interaction for a description of the possible interaction. object done This method can be used by user supplied actions to terminate the interaction with the object. object clear This method can be used by user supplied actions to remove the pager from the terminal. object text text This method can be used to change the text shown by the pager. The pager will reset the dispay to show the first line of the text at the top. object configure object configure option object configure option value... object cget option Standard methods to retrieve and configure the options of the pager. CONFIGURATION
A pager instance recognizes the following options: -in chan Specifies the channel to read character sequences from. Defaults to stdin. -out chan Specifies the channel to write the pager contents to. Defaults to stdout. -column int Specifies the column of the terminal where the left margin of the pager display should appear. Defaults to 0, i.e. the left-most column. -line int Specifies the line of the terminal where the top margin of the pager display should appear. Defaults to 0, i.e. the top-most line. -height int Specifies the number of lines of text to show at most in the display. Defaults to 25. -actions dict Specifies a dictionary containing additional actions, using character sequences as keys. Note that these sequences cannot override the hardwired sequences described in section Interaction. INTERACTION
A pager object recognizes the control sequences listed below and acts as described. The user can supply more control sequences to act on via the configuration, but is not able to overide these defaults. Cursor Up The text is scrolled down a single line, making one more line visible at the top. The pager will not react if the first line of the text is already shown. Cursor Down The text is scrolled up a single line, making one more line visible at the bottom. The pager will not react if the last line of the text is already shown. Page Up The text is scrolled down a page. The pager will not react if the first line of the text is already shown. Page Down The text is scrolled up a page. The pager will not react if the last line of the text is already shown. Enter/Return The interaction with the object is terminated. BUGS, IDEAS, FEEDBACK This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category term of the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have for either package and/or documentation. KEYWORDS
control, pager, terminal, text display CATEGORY
Terminal control COPYRIGHT
Copyright (c) 2006 Andreas Kupries <andreas_kupries@users.sourceforge.net> term 0.1 term::interact::pager(n)
All times are GMT -4. The time now is 07:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy