Hi all,
I have a tab-delimited file and want to remove identical lines, i.e. all of line 1,2,4 because the columns are the same as the columns in other lines. Any input is appreciated.
abc gi4597 9997 cgcgtgcg $%^&*()()*
abc gi4597 9997 cgcgtgcg $%^&*()()*
ttt ... (1 Reply)
I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Hi,
How can I remove duplicates from a file based on group on other column? for example:
Test1|Test2|Test3|Test4|Test5
Test1|Test6|Test7|Test8|Test5
Test1|Test9|Test10|Test11|Test12
Test1|Test13|Test14|Test15|Test16
Test17|Test18|Test19|Test20|Test21
Test17|Test22|Test23|Test24|Test5
... (2 Replies)
Hi All,
i have input file like below...
CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;;
CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;;
CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Hi ,
Some time i got duplicated value in my files ,
bundle_identifier= B
Sometext=ABC
bundle_identifier= A
bundle_unit=500
Sometext123=ABCD
bundle_unit=400
i need to check if there is a duplicated values or not if yes , i need to check if the value is A or B when Bundle_Identified ,... (2 Replies)
I have a 13gb file. It has the following columns:
The 3rd column is basically correlation values. I want to delete those rows which are repeated between the columns:
A B 0.04
B C 0.56
B B 1
A A 1
C D 1
C C 1
Desired Output: (preferably in a .csv format
A,B,0.04
B,C,0.56
C,D,1... (3 Replies)
Dear community,
I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns
The data are like this:
Region 23/11/2014 09:11:36 41752
Medio 23/11/2014 03:11:38 4132
Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies
LEARN ABOUT CENTOS
column
COLUMN(1) User Commands COLUMN(1)NAME
column - columnate lists
SYNOPSIS
column [options] file...
DESCRIPTION
The column utility formats its input into multiple columns. Rows are filled before columns. Input is taken from file or, by default, from
standard input. Empty lines are ignored.
OPTIONS -c, --columns width
Output is formatted to a width specified as number of characters.
-t, --table
Determine the number of columns the input contains and create a table. Columns are delimited with whitespace, by default, or with
the characters supplied using the separator. Table output is useful for pretty-printing.
-s, --separator separators
Specify possible table delimiters (default is whitespace).
-o, --output-separator separators
Specify table output delimiter (default is two whitespaces).
-x, --fillrows
Fill columns before filling rows.
-h, --help
Print help and exit.
ENVIRONMENT
The environment variable COLUMNS is used to determine the size of the screen if no other information is available.
EXAMPLES
sed 's/#.*//' /etc/fstab | column -t
BUGS
The util-linux version 2.23 changed -s option to be non-greedy, for example:
$ printf "a:b:c
1::3
" | column -t -s ':'
old output:
a b c
1 3
new output (since util-linux 2.23)
a b c
1 3
SEE ALSO colrm(1), ls(1), paste(1), sort(1)HISTORY
The column command appeared in 4.3BSD-Reno.
AVAILABILITY
The column command is part of the util-linux package and is available from ftp://ftp.kernel.org/pub/linux/utils/util-linux/.
util-linux October 2010 COLUMN(1)