Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Search based on 1,2,4,5 columns and remove duplicates in the same file.
Top Forums Shell Programming and Scripting Search based on 1,2,4,5 columns and remove duplicates in the same file. Post 302465984 by ygemici on Monday 25th of October 2010 05:00:22 AM
Old 10-25-2010
Code:
for i in `sed = file1 | sed -e N -e 's/\(.*\)\n\(.*\)/\1  \2/' 
| sed -e 's/\(.*\),\(.*\),.*,\(.*\),\(.*\),.*,.*/\1\2\3\4/' 
| sed 'N;s/\([0-9]*\)  \(.*\)\n[0-9]*  \2/\1 \2/' 
| sed 's/\([0-9]*\).*/\1/'`
   do 
    sed -n "$i p" file1 
   done
"1","ccc","information","5000","temp","concept","new"
"1","ddd","information","6000","temp","concept","new"
"1","aaa","information","4000","temp","concept","new"
"1","bbb","information","4000","temp","concept","new"
"1","abc","information","7000","temp","concept","new"

 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicates based on a column in fixed width file

Hi, How to output the duplicate record to another file. We say the record is duplicate based on a column whose position is from 2 and its length is 11 characters. The file is a fixed width file. ex of Record: DTYU12333567opert tjhi kkklTRG9012 The data in bold is the key on which... (1 Reply)
Discussion started by: Qwerty123
1 Replies

2. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

3. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies

4. Shell Programming and Scripting

finding duplicates in csv based on key columns

Hi team, I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record. can one help me on finding the duplicates, Thanks in advance. ... (2 Replies)
Discussion started by: baskivs
2 Replies

5. UNIX for Dummies Questions & Answers

remove duplicates based on a field and criteria

Hi, I have a file with fields like below: A;XYZ;102345;222 B;XYZ;123243;333 C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 desired output: C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 ie, if the 4rth field is a duplicate.. i need only those records where... (5 Replies)
Discussion started by: wanderingmind16
5 Replies

6. Shell Programming and Scripting

Remove duplicates based on query and subject fields from blast output file

Hi all I have a blast outfile file like this : NZ_1540841_1561981 ICMP_1687819_1695946 92.59 27 2 0 12826 12852 3136 3162 0.28 38.2 NZ_1540841_1561981 ICMP_1687819_1695946 95.65 23 1 0 12268 12290 5815 5837 0.28 38.2 NZ_1540841_1561981 ICMP_3674888_3676546 82.70 185 32 0 9454 9638 11 195 6e-24 ... (2 Replies)
Discussion started by: pbioinfo
2 Replies

7. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

8. Shell Programming and Scripting

Removing duplicates from delimited file based on 2 columns

Hi guys,Got a bit of a bind I'm in. I'm looking to remove duplicates from a pipe delimited file, but do so based on 2 columns. Sounds easy enough, but here's the kicker... Column #1 is a simple ID, which is used to identify the duplicate. Once dups are identified, I need to only keep the one... (2 Replies)
Discussion started by: kevinprood
2 Replies

9. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies

LEARN ABOUT DEBIAN

xlhtml

xlhtml(1)						      General Commands Manual							 xlhtml(1)

NAME

       xlhtml - A program for converting Microsoft Excel Files .xls

SYNOPSIS

       xlhtml  [-a] [-asc] [-csv] [-xml] [-bcNNNNNN] [-bi/path] [-c] [-dp] [-v] [-fw] [-m] [-nc] [-nh] [-tcNNNNNN] [-te] [-xc:N-N] [-xp:N] [-xr:N-
       N] FILE

DESCRIPTION

       This manual page explains the xlhtml program. The program xlhtml is used to convert Microsoft Excel Spreadsheet files into either  html	or
       tab  delimitted ASCII. The program can be interfaced with helper scripts for viewing email attachments. Most use of this program is through
       the helper scripts and one would probably rarely resort to using the commandline interface.

OPTIONS

       -a     aggressively optimize html by removing </TR> </TD> or VALIGN="bottom". Some older browsers may not display properly in this mode.

       -asc   Ascii out of -dp and extraction data (-xc, -xp, -xr)

       -csv   Output in Comma Separated Values of -dp and extraction data (-xc, -xp, -xr)

       -xml   Output in XML of -dp and extraction data (-xc, -xp, -xr)

       -bc    Override the background color. e.g. -bc808080	for gray

       -bi    Use background image. e.g. -bi/home/httpd/icon/tar.gif

       -c     Centers the tables horizontally

       -dp    Dump page count and max columns and rows per page

       -v     Prints program version

       -fw    suppress formula warnings about accuracy

       -m     No encoding for multibyte

       -nc    tells it not to colorize the output.

       -nh    Suppress header and body tags in html output

       -tc    Override the text color. e.g. -tcFF0000	  for red

       -te    Trims empty rows & columns at the edges of a worksheet

       -xc    Columns (separated by a dash) for extraction (zero based)

       -xp    Page for extraction (zero based), one page only

       -xr    Rows (separated by a dash) to be extracted (zero based)

       An example of the extraction command line is: xlhtml -fw -asc -xp:0 -xr:2-6 -xc:0-1 Test.xls

       The extraction output is: Formatted output of cells by column left to right, columns separated by a tab, end of row is: 0x0A, end of  file:
       



AUTHOR

       Steve Grubb, Charles N Wyble

xlhtml								   May 15, 2002 							 xlhtml(1)
All times are GMT -4. The time now is 06:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy