Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicate line detail based on column one data
Top Forums Shell Programming and Scripting Remove duplicate line detail based on column one data Post 302384637 by gaurav1086 on Wednesday 6th of January 2010 03:45:02 AM
Old 01-06-2010
Quote:
Originally Posted by patrick87
My input file:
Code:
AVI.out     <detail>named as the RRM .</detail>
AVI.out     <detail>Contains 1 RRM .</detail>

AR0.out     <detail>named as the tellurite-resistance.</detail>

AWG.out     <detail>Contains 2 HTH .</detail>

ADV.out     <detail>named as the DENR family.</detail>
ADV.out     <detail>Contains 1 SUI1 domain.</detail>

AH7.out     <detail>Contains 1 box .</detail>
AH7.out     <detail>Contains 11 rich.</detail>

AZM.out     <detail>named as the family.</detail>

My desired output file:
Code:
AVI.out     <detail>named as the RRM .</detail>

AR0.out     <detail>named as the tellurite-resistance.</detail>

AWG.out     <detail>Contains 2 HTH .</detail>

ADV.out     <detail>named as the DENR family.</detail>

AH7.out     <detail>Contains 1 box .</detail>

AZM.out     <detail>named as the family.</detail>

I would like to remove those detail that appear at the second and keep only those detail at the first row.
Thanks for advice.
try this,
Code:
awk 'n!=$1{print;n=$1}' infile

regards
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

2. Shell Programming and Scripting

remove new line characters from a partcular column data

Dear friends, I have a pipe delimited file having 5 columns. However the column no-3 is having extra new line characters as the data owing to owing , I am having issues. Ideally my file should have only newline termination at the end of each record and not within column data of any of... (1 Reply)
Discussion started by: sureshg_sampat
1 Replies

3. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

4. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric... (2 Replies)
Discussion started by: davegen
2 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

6. Shell Programming and Scripting

Help with figuring division and addition based on column data and line numbers

I have a data file in the format of 1234 xxx 1234 xxx 1234 xxx 1234 xxxI want to be able to calculate the following - COLUMN1+((LINENUMBER-1)/365) The output needs to preserve the 2nd column - 1234 xxx 1234.00274 xxx 1234.00548 xxx What is the best way to do this? I am somewhat... (9 Replies)
Discussion started by: ncwxpanther
9 Replies

7. Shell Programming and Scripting

How to remove a line based on contents of the first column?

Good day all. Using basic UNIX/Linux tools, how would you delete a line based on a character found in column 1? For example, if the CITY name contains an 'a' or 'A', delete the line: New York City; New York Los Angeles; California Chicago; Illinois Houston; Texas Philadelphia;... (3 Replies)
Discussion started by: BRH
3 Replies

8. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies

9. Shell Programming and Scripting

Remove sections based on duplicate first line

Hi, I have a file with many sections in it. Each section is separated by a blank line. The first line of each section would determine if the section is duplicate or not. if the section is duplicate then remove the entire section from the file. below is the example of input and output.... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

10. Shell Programming and Scripting

To append new data at the end of each line based on substring of last column

Hi guys, I need to append new data at the end of each line of the files. This new data is based on substring (3rd fields) of last column. Input file xxx.csv: U1234|1-5X|orange|1-5X|Act|1-5X|0.1 /sac/orange 12345 0 U5678|1-7X|grape|1-7X|Act|1-7X|0.1 /sac/grape 5678 0... (5 Replies)
Discussion started by: null7
5 Replies

LEARN ABOUT SUSE

pia

pia(1)							      General Commands Manual							    pia(1)

NAME

       pia - play media files

SYNOPSIS

       pia [ options ] movie

DESCRIPTION

       pia is a small X11 tool which plays movie files.

OPTIONS

       pia understands the usual toolkit options (-geometry + friends).  Additional options are:

       -h     display help text

       -v     be verbose.

       -debug enable debug messages.

       -dsp <dev>
	      Use OSS device <dev> for sound.

       -noxv  Disable Xvideo extension.

       -nogl  Disable OpenGL.

       -noaudio
	      Don't play sound.

       -novideo
	      Don't play video.

       -slow n
	      Slowdown video playback by factor n.  n=2 doubles playback time, ...  This also turns off audio playback.

SUPPORTED MOVIE FORMATS

       Main  purpose  of this utility is to playback movie files recorded by xawtv, motv and streamer.	It should be able to playback every AVI or
       QuickTime movie written by one of the mentioned utilities.  Feel free to submit a bug report if this doesn't work for some file.

   QuickTime
       pia uses libquicktime to decode quicktime movies, thus it should be able to decode  and	playback  all  movies  with  codecs  supported	by
       libquicktime.

   AVI
       AVI  support  is  very  limited.   Video:  uncompressed RGB data (15 and 24 bpp) and mjpeg is supported.  Audio: only uncompressed PCM data
       works.

   WAV
       WAV files are playable too, with the same limitation like AVI audio: only uncompressed PCM data.

AUTHOR

       Gerd Knorr <kraxel@bytesex.org>

COPYRIGHT

       2002 Gerd Knorr

       Copyright (C) 2002 Gerd Knorr <kraxel@bytesex.org>

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as  published	by
       the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

       This  program  is  distributed  in  the	hope  that  it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER-
       CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program; if not, write to the Free  Software  Foundation,
       Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

								(c) 2002 Gerd Knorr							    pia(1)
All times are GMT -4. The time now is 04:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy