Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate line detail based on column one data Post 302384637 by gaurav1086 on Wednesday 6th of January 2010 03:45:02 AM
Old 01-06-2010
Quote:
Originally Posted by patrick87
My input file:
Code:
AVI.out     <detail>named as the RRM .</detail>
AVI.out     <detail>Contains 1 RRM .</detail>

AR0.out     <detail>named as the tellurite-resistance.</detail>

AWG.out     <detail>Contains 2 HTH .</detail>

ADV.out     <detail>named as the DENR family.</detail>
ADV.out     <detail>Contains 1 SUI1 domain.</detail>

AH7.out     <detail>Contains 1 box .</detail>
AH7.out     <detail>Contains 11 rich.</detail>

AZM.out     <detail>named as the family.</detail>

My desired output file:
Code:
AVI.out     <detail>named as the RRM .</detail>

AR0.out     <detail>named as the tellurite-resistance.</detail>

AWG.out     <detail>Contains 2 HTH .</detail>

ADV.out     <detail>named as the DENR family.</detail>

AH7.out     <detail>Contains 1 box .</detail>

AZM.out     <detail>named as the family.</detail>

I would like to remove those detail that appear at the second and keep only those detail at the first row.
Thanks for advice.
try this,
Code:
awk 'n!=$1{print;n=$1}' infile

regards
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

2. Shell Programming and Scripting

remove new line characters from a partcular column data

Dear friends, I have a pipe delimited file having 5 columns. However the column no-3 is having extra new line characters as the data owing to owing , I am having issues. Ideally my file should have only newline termination at the end of each record and not within column data of any of... (1 Reply)
Discussion started by: sureshg_sampat
1 Replies

3. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

4. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric... (2 Replies)
Discussion started by: davegen
2 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

6. Shell Programming and Scripting

Help with figuring division and addition based on column data and line numbers

I have a data file in the format of 1234 xxx 1234 xxx 1234 xxx 1234 xxxI want to be able to calculate the following - COLUMN1+((LINENUMBER-1)/365) The output needs to preserve the 2nd column - 1234 xxx 1234.00274 xxx 1234.00548 xxx What is the best way to do this? I am somewhat... (9 Replies)
Discussion started by: ncwxpanther
9 Replies

7. Shell Programming and Scripting

How to remove a line based on contents of the first column?

Good day all. Using basic UNIX/Linux tools, how would you delete a line based on a character found in column 1? For example, if the CITY name contains an 'a' or 'A', delete the line: New York City; New York Los Angeles; California Chicago; Illinois Houston; Texas Philadelphia;... (3 Replies)
Discussion started by: BRH
3 Replies

8. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies

9. Shell Programming and Scripting

Remove sections based on duplicate first line

Hi, I have a file with many sections in it. Each section is separated by a blank line. The first line of each section would determine if the section is duplicate or not. if the section is duplicate then remove the entire section from the file. below is the example of input and output.... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

10. Shell Programming and Scripting

To append new data at the end of each line based on substring of last column

Hi guys, I need to append new data at the end of each line of the files. This new data is based on substring (3rd fields) of last column. Input file xxx.csv: U1234|1-5X|orange|1-5X|Act|1-5X|0.1 /sac/orange 12345 0 U5678|1-7X|grape|1-7X|Act|1-7X|0.1 /sac/grape 5678 0... (5 Replies)
Discussion started by: null7
5 Replies
avisplit(1)						      General Commands Manual						       avisplit(1)

NAME
avisplit - split AVI-files into chunks of a maximum size SYNOPSIS
avisplit [ -i file -o base [ -s size ] [ -H num ] [ -t s1-s2[,s3-s4,..] -c -m -b num -f commentfile ] ] [ -v ] COPYRIGHT
avisplit is Copyright (C) by Thomas Oestreich. DESCRIPTION
avisplit splits a single AVI-file into chunks of size size. Each of the created chunks will be an independent file, i.e. it can be played without needing any other of the chunk. OPTIONS
-i file Specify the filename of the file to split into chunks. -o base Specify the base of the output filename(s) avisplit will then split to base-%04d.avi -s size Use this option to specify the maximum size (in units of MB) of the chunks avisplit should create. 0 means dechunk, create as many files as possible. -H num Create only the first num chunks then exit. -t s1-s2[,s3-s4,..] Split the input file based on time/framecode (hh:mm:ss.ms) -c Together with -t. Merge all segments into one AVI-File again instead generating seperate files. -m Together with -t. Force split at upper bondary instead of lower border. -b num Specify if avisplit should write an VBR mp3 header into the AVI file. Default is 1 because it does not hurt. num is either 1 or 0. -f commentfile Read AVI tombstone data for header comments from commentfile. See /docs/avi_comments.txt for a sample. -v Print only version information and exit. EXAMPLES
The command avisplit -s 700 -i my_file.avi will split the file my_file.avi into chunks which's maximum size will not exceed 700 MB, i.e. they will fit onto a CD, each. The created chunks will be named my_file.avi-0000, my_file.avi-0001, etc. avisplit -i my_file.avi -c -o out.avi -t 00:10:00-00:11:00,00:13:00-00:14:00 will grab Minutes 10 to 11 and 13 to 14 from my_file.avi and merge it into out.avi BAD SYNCH
When you split a file with avisplit and the A/V sync for the first file is OK but the sync on all successive files is bad then have a look at the output of tcprobe(1) (shortend). | V: 25.000 fps, codec=dvsd, frames=250, width=720, height=576 | A: 48000 Hz, format=0x01, bits=16, channels=2, bitrate=1536 kbps, | 10 chunks, 1920000 bytes You'll see the AVI file has only 10 Audio chunks but 250 video chunks. That means one audio chunk spans several video frames. avisplit can not cut a chunk in half, it only handles complete chunks. If you do, say, avisplit -s 20, it is possible that the first file will have 6 audio chunks and the second one only 4 meaning there is too much audio in the first AVI file. The solution is to remux the AVI file with transcode -i in.avi -P1 -N 0x1 -y raw -o out.avi (of course -N 0x1 is not correct for all AVI files). Now look at tcprobe again | V: 25.000 fps, codec=dvsd, frames=250, width=720, height=576 | A: 48000 Hz, format=0x01, bits=16, channels=2, bitrate=1536 kbps, | 250 chunks, 1920000 bytes The data in this file is exactly the same (its bit-identical) as it was in in.avi; the AVI file was just written in a different way, we do now have 250 audio chunks which makes splitting much easier and more accurate for avisplit. AUTHORS
avisplit was written by Thomas Oestreich <ostreich@theorie.physik.uni-goettingen.de> with contributions from many others. See AUTHORS for details. SEE ALSO
aviindex(1), avifix(1), avimerge(1), tccat(1), tcdecode(1), tcdemux(1), tcextract(1), tcprobe(1), tcscan(1), transcode(1) avisplit(1) 25th June 2003 avisplit(1)
All times are GMT -4. The time now is 11:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy