Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate line detail based on column one data Post 302384658 by dennis.jacob on Wednesday 6th of January 2010 04:22:29 AM
Old 01-06-2010
Try:

Code:
awk '!NF || ++A[$1]==1'  < file

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

2. Shell Programming and Scripting

remove new line characters from a partcular column data

Dear friends, I have a pipe delimited file having 5 columns. However the column no-3 is having extra new line characters as the data owing to owing , I am having issues. Ideally my file should have only newline termination at the end of each record and not within column data of any of... (1 Reply)
Discussion started by: sureshg_sampat
1 Replies

3. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

4. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric... (2 Replies)
Discussion started by: davegen
2 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

6. Shell Programming and Scripting

Help with figuring division and addition based on column data and line numbers

I have a data file in the format of 1234 xxx 1234 xxx 1234 xxx 1234 xxxI want to be able to calculate the following - COLUMN1+((LINENUMBER-1)/365) The output needs to preserve the 2nd column - 1234 xxx 1234.00274 xxx 1234.00548 xxx What is the best way to do this? I am somewhat... (9 Replies)
Discussion started by: ncwxpanther
9 Replies

7. Shell Programming and Scripting

How to remove a line based on contents of the first column?

Good day all. Using basic UNIX/Linux tools, how would you delete a line based on a character found in column 1? For example, if the CITY name contains an 'a' or 'A', delete the line: New York City; New York Los Angeles; California Chicago; Illinois Houston; Texas Philadelphia;... (3 Replies)
Discussion started by: BRH
3 Replies

8. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies

9. Shell Programming and Scripting

Remove sections based on duplicate first line

Hi, I have a file with many sections in it. Each section is separated by a blank line. The first line of each section would determine if the section is duplicate or not. if the section is duplicate then remove the entire section from the file. below is the example of input and output.... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

10. Shell Programming and Scripting

To append new data at the end of each line based on substring of last column

Hi guys, I need to append new data at the end of each line of the files. This new data is based on substring (3rd fields) of last column. Input file xxx.csv: U1234|1-5X|orange|1-5X|Act|1-5X|0.1 /sac/orange 12345 0 U5678|1-7X|grape|1-7X|Act|1-7X|0.1 /sac/grape 5678 0... (5 Replies)
Discussion started by: null7
5 Replies
avimerge(1)						      General Commands Manual						       avimerge(1)

NAME
avimerge - merge several AVI-files into one SYNOPSIS
avimerge -o name -i file1 [ file2 [ ... ] ] [ -p file ] [ -a num ] [ -A num ] [ -b num ] [ -c ] [ -f commentfile ] [ -x indexfile ] COPYRIGHT
avimerge is Copyright (C) by Thomas Oestreich. DESCRIPTION
avimerge is a versatile tool. It can contatenate several AVI files into one. It can also be used to fix an index of a broken file and can also replace audio tracks or muxes new ones. It can read raw AC3 and MP3 files for multplexing. OPTIONS
-o name Specify the name of the output file. -i file Specify the name(s) of the input file(s) to merge into the output file. -p file Specify the name of the audio file to multiplex into the output file. The type of file can be either another AVI file or an MP3 or AC3 file. -b num Specify if avimerge should write an VBR mp3 header into the AVI file. Default is dependent on the input file (and usually correct). num is either 1 or 0. -c Drop video frames in case audio is missing [off] Only when merging multiple AVI files. Some AVI files run a little bit (usually for one or two video frames) short on audio. This means avimerge cannot keep up sync when concatinating them. The files play fine when played individually but not when merged because audio from the new file gets played back with video from the old file. avimerge will print a message like No audiodata left for track 0->0 (59950.25=59950.25) continuing .. When you turn on the -c option, the video which is too much will be dropped. -f commentfile Read AVI tombstone data for header comments from commentfile. See /docs/avi_comments.txt for a sample. -x indexfile Read the AVI index from indexfile. See aviindex(1) for information on how to create such a file. -a num Specify the number of the audio track you want to use from the input file. -A num Specify the number of the audio track you want to use in the output file. If you specify an existing track number, the track will be replaced. If omitted, the next free slot will be used. EXAMPLES
The command avimerge -o big.avi -i my_file1.avi my_file2.avi my_file3.avi merges the three input files my_file[123].avi into one big AVI-file big.avi. avimerge -o out.avi -i in.avi -p audio2.avi -a 1 merges track number 1 form in.avi to the next free track number in out.avi. You can create audio-only AVI-files using transcode -i song.mp3 -x null,mp3 -g 0x0 -y raw -a 1 -o audio2.avi -u 50 The command avimerge -o out.avi -i in.avi -p sound.mp3 merges the file sound.mp3 as an additional audio track into out.avi. AUTHORS
avimerge was written by Thomas Oestreich <ostreich@theorie.physik.uni-goettingen.de> with contributions from many others. See AUTHORS for details. SEE ALSO
aviindex(1), avifix(1), avisplit(1), tccat(1), tcdecode(1), tcdemux(1), tcextract(1), tcprobe(1), tcscan(1), transcode(1) avimerge(1) 26th January 2004 avimerge(1)
All times are GMT -4. The time now is 11:05 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy