Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Delete lines with duplicate strings based on date Post 302386452 by jim mcnamara on Tuesday 12th of January 2010 01:00:56 PM
Old 01-12-2010
Code:
awk '!arr[$1 $2]++'  inputfile > newfile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to delete duplicate records based on key

For example suppose I have a file which contains data as: $cat data 800,2 100,9 700,3 100,9 200,8 100,3 Now I want the output as 200,8 700,3 800,2 Key is first three characters, I don't want any reords which are having duplicate keys. Like sort +0.0 -0.3 data can we use... (9 Replies)
Discussion started by: sumitc
9 Replies

2. Shell Programming and Scripting

how to delete duplicate rows based on last column

hii i have a huge amt of data stored in a file.Here in this file i need to remove duplicates rows in such a way that the last column has different data & i must check for greatest among last colmn data & print the largest data along with other entries but just one of other duplicate entries is... (16 Replies)
Discussion started by: reva
16 Replies

3. UNIX for Dummies Questions & Answers

Delete strings in file1 based on the list of strings in file2

Hello guys, should be a very easy questn for you: I need to delete strings in file1 based on the list of strings in file2. like file2: word1_word2_ word3_word5_ word3_word4_ word6_word7_ file1: word1_word2_otherwords..,word3_word5_others... (7 Replies)
Discussion started by: roussine
7 Replies

4. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

5. Shell Programming and Scripting

Delete lines in file containing duplicate strings, keeping longer strings

The question is not as simple as the title... I have a file, it looks like this <string name="string1">RZ-LED</string> <string name="string2">2.0</string> <string name="string2">Version 2.0</string> <string name="string3">BP</string> I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies

6. Shell Programming and Scripting

Getting lines between two strings with duplicate set of data

if I have the following lines in a file app.log some lines here <AAAA> abc <id>123456789</id> ddd </AAAA>some lines here too <BBBB> abc <id>123456789</id> ddd </BBBB>some lines here too <AAAA> xyz <id>987654321</id> ssss </AAAA>some lines here again... How do I get the... (5 Replies)
Discussion started by: nariwithu
5 Replies

7. Shell Programming and Scripting

Delete duplicate strings in a line

Hi, i need help to remove duplicates in my file. The problem is i need to delete one duplicate for each line only. the input file as follows and it is not tab delimited:- The output need to remove 2nd word (in red) that duplicate with 1st word (in blue). Other duplicates should remained... (12 Replies)
Discussion started by: redse171
12 Replies

8. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

9. Shell Programming and Scripting

Delete duplicate row based on criteria

Hi, I have an input file as shown below: 20140102;13:30;FR-AUD-LIBOR-1W;2.495 20140103;13:30;FR-AUD-LIBOR-1W;2.475 20140106;13:30;FR-AUD-LIBOR-1W;2.495 20140107;13:30;FR-AUD-LIBOR-1W;2.475 20140108;13:30;FR-AUD-LIBOR-1W;2.475 20140109;13:30;FR-AUD-LIBOR-1W;2.475... (2 Replies)
Discussion started by: shash
2 Replies

10. UNIX for Beginners Questions & Answers

Delete rows based on array of strings

Hi All, I have a file with around 1000 rows and one of the fields is an account number. I have been provided with a few account numbers, when any of the account number appears in a row then the row has to be deleted. Can we place the account numbers in an array and use awk or sed for this task?... (3 Replies)
Discussion started by: mrcool4
3 Replies
TCA_CORRECT(1)							       HUGIN							    TCA_CORRECT(1)

NAME
tca_correct - Calculate transverse chromatic aberration SYNOPSIS
tca_correct [options] inputfile <inputfile> is the base name of 4 image files: <inputfile> Colour file to compute TCA parameters red_<inputfile> Red channel of <inputfile> green_<inputfile> Green channel of <inputfile> blue_<inputfile> Blue channel of <inputfile> The channel images must be colour images with 3 identical channels. If any of -R, -G, or -B is given, this file name is used instead of the derived name. Output: commandline arguments for fulla DESCRIPTION
Generates fulla command-line parameters for correction of transverse chromatic aberration (TCA) from sample photos. Ideal sample photos would have a lot of contrast over the entire image area, little saturated colour and no blown highlights. OPTIONS
-h Display help -l input file is PTO file instead of image -m method optimization method (0 normal, 1 newfit) -o optvars string of variables to optimize ("abcvde") -r Reset values (this will zero a,b,c,d,e params and set v to 10) makes sense only with -l option -s <scale> Scale for corner detection -n <number> number of points per grid cell (default: 10) -g <number> divide image in <number>x<number> grid cells (default: 10) -t num Remove all control points with an error higher than num pixels (default: 1.5) -v Verbose -w filename write PTO file -R <r> Use this file as red channel -G <g> Use this file as green channel -B <b> Use this file as blue channel AUTHORS
Written by Pablo d'Angelo and others. "Version: 2011.4.0" 2011-12-02 TCA_CORRECT(1)
All times are GMT -4. The time now is 12:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy