Filtering duplicate lines Post: 14994

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Issues with filtering duplicate records using gawk script

Hi All, I have huge trade file with milions of trades.I need to remove duplicate records (e.g I have following records) 30/10/2009,trdeId1,..,.. 26/10/2009.tradeId1,..,..,, 30/10/2009,tradeId2,.. In the above case i need to filter duplicate recods and I should get following output....

2. UNIX for Dummies Questions & Answers

Filtering similar lines in a big list

I received this question for homework: We have to write our program into a .sh file, with "#!/bin/bash" as the first line. And we have the list of access logs in a file, looking like this (it's nearly 10,000 lines long): 65.214.44.112 - - "GET /~user0/cgg/msg08400.html HTTP/1.0" 304 -...

3. Shell Programming and Scripting

filtering out duplicate substrings, regex string from a string

My input contains a single word lines. From each line data.txt prjtestBlaBlatestBlaBla prjthisBlaBlathisBlaBla prjthatBlaBladpthatBlaBla prjgoodBlaBladpgoodBlaBla prjgood1BlaBla123dpgood1BlaBla123 Desired output --> data_out.txt prjtestBlaBla prjthisBlaBla...

4. Homework & Coursework Questions

Filtering Unique Lines

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: The uniq command excludes consecutive duplicate lines. It has a -c option to display a count of the number...

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to...

6. Shell Programming and Scripting

Perl: filtering lines based on duplicate values in a column

Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times. 13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG 13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG 13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT 13 18 1 + chromosome 1,...

7. UNIX for Dummies Questions & Answers

Filtering data -extracting specific lines

I have a table to data which one of the columns include string of text from within that, I am searching to include few lines but not others for example I want to to include some combination of word address such as (address.| address? |the address | your address) but not (ip address | email...

8. Shell Programming and Scripting

Filtering out lines in a .csv file

Hi Guys, Would need your expert help with the following situation.. I have a comma seperated .csv file, with a header row and data as follows H1,H2,H3,H4,H5..... (header row) 0,0,0,0,0,1,2.... (data rows follow) 0,0,0,0,0,0,1 ......... ......... i need a code...

9. Shell Programming and Scripting

Awk/sed : help on:Filtering multiple lines to one:

Experts Good day, I want to filter multiple lines of same error of same day , to only 1 error of each day, the first line from the log. Here is the file: May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out) May 26 11:29:19 cmihpx02 vmunix: NFS...

10. Shell Programming and Scripting

Filtering log file with lines older than 10 days.

Hi, I am trying to compare epoch time in a huge log file (2 million lines) with todays date. I have to create two files one which has lines older than 10 days and another file with less than 10 days. I am using while do but it takes forever to complete the script. It would be helpful if you can...

LEARN ABOUT DEBIAN

bp_chaos_plot

BP_CHAOS_PLOT(1p)					User Contributed Perl Documentation					 BP_CHAOS_PLOT(1p)

NAME

       chaos_plot - a chaos plot from DNA and RNA sequences

SYNOPSIS

	 chaos_plot.pl -i/--input=INPUTFILE -f/--format=SEQFORMAT
	       -o/--output=OUTPUTFILE -g/--graphics=GRAPHIC FORMAT
	       -w/--width=WIGHT -h/--height=HEIGHT

DESCRIPTION

       This scripts generates image files using GD image library to visualize nucleotide sequences using chaos plot.

OPTIONS

       Valid graphics formats are currently gd, gd2, png, wbmp, jpeg and gif.

       The default size of the image file is 600x400.

       The sequence input can be provided using any of the three methods:

       unnamed argument
	    chaos_plot filename

       named argument
	    chaos_plot -i filename

       standard input
	    chaos_plot < filename

FEEDBACK

   Mailing Lists
       User feedback is an integral part of the evolution of this and other Bioperl modules. Send your comments and suggestions preferably to the
       Bioperl mailing list.  Your participation is much appreciated.

	 bioperl-l@bioperl.org			- General discussion
	 http://bioperl.org/wiki/Mailing_lists	- About the mailing lists

   Reporting Bugs
       Report bugs to the Bioperl bug tracking system to help us keep track of the bugs and their resolution. Bug reports can be submitted via the
       web:

	 https://redmine.open-bio.org/projects/bioperl/

AUTHOR - Jason Stajich
       Email jason@bioperl.org

HISTORY

       This code is based on EMBOSS C code for chaos.c by Ian Longden.	Included are documentation from EMBOSS code:

       Chaos produces a chaos plot.  The original application is part of the ACEDB genome database package, written by ** Richard Durbin (MRC LMB,
       UK) rd@mrc-lmba.cam.ac.uk, and Jean Thierry-Mieg (CRBM du CNRS, France) mieg@crbm1.cnusc.fr

perl v5.14.2							    2012-03-02							 BP_CHAOS_PLOT(1p)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Issues with filtering duplicate records using gawk script

Discussion started by: nmumbarkar

2. UNIX for Dummies Questions & Answers

Filtering similar lines in a big list

Discussion started by: Andrew9191

3. Shell Programming and Scripting

filtering out duplicate substrings, regex string from a string

Discussion started by: kchinnam

4. Homework & Coursework Questions

Filtering Unique Lines

Discussion started by: billydeanmak