Bash - remove duplicates without sort Post: 302849307

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates [sort , uniq]

Hey Guys, I have file which looks like this, Contig201#numbPA Contig1452#nmdynD6PA dm022p15.r#CG6461PA dm005e16.f#SpatPA IGU001_0015_A06.f#CG17593PA I need to remove duplicates based on the chracter matching upto '#'. for example if we consider this.. Contig201#numbPA...

2. Shell Programming and Scripting

Sort, Uniq, Duplicates

Input File is : ------------- 25060008,0040,03, 25136437,0030,03, 25069457,0040,02, 80303438,0014,03,1st 80321837,0009,03,1st 80321977,0009,03,1st 80341345,0007,03,1st 84176527,0047,03,1st 84176527,0047,03, 20000735,0018,03,1st 25060008,0040,03, I am using the following in the script...

3. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample...

4. Shell Programming and Scripting

bash ps; remove the header, sort and reinsert

Hi, I'm ssh'ing into a server using ruby and sending a one-liner to retrieve the output of the 'ps aux' command. So far, this is what I have: ps aux | sort -r -n -k3 | sed -e '1s/^/this is first\n/' | head -n10 With this I can insert a line at position 1, but I would rather extract the...

5. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks

6. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ...

7. Shell Programming and Scripting

Sort data by date first and then remove duplicates

Hi , I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points : (1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column) (2)after sorting the file based on latest date...

8. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert...

9. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker...

10. Shell Programming and Scripting

Concatenate and sort to remove duplicates

Following is the input. 1st and 3rd block are same(block starts here with '*' and ends before blank line) , 2nd and 4th blocks are also the same: cat <file> * Wed Feb 24 2016 Tariq Saeed <tariq.x.saeed@mail.com> 2.0.7-1.0.7 - add vmcore dump support for ocfs2 * Mon Jun 8 2015 Brian Maly...

LEARN ABOUT DEBIAN

prezip

PREZIP(1)						 Aspell Abbreviated User's Manual						 PREZIP(1)

NAME

       prezip, preunzip, precat -  prefix delta compressor for Aspell

SYNOPSIS

       precat [-dzhLV] [-cfksSq] [file ...]

DESCRIPTION

       prezip  is a a prefix delta compressor.	It is not a general purpose compressor and should only be used on sorted word lists or other simi-
       lar text files.	It will likely increase the size of binary data.

       If invoked as prezip, the default action is to compress.  If invoked as preunzip, the default action is to decompress.  If invoked as  pre-
       cat, the default action is to decompress to stdout.

       If no file names are given then prezip will compress or decompress from the standard input to the standard output.  Short flags can be com-
       bined so that "-c -s" is the same as "-cs".

OPTIONS

       -h --help
	      display help

       -d --decompress
	      force decompression

       -z --compress
	      dorce compression

       -L --license
	      display software license

       -V --version
	      display version

       -c --stdout
	      decompress to standard output

       -f --force
	      force

       -k --keep
	      keep input files

       -s --sort
	      sort and remove duplicates before compressing

       -S --nocwl
	      do not rename .wl suffix to .cwl (use .wl.pz instead)

       -q --quiet
	      suppress all warnings

AUTHOR

       This manual page was written by Brian Nelson <pyro@debian.org> based on the output of help2man.	Aspell is Copyright (C) 2002 Kevin  Atkin-
       son.

GNU
								    March 2010								 PREZIP(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates [sort , uniq]

Discussion started by: sharatz83

2. Shell Programming and Scripting

Sort, Uniq, Duplicates

Discussion started by: Amruta Pitkar

3. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Discussion started by: orahi001

4. Shell Programming and Scripting

bash ps; remove the header, sort and reinsert

Discussion started by: gekeha