How about using sed instead of paste to pre-process the file? We would first create single lines of the blocks, then transform it back into blocks again after processing it through uniq. Here is a naive try which might need refinement:
Transform the blocks to lines:
(edited - see RudiC's post, the same idea. Basically you replace all newline characters inside a block with a temporary replacement character to get one line, RudiC used "\r", but you can use any other string as well.)
or, even simpler, using fmt ("1000" is a number higher than the number of characters a resulting line could grow, replace it with a higher number if it does not suffice). Notice, though, that transforming this back into blocks is a bit more effort because there is no replacement character for the newlines:
Transform the lines back to blocks (enter the ^M literally as an <ENTER>):
I hope this helps.
Hey Guys,
I have file which looks like this,
Contig201#numbPA
Contig1452#nmdynD6PA
dm022p15.r#CG6461PA
dm005e16.f#SpatPA
IGU001_0015_A06.f#CG17593PA
I need to remove duplicates based on the chracter matching upto '#'.
for example if we consider this..
Contig201#numbPA... (4 Replies)
Input File is :
-------------
25060008,0040,03,
25136437,0030,03,
25069457,0040,02,
80303438,0014,03,1st
80321837,0009,03,1st
80321977,0009,03,1st
80341345,0007,03,1st
84176527,0047,03,1st
84176527,0047,03,
20000735,0018,03,1st
25060008,0040,03,
I am using the following in the script... (5 Replies)
Hello experts,
I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter
sort -u -k 2,2 File.csv > Output.csv
File.csv
File Name|Document Name|Document Title|Organization
Word Doc 1.doc|Word Document|Sample... (3 Replies)
Hi:
I am trying to create some script that sorts the files in a subdirectory by date and concatenates the thre most recently created files.
SAy,
file1 date1
file2 date2
file3 date3
file4 date4
file5 date5
file6 date6
i only want to concatenate the first three which are the most... (4 Replies)
I have the following requirement.
1. I have to concatenate the 10 fixed width files.
2. sort based on first 10 characters
3. after that i have remove first 10 chacters from the file.
can you please tell me how to do it.
Thanks in Advance
Samba (1 Reply)
Hi,
I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another.
Thanks (6 Replies)
Hi ,
I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points :
(1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column)
(2)after sorting the file based on latest date... (3 Replies)
I need to use bash to remove duplicates without using sort first.
I can not use:
cat file | sort | uniq
But when I use only
cat file | uniq
some duplicates are not removed. (4 Replies)
Here is my task :
I need to sort two input files and remove duplicates in the output files :
Sort by 13 characters from 97 Ascending
Sort by 1 characters from 96 Ascending
If duplicates are found retain the first value in the file
the input files are variable length, convert... (4 Replies)
I have /tmp dir with filename as:
010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker
010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker
010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker
010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker
010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies
LEARN ABOUT OPENDARWIN
paste
PASTE(1) BSD General Commands Manual PASTE(1)NAME
paste -- merge corresponding or subsequent lines of files
SYNOPSIS
paste [-s] [-d list] file ...
DESCRIPTION
The paste utility concatenates the corresponding lines of the given input files, replacing all but the last file's newline characters with a
single tab character, and writes the resulting lines to standard output. If end-of-file is reached on an input file while other input files
still contain data, the file is treated as if it were an endless source of empty lines.
The options are as follows:
-d list Use one or more of the provided characters to replace the newline characters instead of the default tab. The characters in list
are used circularly, i.e., when list is exhausted the first character from list is reused. This continues until a line from the
last input file (in default operation) or the last line in each file (using the -s option) is displayed, at which time paste
begins selecting characters from the beginning of list again.
The following special characters can also be used in list:
newline character
tab character
\ backslash character
Empty string (not a null character).
Any other character preceded by a backslash is equivalent to the character itself.
-s Concatenate all of the lines of each separate input file in command line order. The newline character of every line except the
last line in each input file is replaced with the tab character, unless otherwise specified by the -d option.
If '-' is specified for one or more of the input files, the standard input is used; standard input is read one line at a time, circularly,
for each instance of '-'.
EXAMPLES
List the files in the current directory in three columns:
ls | paste - - -
Combine pairs of lines from a file into single lines:
paste -s -d '
' myfile
Number the lines in a file, similar to nl(1):
sed = myfile | paste -s -d '
' - -
Create a colon-separated list of directories named bin, suitable for use in the PATH environment variable:
find / -name bin -type d | paste -s -d : -
DIAGNOSTICS
The paste utility exits 0 on success, and >0 if an error occurs.
SEE ALSO cut(1), lam(1)STANDARDS
The paste utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible.
HISTORY
A paste command appeared in Version 32V AT&T UNIX.
BUGS
Multibyte character delimiters cannot be specified with the -d option.
BSD September 20, 2001 BSD