Sponsored Content
Top Forums Shell Programming and Scripting Filter or remove duplicate block of text without distinguishing marks or fields Post 302563518 by samask on Tuesday 11th of October 2011 12:21:40 PM
Old 10-11-2011
I simplified a test case, with different number of newlines:

Code:
[block 4] The grooves separate a series of rounded bars or arches, the
branchial or visceral arches, in which thickening of the mesoderm takes
place (Figs. 40 and 41).


$newpage

[block 4] The grooves separate a series of rounded bars or arches, the
branchial or visceral arches, in which thickening of the mesoderm takes
place (Figs. 40 and 41).
$newpage

[block 5] In all, six arches make their appearance, but of these only
the first four are visible externally.



$newpage

[block 5] In all, six arches make their appearance, but of these only
the first four are visible externally.

With that test case, I get:

Code:
[block 4] The grooves separate a series of rounded bars or arches, the
branchial or visceral arches, in which thickening of the mesoderm takes
place (Figs. 40 and 41).

[block 4] The grooves separate a series of rounded bars or arches, the
branchial or visceral arches, in which thickening of the mesoderm takes
place (Figs. 40 and 41).
[block 5] In all, six arches make their appearance, but of these only
the first four are visible externally.

[block 5] In all, six arches make their appearance, but of these only
the first four are visible externally.

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate text

Hello, I have a log file which is generated by a script which looks like this: userid: 7 starttime: Sat May 24 23:24:13 CEST 2008 endtime: Sat May 24 23:26:57 CEST 2008 total time spent: 2.73072 minutes / 163.843 seconds date: Sat Jun 7 16:09:03 CEST 2008 userid: 8 starttime: Sun May... (7 Replies)
Discussion started by: dejavu88
7 Replies

2. Shell Programming and Scripting

Filter duplicate block of text using SED

Hi, I would like to print a block of text between 2 regular expression using Sed, This can be achieved by using the command as shown below, however my problem is the same block of text is repeated twice. I would like to eliminate the duplicate block of text. For Example If my file... (5 Replies)
Discussion started by: dkumar91
5 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Filter/remove duplicate .dat file with certain criteria

I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records. contents of file looks like 30002157,40342424,OTC,mart_rec,100, ,0 30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Discussion started by: mukeshguliao
6 Replies

5. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

6. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

7. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

8. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

9. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies

10. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies
wrjpgcom(1)							   User Commands						       wrjpgcom(1)

NAME
wrjpgcom - insert text comments into a JPEG file SYNOPSIS
wrjpgcom [-replace] [-comment text] [-cfile name] [filename] DESCRIPTION
wrjpgcom reads the named JPEG or JFIF file, or the standard input if no file is named, and generates a new JPEG or JFIF file on the stan- dard output. A comment block is added to the file. The JPEG standard allows "comment" (COM) blocks to occur within a JPEG file. Although the standard does not actually define the intended function of COM blocks, they are widely used to hold user-supplied text strings. This enables you to add annotations, titles, index terms, and so on to your JPEG files, and later retrieve the COM blocks as text. COM blocks do not interfere with the image stored in the JPEG file. The maximum size of a COM block is 64K, but you can have many COM blocks in one JPEG file. wrjpgcom adds a COM block, containing text that you provide, to a JPEG file. Ordinarily, the COM block is added after any existing COM blocks, but you can delete the old COM blocks if you wish. OPTIONS
The following options are supported: -cfile name Read the text for a new COM block from the named file. -comment text Supply the text for a new COM block on the command line. -replace Delete any existing COM blocks from the file. OPERANDS
The following operands are supported: filename The name of the JPEG file to which you want to add text comments. EXTENDED DESCRIPTION
To add only one line of comment text, use the -comment option to provide the text on the command line. Specify the comment text within quotes, so that the text is treated as a single argument. Longer comments can be read from a text file. If you specify neither the -comment nor the -cfile option, wrjpgcom reads the comment text from standard input. In such cases, you must supply an input image filename. You can enter multiple lines, up to 64KB. Type an end-of-file indicator, usually Ctrl-D, to terminate the comment text entry. wrjpgcom does not add a COM block if the provided comment string is empty. Therefore, you can use -replace -comment "" to delete all COM blocks from a file. EXAMPLES
Example 1: Adding a Short Comment to in.jpg to Produce out.jpg example% wrjpgcom -c "View of my back yard" in.jpg > out.jpg Example 2: Attaching a Long Comment Previously Stored in comment.txt example% wrjpgcom in.jpg < comment.txt > out.jpg or example% wrjpgcom -cfile comment.txt < in.jpg > out.jpg In this example, 1000 is a number that is larger than the number of rows in the source file. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWjpg | +-----------------------------+-----------------------------+ |Interface stability |Volatile | +-----------------------------+-----------------------------+ SEE ALSO
cjpeg(1), djpeg(1), jpegtran(1), rdjpgcom(1) NOTES
This man page was originally written by the Independent JPEG Group. Updated by Breda McColgan, Sun Microsystems Inc., 2004. SunOS 5.11 26 Mar 2004 wrjpgcom(1)
All times are GMT -4. The time now is 01:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy