How to delete a duplicate line and original with sed.
I am completely new to shell scripting but have been assigned the task of creating several batch files to manipulate data. My final task requires me to find lines that have duplicates present then delete not only the duplicate but the original as well. The script will be used in a windows environment so I am using GNU sed. Below is a sample of the data:
In this example the first and last lines are duplicates and I would like to delete them both. I have been searching for several days and have not been able to figure out how to achieve this . Unfortunately I am short on time and would greatly appreciate any help possible. Thanks.
Last edited by Franklin52; 06-20-2011 at 03:32 AM..
Reason: Please use code tags
Hi,
I want to write a sed script which from
batiato:
batiato/giubbe:
pip_b.2.txt
pip_b.3.txt
pip_b.3mmm.txt
bennato:
bennato/peterpan:
123.txt
consoli:
pip_a.12.txt
daniele: (2 Replies)
Please let me know wat would be sed command to delete any partcular line from a file and also moving lines below it to up.
ie wen line #9 is deleted data in line #10 should move to #9 and so on. (2 Replies)
I'm new to using sed and grep commands, but have found them extremely useful. However I am having a hard time figuring this one out:
Delete every line containing the word CEN and the next line as well.
ie. test.txt
blue
324 CEN
green
red
blue
324 CEN
green
red
blue
to produce:... (2 Replies)
Hi ALL
I need a help
I need to retain only the first line of 035 if I have two line before =040 , if only one then need to take that
Eg:
Input
=035 (ABC)12324141241
=035 (XYZPQR)704124
=040 AB$QS$WEWR
=035 (ABC)08080880809
=035 (XYZPQR)9809314
=040 ... (4 Replies)
I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size).
Here is the issue. I have a data file with the following... (4 Replies)
here is what i want to achieve.. i have a file with below contents
cat fileName
blah blah blah
.
.DROP this
REJECT that
.
--sport 7800 -j REJECT --reject-with icmp-port-unreachable
--dport 7800 -j REJECT --reject-with icmp-port-unreachable
.
.
.
more blah blah blah
--dport 3306... (14 Replies)
There are some duplicate field on description column .I want to print duplicate row along with highest version of number and corresponding description column.
file1.txt
number Description
=== ============
34567 nl21a00is-centerdb001:ncdbareq:Error in loading init
34577 ... (7 Replies)
Hi,
I have an input file like this
line1
line2
line3 hello unix how are you
This is what I am expecting my output to be
line1
line2
#line3 hello unix how are you
line3 hello (3 Replies)
Hi,
i need help to remove duplicates in my file. The problem is i need to delete one duplicate for each line only. the input file as follows and it is not tab delimited:-
The output need to remove 2nd word (in red) that duplicate with 1st word (in blue). Other duplicates should remained... (12 Replies)
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
LEARN ABOUT NETBSD
msguniq
MSGUNIQ(1) GNU MSGUNIQ(1)NAME
msguniq - unify duplicate translations in message catalog
SYNOPSIS
msguniq [OPTION] [INPUTFILE]
DESCRIPTION
Unifies duplicate translations in a translation catalog. Finds duplicate translations of the same message ID. Such duplicates are invalid
input for other programs like msgfmt, msgmerge or msgcat. By default, duplicates are merged together. When using the --repeated option,
only duplicates are output, and all other messages are discarded. Comments and extracted comments will be cumulated, except that if
--use-first is specified, they will be taken from the first translation. File positions will be cumulated. When using the --unique
option, duplicates are discarded.
Mandatory arguments to long options are mandatory for short options too.
Input file location:
INPUTFILE
input PO file
-D, --directory=DIRECTORY
add DIRECTORY to list for input files search
If no input file is given or if it is -, standard input is read.
Output file location:
-o, --output-file=FILE
write output to specified file
The results are written to standard output if no output file is specified or if it is -.
Message selection:
-d, --repeated
print only duplicates
-u, --unique
print only unique messages, discard duplicates
Input file syntax:
-P, --properties-input
input file is in Java .properties syntax
--stringtable-input
input file is in NeXTstep/GNUstep .strings syntax
Output details:
-t, --to-code=NAME
encoding for output
--use-first
use first available translation for each message, don't merge several translations
-e, --no-escape
do not use C escapes in output (default)
-E, --escape
use C escapes in output, no extended chars
--force-po
write PO file even if empty
-i, --indent
write the .po file using indented style
--no-location
do not write '#: filename:line' lines
-n, --add-location
generate '#: filename:line' lines (default)
--strict
write out strict Uniforum conforming .po file
-p, --properties-output
write out a Java .properties file
--stringtable-output
write out a NeXTstep/GNUstep .strings file
-w, --width=NUMBER
set output page width
--no-wrap
do not break long message lines, longer than the output page width, into several lines
-s, --sort-output
generate sorted output
-F, --sort-by-file
sort output by file location
Informative output:
-h, --help
display this help and exit
-V, --version
output version information and exit
AUTHOR
Written by Bruno Haible.
REPORTING BUGS
Report bugs to <bug-gnu-gettext@gnu.org>.
COPYRIGHT
Copyright (C) 2001-2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
LAR PURPOSE.
SEE ALSO
The full documentation for msguniq is maintained as a Texinfo manual. If the info and msguniq programs are properly installed at your
site, the command
info msguniq
should give you access to the complete manual.
GNU gettext-tools 0.14.4 April 2005 MSGUNIQ(1)