12-12-2013
Delete duplicate strings in a line
Hi,
i need help to remove duplicates in my file. The problem is i need to delete one duplicate for each line only. the input file as follows and it is not tab delimited:-
Quote:
&aff2g0440 aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aff2g0740 aspl5221 nos:scad1 blablablabla
&aff4g0160 aff4g01600 aspl2251 nos:scad1 blablablabla
&aff9g0020 aff9g0020 aspl3391 nos:scad2 blablablabla
The output need to remove 2nd word (in red) that duplicate with 1st word (in blue). Other duplicates should remained unchanged. my output should be like this:-
Quote:
&aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aspl5221 nos:scad1 blablablabla
&aff4g0160 aspl2251 nos:scad1 blablablabla
&aff9g0020 aspl3391 nos:scad2 blablablabla
i don't know how to do this. i did try but it deleted all the other duplicates as well in that lines. tried to google too and it seems that most of the issue is the duplicate lines. Please kindly help. Thanks
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hey all, a relative bash/script newbie trying solve a problem.
I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like
2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies
2. Shell Programming and Scripting
Hello friends!
Each line of my input file has this format:
word<TAB>tag1<blankspace>lemma<TAB>tag2<blankspace>lemma ... <TAB>tag3<blankspace>lemma
Of this file I need to eliminate all the repeated tags (of the same word) in a line, as in the example here below, but conserving both (all) the... (2 Replies)
Discussion started by: mjomba
2 Replies
3. UNIX for Dummies Questions & Answers
Hello guys,
should be a very easy questn for you:
I need to delete strings in file1 based on the list of strings in file2.
like file2:
word1_word2_
word3_word5_
word3_word4_
word6_word7_
file1:
word1_word2_otherwords..,word3_word5_others... (7 Replies)
Discussion started by: roussine
7 Replies
4. Shell Programming and Scripting
I am completely new to shell scripting but have been assigned the task of creating several batch files to manipulate data. My final task requires me to find lines that have duplicates present then delete not only the duplicate but the original as well. The script will be used in a windows... (9 Replies)
Discussion started by: chino_1
9 Replies
5. UNIX for Dummies Questions & Answers
Hi ALL
I need a help
I need to retain only the first line of 035 if I have two line before =040 , if only one then need to take that
Eg:
Input
=035 (ABC)12324141241
=035 (XYZPQR)704124
=040 AB$QS$WEWR
=035 (ABC)08080880809
=035 (XYZPQR)9809314
=040 ... (4 Replies)
Discussion started by: umapearl
4 Replies
6. Shell Programming and Scripting
The question is not as simple as the title... I have a file, it looks like this
<string name="string1">RZ-LED</string>
<string name="string2">2.0</string>
<string name="string2">Version 2.0</string>
<string name="string3">BP</string>
I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies
7. Shell Programming and Scripting
I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size).
Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies
8. Shell Programming and Scripting
Hi guys, I wonder if it's possible to search for a line containing 2 strings and delete that line and perhaps replace the source file with already deleted line(s).
What I mean is something like this:
sourcefile.txt
line1: something 122344 somethin2 24334 45554676
line2: another something... (6 Replies)
Discussion started by: netrom
6 Replies
9. UNIX for Dummies Questions & Answers
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
10. Shell Programming and Scripting
Dear folks
I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies
LEARN ABOUT OSX
db_upgrade
db_upgrade(1) BSD General Commands Manual db_upgrade(1)
NAME
db_upgrade
SYNOPSIS
db_upgrade [-NsV] [-h home] [-P password] file ...
DESCRIPTION
The db_upgrade utility upgrades the Berkeley DB version of one or more files and the databases they contain to the current release version.
The options are as follows:
-h
Specify a home directory for the database environment; by default, the current working directory is used.
-N
Do not acquire shared region mutexes while running. Other problems, such as potentially fatal errors in Berkeley DB, will be ignored as
well. This option is intended only for debugging errors, and should not be used under any other circumstances.
-P
Specify an environment password. Although Berkeley DB utilities overwrite password strings as soon as possible, be aware there may be a
window of vulnerability on systems where unprivileged users can see command-line arguments or where utilities are not able to overwrite
the memory containing the command-line arguments.
-s
This flag is only meaningful when upgrading databases from releases before the Berkeley DB 3.1 release.
As part of the upgrade from the Berkeley DB 3.0 release to the 3.1 release, the on-disk format of duplicate data items changed. To cor-
rectly upgrade the format requires that applications specify whether duplicate data items in the database are sorted or not. Specifying
the -s flag means that the duplicates are sorted; otherwise, they are assumed to be unsorted. Incorrectly specifying the value of this
flag may lead to database corruption.
Because the db_upgrade utility upgrades a physical file (including all the databases it contains), it is not possible to use db_upgrade to
upgrade files where some of the databases it includes have sorted duplicate data items, and some of the databases it includes have
unsorted duplicate data items. If the file does not have more than a single database, if the databases do not support duplicate data
items, or if all the databases that support duplicate data items support the same style of duplicates (either sorted or unsorted),
db_upgrade will work correctly as long as the -s flag is correctly specified. Otherwise, the file cannot be upgraded using db_upgrade, and
must be upgraded manually using the db_dump and db_load utilities.
-V
Write the library version number to the standard output, and exit.
It is important to realize that Berkeley DB database upgrades are done in place, and so are potentially destructive. This means that if the
system crashes during the upgrade procedure, or if the upgrade procedure runs out of disk space, the databases may be left in an inconsistent
and unrecoverable state. See Upgrading databases for more information.
The db_upgrade utility may be used with a Berkeley DB environment (as described for the -h option, the environment variable DB_HOME, or
because the utility was run in a directory containing a Berkeley DB environment). In order to avoid environment corruption when using a
Berkeley DB environment, db_upgrade should always be given the chance to detach from the environment and exit gracefully. To cause db_upgrade
to release all environment resources and exit cleanly, send it an interrupt signal (SIGINT).
The db_upgrade utility exits 0 on success, and >0 if an error occurs.
ENVIRONMENT
DB_HOME If the -h option is not specified and the environment variable DB_HOME is set, it is used as the path of the database home, as
described in DB_ENV->open.
SEE ALSO
db_archive(1), db_checkpoint(1), db_deadlock(1), db_dump(1), db_load(1), db_printlog(1), db_recover(1), db_stat(1), db_verify(1)
Darwin December 3, 2003 Darwin