Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Delete only if duplicates found in each record
Top Forums Shell Programming and Scripting Delete only if duplicates found in each record Post 302912210 by redse171 on Wednesday 6th of August 2014 01:48:15 PM
Old 08-06-2014
Delete only if duplicates found in each record
Hi,

i have another problem. I have been trying to solve it by myself but failed.

inputfile
Code:
;;
ID	T08578
NAME	T08578
SBASE	30696
EBASE	32083
TYPE	P
func	just test
func	chronology
func	cholesterol
func	null
INT	30765-37333
INT	37154-37318
Link	5546
Link	8142
Link	5485
@@
ID	T09378
NAME	T09378
SBASE	35275
EBASE	35282
TYPE	W
func	Dito and barney
func	null
CODE	2.6
INT	21783-35274
Link	3899
@@
ID	T09386
NAME	T09386
SBASE	3505918
EBASE	3506467
TYPE	R
func	null
INT	5974-6088
@@
ID	T08594
NAME	T08594
SBASE	95156
EBASE	95174
TYPE	W
func	null
INT	9585-9562
@@


I need to remove any duplicate for "func" with "null" (red color) if there is another func in each record separated by "@@". If there is no duplicate (like the blue ones), then, it will remains as it is. The output should be:-

Code:
;;
ID	T08578
NAME	T08578
SBASE	30696
EBASE	32083
TYPE	P
func	just test
func	chronology
func	cholesterol
INT	30765-37333
INT	37154-37318
Link	5546
Link	8142
Link	5485
@@
ID	T09378
NAME	T09378
SBASE	35275
EBASE	35282
TYPE	W
func	Dito and barney
CODE	2.6
INT	21783-35274
Link	3899
@@
ID	T09386
NAME	T09386
SBASE	3505918
EBASE	3506467
TYPE	R
func	null
INT	5974-6088
@@
ID	T08594
NAME	T08594
SBASE	95156
EBASE	95174
TYPE	W
func	null
INT	9585-9562
@@

I tried couple of ways to do it but the results is either it deleted all with "func" or it will remains as it is. The last code that i tried is the one that i got from one of the thread here but it just deleted "func" with "null" for the first record only. The rest it remains the same. The code is below:-

Code:
awk '!NF{$0=x}1' inputfile | awk 'gsub (/func/, "&") > 1 {sub (/\n func\tnull\n/,"\n")}1' RS=\n

I tried to use sed too as i wanted to update the input file instead of creating another output file. but i did not get what i want. Any help is appreciated.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How can I parse a record found in /etc/passwd into variables?

I am working with the Oracle 10.2.0.3 job scheduler on Solaris 10, and unfortunately, the scheduler executes scripts in such a way that several default shell environment variables are not defined. For example, $HOME, $USER, and $LOGNAME are missing. How can I parse the appropriate record in... (7 Replies)
Discussion started by: shew01
7 Replies

2. Shell Programming and Scripting

An interactive way to delete duplicates

1)I am trying to write a script that works interactively lists duplicated records on certain field/column and asks user to delete one or more. And finally it deletes all the records the used has asked for. I have an idea to store those line numbers in an array, not sure how to do this in... (3 Replies)
Discussion started by: chvs2000
3 Replies

3. Shell Programming and Scripting

how can I delete duplicates in the log?

I have a log file and I am trying to run a script against it to search for key issues such as invalid users, errors etc. In one part, I grep for session closed and get a lot of the same thing,, ie. root username etc. I want to remove the multiple root and just have it do a count, like wc -l ... (5 Replies)
Discussion started by: taekwondo
5 Replies

4. Shell Programming and Scripting

Delete duplicates via script?

Hello, i have the following problem: there are two folders with a lot of files. Example: FolderA contains AAA, BBB, CCC FolderB contains DDD, EEE, AAA How can i via script identify AAA as duplicate in Folder B and delete it there? So that only DDD and EEE remain, in Folder B? Thank you... (16 Replies)
Discussion started by: Y-T
16 Replies

5. Shell Programming and Scripting

prompt to delete each record when pattern is found

Hello!. I am working on a very simple program and I have been trying different things. This is so far what I have done and there is one small detail that still does not work. It finds all the records in a phonebook per say: ./rem Susan More than one match; Please select the one to remove: ... (3 Replies)
Discussion started by: bartsimpsong
3 Replies

6. Shell Programming and Scripting

Fastest way to delete duplicates from a large filelist.....

OK I have two filelists...... The first is formatted like this.... /path/to/the/actual/file/location/filename.jpg and has up to a million records The second list shows filename.jpg where there is more then on instance. and has maybe up to 65,000 records I want to copy files... (4 Replies)
Discussion started by: Bashingaway
4 Replies

7. Shell Programming and Scripting

delete from line and remove duplicates

My Input.....file1 ABCDE4435 Connected to 107.71.136.122 (SubNetwork=ONRM_RootMo_R SubNetwork=XYVLTN29CRBR99 MeContext=ABCDE4435 ManagedElement=1) ABCDE4478 Connected to 166.208.30.57 (SubNetwork=ONRM_RootMo_R SubNetwork=KLFMTN29CR0R04 MeContext=ABCDE4478 ManagedElement=1) ABCDE4478... (5 Replies)
Discussion started by: pareshkp
5 Replies

8. Shell Programming and Scripting

Remove Duplicates on multiple Key Columns and get the Latest Record from Date/Time Column

Hi Experts , we have a CDC file where we need to get the latest record of the Key columns Key Columns will be CDC_FLAG and SRC_PMTN_I and fetch the latest record from the CDC_PRCS_TS Can we do it with a single awk command. Please help.... (3 Replies)
Discussion started by: vijaykodukula
3 Replies

9. Shell Programming and Scripting

Delete duplicates in CA bundle

I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates. The alignment of the certificate within the bundle seems to be different. Example: Cert 1... (7 Replies)
Discussion started by: diva_thilak
7 Replies

10. Shell Programming and Scripting

To Delete the duplicates using Part of File Name

I am using the below script to delete duplicate files but it is not working for directories with more than 10k files "Argument is too long" is getting for ls -t. Tried to replace ls -t with find . -type f \( -iname "*.xml" \) -printf '%T@ %p\n' | sort -rg | sed -r 's/* //' | awk... (8 Replies)
Discussion started by: gold2k8
8 Replies

LEARN ABOUT V7

set_color

set_color(1)							       fish							      set_color(1)

NAME

       set_color - set_color - set the terminal color

set_color - set the terminal color
   Synopsis
       set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR]

   Description
       Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple,
       cyan, white and normal.

       o -b, --background Set the background color

       o -c, --print-colors Prints a list of all valid color names

       o -h, --help Display help message and exit

       o -o, --bold Set bold or extra bright mode

       o -u, --underline Set underlined mode

       o -v, --version Display version and exit

       Calling set_color normal will set the terminal color to whatever is the default color of the terminal.

       Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey
       font color, while set_color --bold white will result in a white font color.

       Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator.

       set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and
       incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of
       ncurses and recompile fish against it in order to fix this issue.

Version 1.23.1							  Sun Jan 8 2012						      set_color(1)
All times are GMT -4. The time now is 08:38 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy