Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove duplicate line starting with a pattern

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate line starting with a pattern
# 1  
Old 12-07-2013
Remove duplicate line starting with a pattern
HI,

I have the below input file

Code:
  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
 
FILE_TRIGGER CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB

Code:
UNIX_JOB

or
Code:
FILE_TRIGGER

marks the beiginning of a job and
Code:
ENDJOB

marks the end. If you see 1st one has two entries for RUN..so what i want is to remove the RUN ANY line if there are two RUN statements

output should look like below

Code:
  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
UNIX_JOB CMDS065J
  RUN ANY
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB
 
  /* ----------------- cmdsDlyStartFWJ -----------------*/
FILE_TRIGGER CMDS065J
  CMDNAME sleep 5
  AGENT CMDSHP
  USER proddata
  RUN MON,TUE,WED,THU,FRI
  DELAYSUB 02:00
  /* "Triggers daily file watcher jobs" */
  ENVAR STDOUT=/home3/cmdsprod/stdout/cmdsDlyStartFWJ
ENDJOB

# 2  
Old 12-07-2013
Try this (may need some polishing):
Code:
awk 'gsub (/RUN/, "&") > 1 {sub (/\n *RUN ANY *\n/,"\n")}1' RS= file

# 3  
Old 12-07-2013
Code:
nawk 'gsub (/RUN/, "&") > 1 {sub (/\n *RUN ANY *\n/,"\n")}1' RS= cmdsDly_JIL.pull_job_esp
nawk: input record `UNIX_JOB CMDJ000J
  ...' too long
 source line number 1

Last edited by Scrutinizer; 12-08-2013 at 04:49 PM.. Reason: code tags
# 4  
Old 12-07-2013
The empty lines contain spaces. Try correcting the file with nawk '!NF{$0=x}1' to remove spaces from empty lines, before trying RudiC' suggestion. So it becomes:
Code:
awk '!NF{$0=x}1' file | awk 'gsub (/RUN/, "&") > 1 {sub (/\n *RUN ANY *\n/,"\n")}1' RS=

# 5  
Old 12-08-2013
It worked fine.. but it got rid of empty lines, I dont want that to happen. I need them for my formating purpose.

every job should have a blank line before next jon starts
# 6  
Old 12-08-2013
You can add ORS='\n\n' at the end
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove new line starting with a numeric value and append it to the previous line

Hi, i have a file with multiple entries. After some tests with sed i managed to get the file output as follows: lsn=X-LINK-IN0,apc=661:0,state=avail,avail/links=1/1, 00,2110597,2094790,0,81,529,75649011,56435363, lsn=TM1ITP1-AM1ITP1-LS,apc=500:0,state=avail,avail/links=1/1,... (5 Replies)
Discussion started by: nms
5 Replies

2. UNIX for Beginners Questions & Answers

Grep file starting from pattern matching line

I have a file with a list of references towards the end and want to apply a grep for some string. text .... @unnumbered References @sp 1 @paragraphindent 0 2017. @strong{Chalenski, D.A.}; Wang, K.; Tatanova, Maria; Lopez, Jorge L.; Hatchell, P.; Dutta, P.; @strong{Small airgun... (1 Reply)
Discussion started by: kristinu
1 Replies

3. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies

4. Shell Programming and Scripting

Remove line starting from space till end.

Hi, I have a code tag, from which i have the below snippet: intelrpt.GetCMB_FB type=ODBC> intelrpt.GetCMB_FB type=SYBASE> I want the output like: intelrpt.GetCMB_FB intelrpt.GetCMB_FB That is remove the lines starting from WHITESPACE till end. Please help. I am new to... (7 Replies)
Discussion started by: anupdas
7 Replies

5. Shell Programming and Scripting

How to remove all words starting from a matching word in a line

Hi Guys, I have a file like this: wwwe 1 ioie ewew yyy uuu 88 erehrlk 4 ihoiwhe lkjhassad lkhsad yyy mmm 45 jhash lhasdhs lkhsdkjsn ouiyrshroi oihoihswodnw oiyhewe yyy ggg 77 I want to remove everything after "yyy" and including "yyy" from each line in the file. So I want:... (2 Replies)
Discussion started by: npatwardhan
2 Replies

6. Shell Programming and Scripting

shell script to remove all lines from a file before a line starting with pattern

hi,, i hav a file with many lines.i need to remove all lines before a line begginning with a specific pattern from the file because these lines are not required. Can u help me out with either a perl script or shell script example:- if file initially contains lines: a b c d .1.2 d e f... (2 Replies)
Discussion started by: raksha.s
2 Replies

7. Shell Programming and Scripting

comment/delete a particular pattern starting from second line of the matching pattern

Hi, I have file 1.txt with following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433 ** ** ** In file 2.txt I have the following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433... (4 Replies)
Discussion started by: imas
4 Replies

8. UNIX for Dummies Questions & Answers

modify a particular pattern starting from second line of the search pattern

Hi, I think you ppl did not get my question correctly, let me explain I have 1.txt with following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433 ** ** ** In file 2.txt I have the following entries as shown: ... (1 Reply)
Discussion started by: imas
1 Replies

9. UNIX for Dummies Questions & Answers

modify a particular pattern starting from second line of the search pattern

Hi, I am new to this forum and i would like to get help in this issue. I have a file 1.txt as shown: apple banana orange apple grapes banana orange grapes orange .... Now i would like to search for pattern say apple or orange and then put a # at the beginning of the pattern... (2 Replies)
Discussion started by: imas
2 Replies

10. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies
Login or Register to Ask a Question