Shell script to merge and delete lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to merge and delete lines
# 1  
Old 11-29-2018
Shell script to merge and delete lines

Code:
POLY_STATS   
{     
EqName 103_tri    
Id 123    
act_polyCount 1    
act_polyValue 0    
}     
POLY_STATS
{     
EqName 103_tri    
Id 123    
pass_polyCount 2
pass_polyValue 0    
}     
POLY_STATS
{     
EqName 103    
Id 123    
mes_polyCount 2    
mes_polyValue 0    
}     
POLY_STATS
{   
EqName 504    
Id 123    
mes_polyCount 2    
mes_polyValue 0    
}     
}

My requirement is a shell script to merge the block where the EqName and Id are same and remove the second block.The value of EqName and Id may change. So there should be a dynamic approach to get the value of EqName and Id
The output should be:
Code:
POLY_STATS   
{     
EqName 103_tri    
Id 123    
act_polyCount 1    
act_polyValue 0 
pass_polyCount 2
pass_polyValue 0    
}     
 
POLY_STATS
{     
EqName 103    
Id 123    
mes_polyCount 2    
mes_polyValue 0    
}     
POLY_STATS
{   
EqName 504    
Id 123    
mes_polyCount 2    
mes_polyValue 0    
}     
}

The algorithm i am thinking of is:
Get the value of EqName
Get the value of Id
Match for the second occurence of EqName and Id
Get the two lines below the second occurence of Id and move those two lines after the first occurence of Id.
Delete two lines above and 4 lines below the first occurence of EqName

Is the above algorithm seems good .Can anyone pl. help me with any other oneliners or using awk/sed by which we can do this easily
# 2  
Old 11-29-2018
Is this a homework assignment?

What operating system are you using?

What shell are you using?

Why are there two closing braces (}) at the end of your input and output files?

Are the blocks you want to merge always located adjacently in you input files, or do they just happen to be that way in your sample input?

What do you mean by "The value of EqName and Id may change. So there should be a dynamic approach to get the value of EqName and Id"? Are you saying the values following those fixed strings may change? (That seems obvious as a means of determining which blocks to merge.) Or, are you saying that the fixed strings EqName and Id may change as well as the values that follow them? Is there some reason that you can't just compare the first pair of lines following a line containing an opening brace without caring what characters are on those pairs of lines?

What have you tried to solve this problem on your own? Where are you stuck?
# 3  
Old 11-30-2018
Hi,
I am using Linux operating system and using bash shell.
The } are part of the input file and cannot be changed as i need them in the output file as well.
The blocks i want to merge is not always located adjacent to each other. They can be positioned at any line number in the file.
Eq Name and Id are fixed strings

I have tried with the following code and i am stuck on how to move the lines and delete the block.
Code:
#cat -n file
 1  POLY_STATS
     2  {
     3  EqName 103_tri
     4  Id 123
     5  act_polyCount 1
     6  act_polyValue 0
     7  }
     8  POLY_STATS
     9  {
    10  EqName 103_tri
    11  Id 123
    12  pass_polyCount 2
    13  pass_polyValue 0
    14  }
    15  POLY_STATS
    16  {
    17  EqName 103
    18  Id 123
    19  mes_polyCount 2
    20  mes_polyValue 0
    21  }
    22  POLY_STATS
    23  {
    24  EqName 504
    25  Id 123
    26  mes_polyCount 2
    27  mes_polyValue 0
    28  }
    29  }

Code:
 sort file | uniq -c | grep EqName|gawk '$1==2{print $3}' #Get the value of Eqname,which is repeating twice
 cat -n file|sed -n '/103_tri/p' #get line number where EqName is coming twice
 From=$(cat -n file|sed -n '/103_tri/p'|awk '{print $1}'|tail -1)
 echo $From #10
 From=$(echo "$From + 2" | bc)
 To=$(echo "$From + 1" | bc)
 sed -n $From,${To}p file # Printing the lines from the second block which i want to move to the first block

I have printed the lines to be moved
Code:
pass_polyCount 2
pass_polyValue 0

Now i want to insert the above two lines after line no 6 and delete the lines from 8 to 14. Pl help
# 4  
Old 11-30-2018
You did not answer Don Cragun's first question. And, if the records belonging to each other are not adjacent, what order should they be printed in?
# 5  
Old 11-30-2018
This looks like a continuation of your earlier thread in September (rather than homework/coursework).
Here is an awk script that does a merge like you want:
Code:
#!/bin/sh
awk '
$1=="}" { bl-- }
bl>0 {
  # we are within a { } frame
  if ($1=="EqName") {
    eqname=$0
  } else if ($1=="Id") {
    id=$0
  } else {
    # ORS is a newline
    key=(eqname ORS id)
    if (key in store) {
      # append string
      store[key]=(store[key] ORS $0)
    } else {
      # new string
      store[key]=$0
      # enforce the original order with a number-indexed array
      ind[++cnt]=key
    }
  }
}
$1=="{" { bl++ }
END {
  # walk through the number-indexed array
  # "for (key in store)" would be a random order
  for (c=1; c<=cnt; c++) {
    key=ind[c]
    print "POLY_STATS"
    print "{"
    print key
    print store[key]
    print "}"
  }
}
' file

The { } frame and the text outside are hard-coded in print statements.
This User Gave Thanks to MadeInGermany For This Post:
# 6  
Old 12-01-2018
Thanks a ton !! It helped.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge multi-lines into one single line using shell script or Linux command

Hi, Can anyone help me for merge the following multi-line log which beginning with a " and line ending with ": into one line. *****Original Log***** 087;2008-12-06;084403;"mc;;SYHLR6AP1D\LNZW;AD-703;1;12475;SYHLR6AP1B;1.1.1.1;0000000062;HGPDI:MSISDN=12345678,APNID=1,EQOSID=365;... (3 Replies)
Discussion started by: rajeshlinux2010
3 Replies

2. Shell Programming and Scripting

Delete all CONSECUTIVE text lines from file shell scripting

Hi I have a text file like below. THe content of the text will vary. Entire text file have four consecutive lines followed with blank line. I want to delete the occurrence of the two consicutive lines in the text file. I don't have pattern to match and delete. Just i need to delete all... (5 Replies)
Discussion started by: RJSKR28
5 Replies

3. Shell Programming and Scripting

Bash/shell merge similar lines

Hello, I've been working on a bash script to parse through firewall logs (cisco). I'm nearing the end and have a dilemma. My data looks as such (actual data is several gigs worth of logs - without the headers): sourceIP destinationIP destinationProtocol destinationPort 1.1.1.1 2.2.2.2 ... (2 Replies)
Discussion started by: umang2382
2 Replies

4. Shell Programming and Scripting

How to merge Expect script inside shell script?

Hi I have two scripts one is Expect and other is shell. I want to merge Expect code in to Shell script so that i can run it using only one script. Can somebody help me out ? Order to execute: Run Expect_install.sh first and then when installation completes run runTests.sh shell script. ... (1 Reply)
Discussion started by: ashish_neekhra
1 Replies

5. UNIX for Dummies Questions & Answers

How to delete lines above a certin line number in bash shell

Hi, I have written a script that returns the line number of the pattern i want and i stored the line number in a variable(getlinenumber).Now i want to delete all the lines in a file above this line number which is stored in a variable. i am using sed '1,$getlinenumberd' > file1.txt which is... (2 Replies)
Discussion started by: learninguser235
2 Replies

6. Shell Programming and Scripting

How to select/delete some lines in shell?

I need to delete half(approx) the file or select half the file by existence of some character My file looks like 1 2 3 4 . . . 50 . . 100I need to select only 50 to rest of the file or needs to delete the file upto 50. Please help me out.. (6 Replies)
Discussion started by: SujeethP
6 Replies

7. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

8. Shell Programming and Scripting

Help with merge Shell Script

I have a file test.log The content of the file is : a:R_yz:1 a:R_cd:2 a:F_bc:0 a:F_xx:3 b:R_dg:5 b:R_gf:1 b:F_fd:4 I want the output is : :a R_yz 1 3 3 : R_cd 2 : F_bc 0 : F_xx 3 :b R_dg 5 6 4 : R_gf 1 : F_fd 4 (8 Replies)
Discussion started by: mnmonu
8 Replies

9. Shell Programming and Scripting

Need Shell Script to delete lines in a file

Hi All, I have a file with 3 columns (Bank Name, Account Number and Amount). My requirement, I need to delete lines using Unix Shell script: 1. Which are having Alphanumeric characters in Account Number (eg. Line3). 2. Which are having 0.00 in amount. (eg. Line4) 3. And also I need to... (4 Replies)
Discussion started by: phani333
4 Replies

10. Shell Programming and Scripting

shell script to merge files

Can anybody help me out with this problem " a shell program that takes one or any number of file names as input; sorts the lines of each file in ascending order and displays the non blank lines of each sorted file and merge them as one combined sorted file. The program generates an error... (1 Reply)
Discussion started by: arya
1 Replies
Login or Register to Ask a Question