Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?
# 1  
Old 10-10-2013
Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?
Consolidate several lines of a CSV file with firewall rules

Hi guys.
I have a CSV file, which I created using an HTML export from a Check Point firewall policy.
Each rule is represented as several lines, in some cases. That occurs when a rule has several address sources, destinations or services.
I need the output to have each rule described in only one line.
It's easy to distinguish when each rule begins. In the first column, there's the rule ID, which is a number.

Let me show you an example. The strings that should be moved are in bold:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster;mcast_vrrp;;vrrp;accept;Log;fwxcluster;Any;"VRRP;;*Comment suppressed*
;;;;;[b]igmp**;;;;;
2;;fwxcluster;fwxcluster;;FireWall;accept;Log;fwxcluster;Any;"Management FWg;*Comment suppressed*
;;[b]fwmgmpe**;[b]fwmgmpe**;;[b]ssh**;;;;;
;;[b]fwmgm**;[b]fwmgm**;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;[b]cmm_ntpserver_pe02**;;;;;;;

What I need ,explained in pseudo code, is this:

Read the first column of the next line. If there's a number:
Evaluate the first column of the next line. If there's no number there, concatenate (separating with a comma) \
the strings in the columns of this line with the last one and eliminate the text in the current one

The output should be something like this. The strings in bold are the ones that were moved:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster,[b]fwmgmpe**,[b]fwmgm**;mcast_vrrp,[b]fwmgmpe**,[b]fwmgm**;;vrrp,[b]ssh**;accept;Log;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;
;;;;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01,[b]cmm_ntpserver_pe02**;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;

The empty lines are there only to be more clear, I don't actually need them.

Thanks!
Last edited by starriol; 10-11-2013 at 11:34 AM.. Reason: Edit to explain myself better.
# 2  
Old 10-10-2013
Try this awk solution:

Code:
awk -F\; '
$1{Save=$0}
/^;/{
  for(i=split(Save,Prev,";");i;i--)
    if (!length($i)) $i=Prev[i]
}1' OFS=\; infile

# 3  
Old 10-11-2013
Quote:
Originally Posted by Chubler_XL
Try this awk solution:

Code:
awk -F\; '
$1{Save=$0}
/^;/{
  for(i=split(Save,Prev,";");i;i--)
    if (!length($i)) $i=Prev[i]
}1' OFS=\; infile

Hey Cluber, thanks for this code.
My knowledge of awk is null, basically.
I ran it and got to this:
Code:
pgawk: script.awk:1: awk -F\; '
pgawk: script.awk:1:       ^ backslash not last character on line

I'm running this with Gawk for Windows (not at my laptop here).
Is that the reason why I'm getting this error?
If it is, I'll get a Linux VM running to test it out.

Thanks.
Moderator's Comments:
Mod Comment Use CODE tags for sample input and sample output; not just for code segments. Without the CODE tags, the spacing in the diagnostic messages from awk is lost.
Last edited by Don Cragun; 10-13-2013 at 12:08 AM.. Reason: Add CODE tags
# 4  
Old 10-12-2013
Hi guys!

If anyone needs it, I ended up solving this with this code:

Code:
import csv 
# adjust these 3 lines 
WRITE_EMPTIES = False 
INFILE = "input.csv"
OUTFILE = "output.csv"
with open(INFILE, "r") as in_file: 
  r = csv.reader(in_file, delimiter=";") 
  with open(OUTFILE, "wb") as out_file: 
    previous = None 
    empties_to_write = 0 
    out_writer = csv.writer(out_file, delimiter=";") 
    for i, row in enumerate(r): 
      first_val = row[0].strip() 
      if first_val: 
        if previous: 
          out_writer.writerow(previous) 
          if WRITE_EMPTIES and empties_to_write: 
            out_writer.writerows( 
              [["" for _ in previous]] * empties_to_write 
              ) 
            empties_to_write = 0 
        previous = row 
      else: # append sub-portions to each other 
        previous = [ 
          "|".join( 
            subitem 
            for subitem in existing.split(",") + [new] 
            if subitem 
            ) 
          for existing, new in zip(previous, row) 
          ] 
        empties_to_write += 1 
    if previous: # take care of the last row 
      out_writer.writerow(previous) 
      if WRITE_EMPTIES and empties_to_write: 
        out_writer.writerows( 
          [["" for _ in previous]] * empties_to_write 
          )

# 5  
Old 10-13-2013
You can still use gawk on windows.

Place code in file like expand_csv.awk:
Code:
BEGIN{OFS=FS}
$1{Save=$0}
/^;/{
  for(i=split(Save,Prev,";");i;i--)
    if (!length($i)) $i=Prev[i]
}1

And run it from the windows command line like this:

Code:
C:> awk -F; -f expand_csv.awk infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies

2. Shell Programming and Scripting

Multiple lines consolidate

This post is start for me ... I stumped at something that I not sure as to how start on ... I tried so of your script that i honestly lost mind looking and looking here ... please help COL1 COl2 COL3 12222 AUH FLUEH 12222 SSC OPERA 12222 SSC ... (8 Replies)
Discussion started by: Sebastian.Thoma
8 Replies

3. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

4. Shell Programming and Scripting

how to parse this file and obtain a .csv or .xls

Hello Expert, I have a file in the following format: SYNTAX_VERSION 5 MONITOR "NAME_TEMPLATES" DESCRIPTION "Monitors for contents of error " INTERVAL "1m" MONPROG "script.sh NAME_TEMPLATES" MAXTHRESHOLD GEN_BELOW_RESET SEVERITY Major ... (17 Replies)
Discussion started by: Ant-one
17 Replies

5. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double... (7 Replies)
Discussion started by: vfrg
7 Replies

6. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

7. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

8. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

9. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies

10. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using... (2 Replies)
Discussion started by: cdesiks
2 Replies
Login or Register to Ask a Question