Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?
Top Forums Shell Programming and Scripting Consolidate several lines of a CSV file with firewall rules, in order to parse them easier? Post 302862415 by starriol on Thursday 10th of October 2013 04:46:53 PM
Old 10-10-2013
Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?
Consolidate several lines of a CSV file with firewall rules

Hi guys.
I have a CSV file, which I created using an HTML export from a Check Point firewall policy.
Each rule is represented as several lines, in some cases. That occurs when a rule has several address sources, destinations or services.
I need the output to have each rule described in only one line.
It's easy to distinguish when each rule begins. In the first column, there's the rule ID, which is a number.

Let me show you an example. The strings that should be moved are in bold:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster;mcast_vrrp;;vrrp;accept;Log;fwxcluster;Any;"VRRP;;*Comment suppressed*
;;;;;[b]igmp**;;;;;
2;;fwxcluster;fwxcluster;;FireWall;accept;Log;fwxcluster;Any;"Management FWg;*Comment suppressed*
;;[b]fwmgmpe**;[b]fwmgmpe**;;[b]ssh**;;;;;
;;[b]fwmgm**;[b]fwmgm**;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;[b]cmm_ntpserver_pe02**;;;;;;;

What I need ,explained in pseudo code, is this:

Read the first column of the next line. If there's a number:
Evaluate the first column of the next line. If there's no number there, concatenate (separating with a comma) \
the strings in the columns of this line with the last one and eliminate the text in the current one

The output should be something like this. The strings in bold are the ones that were moved:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster,[b]fwmgmpe**,[b]fwmgm**;mcast_vrrp,[b]fwmgmpe**,[b]fwmgm**;;vrrp,[b]ssh**;accept;Log;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;
;;;;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01,[b]cmm_ntpserver_pe02**;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;

The empty lines are there only to be more clear, I don't actually need them.

Thanks!
Last edited by starriol; 10-11-2013 at 11:34 AM.. Reason: Edit to explain myself better.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using... (2 Replies)
Discussion started by: cdesiks
2 Replies

2. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies

3. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

4. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

5. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

6. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double... (7 Replies)
Discussion started by: vfrg
7 Replies

7. Shell Programming and Scripting

how to parse this file and obtain a .csv or .xls

Hello Expert, I have a file in the following format: SYNTAX_VERSION 5 MONITOR "NAME_TEMPLATES" DESCRIPTION "Monitors for contents of error " INTERVAL "1m" MONPROG "script.sh NAME_TEMPLATES" MAXTHRESHOLD GEN_BELOW_RESET SEVERITY Major ... (17 Replies)
Discussion started by: Ant-one
17 Replies

8. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

9. Shell Programming and Scripting

Multiple lines consolidate

This post is start for me ... I stumped at something that I not sure as to how start on ... I tried so of your script that i honestly lost mind looking and looking here ... please help COL1 COl2 COL3 12222 AUH FLUEH 12222 SSC OPERA 12222 SSC ... (8 Replies)
Discussion started by: Sebastian.Thoma
8 Replies

10. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies

LEARN ABOUT DEBIAN

apf

APF(1)							      General Commands Manual							    APF(1)

NAME

       apf - easy iptables based firewall system

SYNOPSIS

       apf

DESCRIPTION

       This  manual page documents briefly the apf command.  This manual page was written for the Debian distribution because the original program
       does not have a manual page.

       Advanced Policy Firewall (APF) is an iptables (netfilter) based firewall system designed around the essential  needs  of  today's  Internet
       deployed  servers  and the unique needs of custom deployed Linux installations. The configuration of APF is designed to be very informative
       and present the user with an easy to follow process, from top to bottom of the configuration file. The management of APF  on  a	day-to-day
       basis  is  conducted from the command line with the 'apf' command, which includes detailed usage information and all the features one would
       expect from a current and forward thinking firewall solution.

OPTIONS

       apf follow the usual GNU command line syntax, with long options starting with two dashes (`-').	A summary of options is included below.

       -s|--start
	      load all firewall rules

       -r|--restart
	      stop (flush) & reload firewall rules

       -l|--list
	      list all firewall rules

       -t|--status
	      output firewall status log

       -e|--refresh
	      refresh & resolve dns names in trust rules

       -a <HOST CMT|--allow <HOST COMMENT>
	      add host (IP/FQDN) to allow_hosts.rules and immediately load new rule into firewall

       -d <HOST CMT|--deny <HOST COMMENT>
	      add host (IP/FQDN) to deny_hosts.rules and immediately load new rule into firewall

       -u <HOST>|--remove <HOST>
	      remove host from [glob]*_hosts.rules and immediately remove rule from firewall

       -o|--ovars
	      output all configuration options

COPYRIGHT

       Copyright (C) 1999-2007, R-fx Networks <proj@r-fx.org>
       Copyright (C) 2007, Ryan MacDonald <ryan@r-fx.org> This program may be freely redistributed under the terms of the GNU GPL

       This manual page was written by Giuseppe Iuculano <giuseppe@iuculano.it>, for the Debian project (but may be used by others).

								  August 17, 2008							    APF(1)
All times are GMT -4. The time now is 03:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy