Sponsored Content
Top Forums Shell Programming and Scripting Consolidate several lines of a CSV file with firewall rules, in order to parse them easier? Post 302862415 by starriol on Thursday 10th of October 2013 04:46:53 PM
Old 10-10-2013
Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?

Consolidate several lines of a CSV file with firewall rules

Hi guys.
I have a CSV file, which I created using an HTML export from a Check Point firewall policy.
Each rule is represented as several lines, in some cases. That occurs when a rule has several address sources, destinations or services.
I need the output to have each rule described in only one line.
It's easy to distinguish when each rule begins. In the first column, there's the rule ID, which is a number.

Let me show you an example. The strings that should be moved are in bold:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster;mcast_vrrp;;vrrp;accept;Log;fwxcluster;Any;"VRRP;;*Comment suppressed*
;;;;;[b]igmp**;;;;;
2;;fwxcluster;fwxcluster;;FireWall;accept;Log;fwxcluster;Any;"Management FWg;*Comment suppressed*
;;[b]fwmgmpe**;[b]fwmgmpe**;;[b]ssh**;;;;;
;;[b]fwmgm**;[b]fwmgm**;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;[b]cmm_ntpserver_pe02**;;;;;;;

What I need ,explained in pseudo code, is this:

Read the first column of the next line. If there's a number:
Evaluate the first column of the next line. If there's no number there, concatenate (separating with a comma) \
the strings in the columns of this line with the last one and eliminate the text in the current one

The output should be something like this. The strings in bold are the ones that were moved:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster,[b]fwmgmpe**,[b]fwmgm**;mcast_vrrp,[b]fwmgmpe**,[b]fwmgm**;;vrrp,[b]ssh**;accept;Log;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;
;;;;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01,[b]cmm_ntpserver_pe02**;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;

The empty lines are there only to be more clear, I don't actually need them.

Thanks!

Last edited by starriol; 10-11-2013 at 11:34 AM.. Reason: Edit to explain myself better.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using... (2 Replies)
Discussion started by: cdesiks
2 Replies

2. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies

3. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

4. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

5. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

6. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double... (7 Replies)
Discussion started by: vfrg
7 Replies

7. Shell Programming and Scripting

how to parse this file and obtain a .csv or .xls

Hello Expert, I have a file in the following format: SYNTAX_VERSION 5 MONITOR "NAME_TEMPLATES" DESCRIPTION "Monitors for contents of error " INTERVAL "1m" MONPROG "script.sh NAME_TEMPLATES" MAXTHRESHOLD GEN_BELOW_RESET SEVERITY Major ... (17 Replies)
Discussion started by: Ant-one
17 Replies

8. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

9. Shell Programming and Scripting

Multiple lines consolidate

This post is start for me ... I stumped at something that I not sure as to how start on ... I tried so of your script that i honestly lost mind looking and looking here ... please help COL1 COl2 COL3 12222 AUH FLUEH 12222 SSC OPERA 12222 SSC ... (8 Replies)
Discussion started by: Sebastian.Thoma
8 Replies

10. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies
PPI::Token::Comment(3)					User Contributed Perl Documentation				    PPI::Token::Comment(3)

NAME
PPI::Token::Comment - A comment in Perl source code INHERITANCE
PPI::Token::Comment isa PPI::Token isa PPI::Element SYNOPSIS
# This is a PPI::Token::Comment print "Hello World!"; # So it this $string =~ s/ foo # This, unfortunately, is not :( bar /w; DESCRIPTION
In PPI, comments are represented by "PPI::Token::Comment" objects. These come in two flavours, line comment and inline comments. A "line comment" is a comment that stands on its own line. These comments hold their own newline and whitespace (both leading and trailing) as part of the one "PPI::Token::Comment" object. An inline comment is a comment that appears after some code, and continues to the end of the line. This does not include whitespace, and the terminating newlines is considered a separate PPI::Token::Whitespace token. This is largely a convenience, simplifying a lot of normal code relating to the common things people do with comments. Most commonly, it means when you "prune" or "delete" a comment, a line comment disappears taking the entire line with it, and an inline comment is removed from the inside of the line, allowing the newline to drop back onto the end of the code, as you would expect. It also means you can move comments around in blocks much more easily. For now, this is a suitably handy way to do things. However, I do reserve the right to change my mind on this one if it gets dangerously anachronistic somewhere down the line. METHODS
Only very limited methods are available, beyond those provided by our parent PPI::Token and PPI::Element classes. line The "line" accessor returns true if the "PPI::Token::Comment" is a line comment, or false if it is an inline comment. SUPPORT
See the support section in the main module. AUTHOR
Adam Kennedy <adamk@cpan.org> COPYRIGHT
Copyright 2001 - 2011 Adam Kennedy. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. perl v5.16.2 2011-02-25 PPI::Token::Comment(3)
All times are GMT -4. The time now is 06:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy