Sponsored Content
Top Forums Shell Programming and Scripting Consolidate several lines of a CSV file with firewall rules, in order to parse them easier? Post 302862415 by starriol on Thursday 10th of October 2013 04:46:53 PM
Old 10-10-2013
Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?

Consolidate several lines of a CSV file with firewall rules

Hi guys.
I have a CSV file, which I created using an HTML export from a Check Point firewall policy.
Each rule is represented as several lines, in some cases. That occurs when a rule has several address sources, destinations or services.
I need the output to have each rule described in only one line.
It's easy to distinguish when each rule begins. In the first column, there's the rule ID, which is a number.

Let me show you an example. The strings that should be moved are in bold:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster;mcast_vrrp;;vrrp;accept;Log;fwxcluster;Any;"VRRP;;*Comment suppressed*
;;;;;[b]igmp**;;;;;
2;;fwxcluster;fwxcluster;;FireWall;accept;Log;fwxcluster;Any;"Management FWg;*Comment suppressed*
;;[b]fwmgmpe**;[b]fwmgmpe**;;[b]ssh**;;;;;
;;[b]fwmgm**;[b]fwmgm**;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;[b]cmm_ntpserver_pe02**;;;;;;;

What I need ,explained in pseudo code, is this:

Read the first column of the next line. If there's a number:
Evaluate the first column of the next line. If there's no number there, concatenate (separating with a comma) \
the strings in the columns of this line with the last one and eliminate the text in the current one

The output should be something like this. The strings in bold are the ones that were moved:

Code:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster,[b]fwmgmpe**,[b]fwmgm**;mcast_vrrp,[b]fwmgmpe**,[b]fwmgm**;;vrrp,[b]ssh**;accept;Log;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;
;;;;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01,[b]cmm_ntpserver_pe02**;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;;;;;;;;

The empty lines are there only to be more clear, I don't actually need them.

Thanks!

Last edited by starriol; 10-11-2013 at 11:34 AM.. Reason: Edit to explain myself better.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using... (2 Replies)
Discussion started by: cdesiks
2 Replies

2. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies

3. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

4. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

5. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

6. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double... (7 Replies)
Discussion started by: vfrg
7 Replies

7. Shell Programming and Scripting

how to parse this file and obtain a .csv or .xls

Hello Expert, I have a file in the following format: SYNTAX_VERSION 5 MONITOR "NAME_TEMPLATES" DESCRIPTION "Monitors for contents of error " INTERVAL "1m" MONPROG "script.sh NAME_TEMPLATES" MAXTHRESHOLD GEN_BELOW_RESET SEVERITY Major ... (17 Replies)
Discussion started by: Ant-one
17 Replies

8. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

9. Shell Programming and Scripting

Multiple lines consolidate

This post is start for me ... I stumped at something that I not sure as to how start on ... I tried so of your script that i honestly lost mind looking and looking here ... please help COL1 COl2 COL3 12222 AUH FLUEH 12222 SSC OPERA 12222 SSC ... (8 Replies)
Discussion started by: Sebastian.Thoma
8 Replies

10. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies
SHOREWALL6-ACCOUNTI(5)						  [FIXME: manual]					    SHOREWALL6-ACCOUNTI(5)

NAME
accounting - Shorewall6 Accounting file SYNOPSIS
/etc/shorewall6/accounting DESCRIPTION
Accounting rules exist simply to count packets and bytes in categories that you define in this file. You may display these rules and their packet and byte counters using the shorewall6 show accounting command. Beginning with Shorewall 4.4.18, the accounting structure can be created with three root chains: o accountin: Rules that are valid in the INPUT chain (may not specify an output interface). o accountout: Rules that are valid in the OUTPUT chain (may not specify an input interface or a MAC address). o accounting: Other rules. The new structure is enabled by sectioning the accounting file in a manner similar to the rules file[1]. The sections are INPUT, OUTPUT and FORWARD and must appear in that order (although any of them may be omitted). The first non-commentary record in the accounting file must be a section header when sectioning is used. Warning If sections are not used, the Shorewall rules compiler cannot detect certain violations of netfilter restrictions. These violations can result in run-time errors such as the following: ip6tables-restore v1.4.13: Can't use -o with INPUT Beginning with Shorewall 4.4.20, the ACCOUNTING_TABLE setting was added to shorewall.conf and shorewall6.conf. That setting determines the Netfilter table (filter or mangle) where the accounting rules are added. When ACCOUNTING_TABLE=mangle is specified, the available sections are PREROUTING, INPUT, OUTPUT, FORWARD and POSTROUTING. Section headers have the form: SECTION section-name When sections are enabled: o A jump to a user-defined accounting chain must appear before entries that add rules to that chain. This eliminates loops and unreferenced chains. o An output interface may not be specified in the PREROUTING and INPUT sections. o In the OUTPUT and POSTROUTING sections: o An input interface may not be specified o Jumps to a chain defined in the INPUT or PREROUTING sections that specifies an input interface are prohibited o MAC addresses may not be used o Jump to a chain defined in the INPUT or PREROUTING section that specifies a MAC address are prohibited. o The default value of the CHAIN column is: o accountin in the INPUT section o accounout in the OUTPUT section o accountfwd in the FORWARD section o accountpre in the PREROUTING section o accountpost in the POSTROUTING section o Traffic addressed to the firewall goes through the rules defined in the INPUT section. o Traffic originating on the firewall goes through the rules defined in the OUTPUT section. o Traffic being forwarded through the firewall goes through the rules from the FORWARD sections. The columns in the file are as follows (where the column name is followed by a different name in parentheses, the different name is used in the alternate specification syntax). ACTION - {COUNT|DONE|chain[:{COUNT|JUMP}]|COMMENT comment} What to do when a matching packet is found. COUNT Simply count the match and continue with the next rule DONE Count the match and don't attempt to match any other accounting rules in the chain specified in the CHAIN column. chain[:COUNT] Where chain is the name of a chain; shorewall6 will create the chain automatically if it doesn't already exist. Causes a jump to that chain to be added to the chain specified in the CHAIN column. If :COUNT is included, a counting rule matching this entry will be added to chain. The chain may not exceed 29 characters in length and may be composed of letters, digits, dash ('-') and underscore ('_'). chain:JUMP Like the previous option without the :COUNT part. NFLOG[(nflog-parameters)] - Added in Shorewall-4.4.20. Causes each matching packet to be sent via the currently loaded logging backend (usually nfnetlink_log) where it is available to accounting daemons through a netlink socket. COMMENT The remainder of the line is treated as a comment which is attached to subsequent rules until another COMMENT line is found or until the end of the file is reached. To stop adding comments to rules, use a line with only the word COMMENT. CHAIN - {-|chain} The name of a chain. If specified as - the accounting chain is assumed. This is the chain where the accounting rule is added. The chain will be created if it doesn't already exist. The chain may not exceed 29 characters in length. SOURCE - {-|any|all|interface|interface:[address]|address} Packet Source. The name of an interface, an address (host or net) or an interface name followed by ":" and a host or net address. DESTINATION (dest) - {-|any|all|interface|interface:[address]|address} Packet Destination. Format same as SOURCE column. PROTOCOL (proto) - {-|any|all|protocol-name|protocol-number|ipp2p[:{udp|all}]} A protocol-name (from protocols(5)), a protocol-number, ipp2p, ipp2p:udp or ipp2p:all DEST PORT(S) (dport) - {-|any|all|ipp2p-option|port-name-or-number[,port-name-or-number]...} Destination Port number. Service name from services(5) or port number. May only be specified if the protocol is TCP (6), UDP (17), DCCP (33), SCTP (132) or UDPLITE (136). You may place a comma-separated list of port names or numbers in this column if your kernel and ip6tables include multiport match support. If the PROTOCOL is ipp2p then this column must contain an ipp2p-option ("ip6tables -m ipp2p --help") without the leading "--". If no option is given in this column, ipp2p is assumed. SOURCE PORT(S) (sport) - {-|any|all|port-name-or-number[,port-name-or-number]...} Service name from services(5) or port number. May only be specified if the protocol is TCP (6), UDP (17), DCCP (33), SCTP (132) or UDPLITE (136). You may place a comma-separated list of port numbers in this column if your kernel and ip6tables include multiport match support. USER/GROUP (user) - [!][user-name-or-number][:group-name-or-number][+program-name] This column may only be non-empty if the CHAIN is OUTPUT. When this column is non-empty, the rule applies only if the program generating the output is running under the effective user and/or group specified (or is NOT running under that id if "!" is given). Examples: joe program must be run by joe :kids program must be run by a member of the 'kids' group !:kids program must not be run by a member of the 'kids' group +upnpd #program named upnpd Important The ability to specify a program name was removed from Netfilter in kernel version 2.6.14. MARK - [!]value[/mask][:C] Defines a test on the existing packet or connection mark. The rule will match only if the test returns true. If you don't want to define a test but need to specify anything in the following columns, place a "-" in this field. ! Inverts the test (not equal) value Value of the packet or connection mark. mask A mask to be applied to the mark before testing. :C Designates a connection mark. If omitted, the packet mark's value is tested. IPSEC - option-list (Optional - Added in Shorewall 4.4.13 but broken until 4.5.4.1 ) The option-list consists of a comma-separated list of options from the following list. Only packets that will be encrypted or have been de-crypted via an SA that matches these options will have their source address changed. May only be specified when sections are used. reqid=number where number is specified using setkey(8) using the 'unique:number option for the SPD level. spi=<number> where number is the SPI of the SA used to encrypt/decrypt packets. proto=ah|esp|ipcomp IPSEC Encapsulation Protocol mss=number sets the MSS field in TCP packets mode=transport|tunnel IPSEC mode tunnel-src=address[/mask] only available with mode=tunnel tunnel-dst=address[/mask] only available with mode=tunnel strict Means that packets must match all rules. next Separates rules; can only be used with strict yes or ipsec When used by itself, causes all traffic that will be encrypted/encapsulated or has been decrypted/un-encapsulted to match the rule. no or none When used by itself, causes all traffic that will not be encrypted/encapsulated or has been decrypted/un-encapsulted to match the rule. in May only be used in the FORWARD section and must be the first or the only item the list. Indicates that matching packets have been decrypted in input. out May only be used in the FORWARD section and must be the first or the only item in the list. Indicates that matching packets will be encrypted on output. If this column is non-empty and sections are not used, then: o A chain NAME appearing in the ACTION column must be a chain branched either directly or indirectly from the accipsecin or accipsecout chain. o The CHAIN column must contain either accipsecin or accipsecout or a chain branched either directly or indirectly from those chains. o These rules will NOT appear in the accounting chain. HEADERS - [!][any:|exactly:]header-list (Optional - Added in Shorewall 4.4.15) The header-list consists of a comma-separated list of headers from the following list. auth, ah, or 51 Authentication Headers extension header. esp, or 50 Encrypted Security Payload extension header. hop, hop-by-hop or 0 Hop-by-hop options extension header. route, ipv6-route or 41 IPv6 Route extension header. frag, ipv6-frag or 44 IPv6 fragmentation extension header. none, ipv6-nonxt or 59 No next header proto, protocol or 255 Any protocol header. If any: is specified, the rule will match if any of the listed headers are present. If exactly: is specified, the will match packets that exactly include all specified headers. If neither is given, any: is assumed. If ! is entered, the rule will match those packets which would not be matched when ! is omitted. In all of the above columns except ACTION and CHAIN, the values -, any and all may be used as wildcards. Omitted trailing columns are also treated as wildcards. FILES
/etc/shorewall6/accounting SEE ALSO
http://shorewall.net/Accounting.html[2] http://shorewall.net/shorewall_logging.html http://shorewall.net/configuration_file_basics.htm#Pairs shorewall6(8), shorewall6-actions(5), shorewall6-blacklist(5), shorewall6-hosts(5), shorewall6-interfaces(5), shorewall6-maclist(5), shoewall6-netmap(5),shorewall6-params(5), shorewall6-policy(5), shorewall6-providers(5), shorewall6-rtrules(5), shorewall6-routestopped(5), shorewall6-rules(5), shorewall6.conf(5), shorewall6-secmarks(5), shorewall6-tcclasses(5), shorewall6-tcdevices(5), shorewall6-tcrules(5), shorewall6-tos(5), shorewall6-tunnels(5), shorewall6-zones(5) NOTES
1. rules file http://www.shorewall.net/manpages6/shorewall-rules.html 2. http://shorewall.net/Accounting.html http://shorewall.net/Accounting.html [FIXME: source] 06/28/2012 SHOREWALL6-ACCOUNTI(5)
All times are GMT -4. The time now is 06:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy