Sponsored Content
Top Forums Shell Programming and Scripting Consolidate several lines of a CSV file with firewall rules, in order to parse them easier? Post 302863163 by starriol on Saturday 12th of October 2013 08:13:21 PM
Old 10-12-2013
Hi guys!

If anyone needs it, I ended up solving this with this code:

Code:
import csv 
# adjust these 3 lines 
WRITE_EMPTIES = False 
INFILE = "input.csv"
OUTFILE = "output.csv"
with open(INFILE, "r") as in_file: 
  r = csv.reader(in_file, delimiter=";") 
  with open(OUTFILE, "wb") as out_file: 
    previous = None 
    empties_to_write = 0 
    out_writer = csv.writer(out_file, delimiter=";") 
    for i, row in enumerate(r): 
      first_val = row[0].strip() 
      if first_val: 
        if previous: 
          out_writer.writerow(previous) 
          if WRITE_EMPTIES and empties_to_write: 
            out_writer.writerows( 
              [["" for _ in previous]] * empties_to_write 
              ) 
            empties_to_write = 0 
        previous = row 
      else: # append sub-portions to each other 
        previous = [ 
          "|".join( 
            subitem 
            for subitem in existing.split(",") + [new] 
            if subitem 
            ) 
          for existing, new in zip(previous, row) 
          ] 
        empties_to_write += 1 
    if previous: # take care of the last row 
      out_writer.writerow(previous) 
      if WRITE_EMPTIES and empties_to_write: 
        out_writer.writerows( 
          [["" for _ in previous]] * empties_to_write 
          )

 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using... (2 Replies)
Discussion started by: cdesiks
2 Replies

2. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script... (13 Replies)
Discussion started by: lodey
13 Replies

3. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

4. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

5. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

6. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double... (7 Replies)
Discussion started by: vfrg
7 Replies

7. Shell Programming and Scripting

how to parse this file and obtain a .csv or .xls

Hello Expert, I have a file in the following format: SYNTAX_VERSION 5 MONITOR "NAME_TEMPLATES" DESCRIPTION "Monitors for contents of error " INTERVAL "1m" MONPROG "script.sh NAME_TEMPLATES" MAXTHRESHOLD GEN_BELOW_RESET SEVERITY Major ... (17 Replies)
Discussion started by: Ant-one
17 Replies

8. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

9. Shell Programming and Scripting

Multiple lines consolidate

This post is start for me ... I stumped at something that I not sure as to how start on ... I tried so of your script that i honestly lost mind looking and looking here ... please help COL1 COl2 COL3 12222 AUH FLUEH 12222 SSC OPERA 12222 SSC ... (8 Replies)
Discussion started by: Sebastian.Thoma
8 Replies

10. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies
fwb_ipt(1)							 Firewall Builder							fwb_ipt(1)

NAME
fwb_ipt - Policy compiler for iptables SYNOPSIS
fwb_ipt -fdata_file.xml [-4] [-6] [-V] [-dwdir] [-i] [-ooutput.fw] [-Ofw1_id,fw1_output.fw[,fw2_id,fw2_output.fw]] [-v] [-xc] [-xnN] [-xpN] [-xt] object_name DESCRIPTION
fwb_ipt is a firewall policy compiler component of Firewall Builder (see fwbuilder(1)). Compiler reads objects definitions and firewall description from the data file specified with "-f" option and generates resultant iptables script. The script is written to the file with the name the same as the name of the firewall object, plus extension ".fw". The data file and the name of the firewall objects must be specified on the command line. Other command line parameters are optional. OPTIONS
-4 Generate iptables script for IPv4 part of the policy. If any rules of the firewall refer to IPv6 addresses, compiler will skip these rules. Options "-4" and "-6" are exclusive. If neither option is used, compiler tries to generate both parts of the script, although generation of the IPv6 part is controlled by the option "Enable IPv6 support" in the "IPv6" tab of the firewall object advanced settings dialog. This option is off by default. -6 Generate iptables script for IPv6 part of the policy. If any rules of the firewall refer to IPv6 addresses, compiler will skip these rules. -f FILE Specify the name of the data file to be processed. -o output.fw Specify output file name -O fw1_id,fw1_output.fw[,fw2_id,fw2_output.fw] The argument is a comma separated list of firewall object IDs and corresponding output file names. This option is used by fwbuilder GUI while compiling firewall clusters. -d wdir Specify working directory. Compiler creates file with iptables script in this directory. If this parameter is missing, then ipta- bles script will be placed in the current working directory. -v Be verbose: compiler prints diagnostic messages when it works. -V Print version number and quit. -i When this option is present, the last argument on the command line is supposed to be firewall object ID rather than its name -xc When output file name is determined automatically (i.e. flags -o or -O are not present), the file name is composed of the cluster name and member firewall name rather than just member firewall name. This is used mostly for testing when the same member firewall object can be a part of different clusters with different configurations. -xt This flag makes compiler treat all fatal errors as warnings and continue processing rules. Generated configuration script most likely will be incorrect but will include error message as a comment; this flag is used for testing and debugging. -xp N Debugging flag: this causes compiler to print detailed description of the policy rule number "N" as it precesses it, step by step. -xn N Debugging flag: this causes compiler to print detailed description of the NAT rule number "N" as it precesses it, step by step. URL
Firewall Builder home page is located at the following URL: http://www.fwbuilder.org/ BUGS
Please report bugs using bug tracking system on SourceForge: http://sourceforge.net/tracker/?group_id=5314&atid=105314 SEE ALSO
fwbuilder(1), fwb_ipf(1), fwb_pf(1) FWB
fwb_ipt(1)
All times are GMT -4. The time now is 11:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy