Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

CSV File parse help in Perl

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting CSV File parse help in Perl
# 8  
Old 03-29-2008
thanks for ur help....sorry for annoying u with my questions
# 9  
Old 03-29-2008
Hey, no problem.

If you want development ideas for your own script, think about the following.

The Unix toolkit design dictates that tools should read standard input and print to standard output. I'd suggest you try to think about how to change your script to a general-purpose tool. Start by throwing out the file handling -- the invoker can specify which file to read, and where to redirect the output.

How about an "uncut" tool? ("un" is supposed to stand for "unique", here.) Like cut, but only print the first occurrence of a given combination of field values. (Then expand it to accept other field separators than tab.)

This is just one out of many possibilities, but you should always be thinking about how to generalize your work. Over the years, you build up a collection of tools which suit your way of thinking, your way of working.

Your script doesn't do any actual parsing of the input yet -- the effect of the -a option can be duplicated with @F = split /\t/. I imagine you can take it from there.
# 10  
Old 03-30-2008
I have put the split in as a while loop...but it still doesnt work...I have done this correctly?

Code:
#!/usr/bin/perl -w

use strict;
my $csvfile = 'probecards.csv';
my $newfile = 'new.csv';
my $fieldnames = 1;
open (IN, "<$csvfile")  or die "Couldn't open input CSV file: $!";
open (OUT, ">$newfile") or die "Couldn't open output file: $!";
my $header;
$header = <IN> if $fieldnames;
my @data = sort <IN>;
while( <IN> ) {
    push @data, join "\t", (split /\t/)[4,5,8];
}
print OUT $header;
my $n = 0;
my $lastline = '';
foreach my $currentline (@data) {

  next if $currentline eq $lastline;
  print OUT $currentline;
  $lastline = $currentline;
  $n++;
}
close IN; close OUT;
print "Processing complete. In = " . scalar @data . " records, Out = $n records\n";

# 11  
Old 03-30-2008
Looks correct, superficially, but the sort needs to happen after the split; sort @data when you've finished reading the input. And you attempt to read <IN> twice, but taking out the line where you assign @data = sort <IN> will fix that, too. (You need to leave the my @data declaration, of course.)

The field numbers don't match what was discussed earlier but I guess you know what you are doing there. Or maybe not? They are off by two so maybe you adjusted them the wrong way. Array indices are zero-based in Perl so the first field is [0], the second is [1], the third is [2], and so forth.
Last edited by era; 03-30-2008 at 03:03 PM.. Reason: Explain zero-based indices a little bit more after all
# 12  
Old 03-30-2008
Hi era,

I removed the sort from the code and ran the script. Now what happens is that when it has been ran the number of duplicates are now zero...the complete opposite to what was happening before. I looked at the split and yes it was a mistype...it looks like I am doing something wrong as I know that the field for the operation and the tester do have duplicates
# 13  
Old 03-31-2008
You are removing duplicates with the next if ...
# 14  
Old 03-31-2008
I think that I have it sorted now.

code is as follows:
Code:
#!/usr/bin/perl -w

use strict;
my $csvfile = 'probecards.csv';
my $newfile = 'new.csv';
my $fieldnames = 1;
open (IN, "<$csvfile")  or die "Couldn't open input CSV file: $!";
open (OUT, ">$newfile") or die "Couldn't open output file: $!";

my $header;
$header = <IN> if $fieldnames;
my @data ;
while( <IN> ) {
my (${LAO_START_WW},${PROGRAM},${ID},${OP},${PROBE_CARD},${DEVREVSTEP},${TEST_START},${TESTER_ID} ) = split(',',$_);
my $tempString = "${OP},${PROBE_CARD},${TESTER_ID}";
push @data,$tempString;
}
print OUT $header;
my $n = 0;
my $lastline = '';

foreach my $currentline (@data) {

  next if $currentline eq $lastline;
  print OUT $currentline;
  $lastline = $currentline;
  $n++;
}
close IN; close OUT;
print "Processing complete. In = " . scalar @data . " records, Out = $n records\n";

The only issue that I have now is that the header fields stay the same in the output fiile while the data is split...but I will try and work this out

Thanks for all your help era and pointing me in the right direction
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies

2. Shell Programming and Scripting

Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?

Consolidate several lines of a CSV file with firewall rules Hi guys. I have a CSV file, which I created using an HTML export from a Check Point firewall policy. Each rule is represented as several lines, in some cases. That occurs when a rule has several address sources, destinations or... (4 Replies)
Discussion started by: starriol
4 Replies

3. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

4. Shell Programming and Scripting

Korn shell program to parse CSV text file and insert values into Oracle database

Enclosed is comma separated text file. I need to write a korn shell program that will parse the text file and insert the values into Oracle database. I need to write the korn shell program on Red Hat Enterprise Linux server. Oracle database is 10g. (15 Replies)
Discussion started by: shellguy
15 Replies

5. Shell Programming and Scripting

how to parse this file and obtain a .csv or .xls

Hello Expert, I have a file in the following format: SYNTAX_VERSION 5 MONITOR "NAME_TEMPLATES" DESCRIPTION "Monitors for contents of error " INTERVAL "1m" MONPROG "script.sh NAME_TEMPLATES" MAXTHRESHOLD GEN_BELOW_RESET SEVERITY Major ... (17 Replies)
Discussion started by: Ant-one
17 Replies

6. Shell Programming and Scripting

How to read and parse the content of csv file containing # as delimeter into fields using Bash?

#!/bin/bash i=0 cat 1.csv | while read fileline do echo "$fileline" IFS="#" flds=( $fileline ) nrofflds=${#flds} echo "noof fields$nrofflds" fld=0 while do echo "noof counter$fld" echo "$nrofflds" #fld1="${flds}" trying to store the content of line to fields but i... (4 Replies)
Discussion started by: barani75
4 Replies

7. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double... (7 Replies)
Discussion started by: vfrg
7 Replies

8. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

9. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

10. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using... (2 Replies)
Discussion started by: cdesiks
2 Replies
Login or Register to Ask a Question