Sponsored Content
Top Forums Shell Programming and Scripting CSV with commas in field values, remove duplicates, cut columns Post 302580223 by balajesuri on Thursday 8th of December 2011 01:25:21 AM
Old 12-08-2011
The below program will print 4 fields out of 6. Its may not be efficient as it goes through a lot of loops. Or if you have Perl's Text::CSV module installed, you may try that too.

input.txt:
Code:
test0,"hello,world,earth,universe","asia",antartica,"arctic",pacific,
test1,"abc,pqr,xyz,123","sed,awk,grep",perl,c,java
test2,"hel,wor,ear,uni","asi",ant,"arc",pac
test3,"start,middle,end",blue,green,red,white,

Code:
#! /usr/bin/perl -w
use strict;
open INPUT, "< input.txt";
for (<INPUT>) {
    my @y = split //,$_;
    my ($flag, $ln, $count) = (0, '', 0);
    for (@y) {
        if ($count == 4) {
            print "$ln\n";
            last;
        }
        if ($_ eq '"' && $flag == 0) {
            $flag = 1;
            $ln .= $_;
        }
        elsif ($_ eq '"' && $flag == 1) {
            $flag = 0;
            $ln .= $_;
        }
        elsif ($_ eq ',' && $flag == 0) {
            $count++;
            $ln .= $_;
        }
        else {
            $ln .= $_;
        }
    }
}
close INPUT;

output:
Code:
test0,"hello,world,earth,universe","asia",antartica,
test1,"abc,pqr,xyz,123","sed,awk,grep",perl,
test2,"hel,wor,ear,uni","asi",ant,
test3,"start,middle,end",blue,green,

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate commas after exporting excel file to csv

Hello everyone I'm new here and this is my first post so first of all I want to say that this is a great forum and I have managed to found most of my answers in these forums : ) So with that I ask you my first question: I have an excel file which I saved as a csv. However the excel file... (3 Replies)
Discussion started by: Spunkerspawn
3 Replies

2. Shell Programming and Scripting

shell script to remove extra commas from CSV outp file

Name,,,,,,,,,,,,,,,,,,,,Domain,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Contact,Phone,Email,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Location -----------------------,------------------------------------------------,-------,-----,---------------------------------,------------------------------------ ----... (1 Reply)
Discussion started by: sreenath1037
1 Replies

3. Shell Programming and Scripting

finding duplicates in csv based on key columns

Hi team, I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record. can one help me on finding the duplicates, Thanks in advance. ... (2 Replies)
Discussion started by: baskivs
2 Replies

4. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

5. Linux

How do I format a Date field of a .CSV file with multiple commas in a string field?

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 "12","B000QRIGJ4","4432","string with quotes, and with a comma, and colon: in... (3 Replies)
Discussion started by: dhruuv369
3 Replies

6. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

7. Shell Programming and Scripting

Shell script that should remove unnecessary commas between double quotes in CSV file

i have data as below 123,"paul phiri",paul@yahoo.com,"po.box 23, BT","Eco Bank,Blantyre,Malawi" i need an output to be 123,"paul phiri",paul@yahoo.com,"po.box 23 BT","Eco Bank Blantyre Malawi" (5 Replies)
Discussion started by: mathias23
5 Replies

8. Shell Programming and Scripting

Match columns from two csv files and update field in one of the csv file

Hi, I have a file of csv data, which looks like this: file1: 1AA,LGV_PONCEY_LES_ATHEE,1,\N,1,00020460E1,0,\N,\N,\N,\N,2,00.22335321,0.00466628 2BB,LES_POUGES_ASF,\N,200,200,00006298G1,0,\N,\N,\N,\N,1,00.30887539,0.00050312... (10 Replies)
Discussion started by: djoseph
10 Replies

9. Shell Programming and Scripting

Remove quotes and commas from field

In the attached file I am trying to remove all the "" and , (quotes and commas) from $2 and $3 and the "" (quotes) from $4. I tried the below as a start: awk -F"|" '{gsub(/\,/,X,$2)} 1' OFS="\t" enhancer.txt > comma.txt Thank you :). (6 Replies)
Discussion started by: cmccabe
6 Replies

10. Shell Programming and Scripting

How to remove unwanted commas from a .csv file?

how to remove unwanted commas from a .csv file Input file format "Server1","server-PRI-Windows","PRI-VC01","Microsoft Windows Server 2012, (64-bit)","Powered On","1,696.12","server-GEN-SFCHT2-VMS-R013,server-GEN-SFCHT2-VMS-R031,server-GEN-SFCHT2-VMS-R023"... (5 Replies)
Discussion started by: ranjancom2000
5 Replies
CGI::FormBuilder::Source::YAML(3pm)			User Contributed Perl Documentation		       CGI::FormBuilder::Source::YAML(3pm)

NAME
CGI::FormBuilder::Source::YAML - Initialize FormBuilder from YAML file SYNOPSIS
use CGI::FormBuilder; my $form = CGI::FormBuilder->new( source => { source => 'form.fb', type => 'YAML', }, ); my $lname = $form->field('lname'); # like normal DESCRIPTION
This reads a YAML (YAML::Syck) file that contains FormBuilder config options and returns a hash to be fed to CGI::FormBuilder->new(). Instead of the syntax read by CGI::FormBuilder::Source::File, it uses YAML syntax as read by YAML::Syck. That means you fully specify the entire data structure. LoadCode is enabled, so you can use YAML syntax for defining subroutines. This is convenient if you have a function that generates valida- tion subrefs, for example, I have one that can check profanity using Regexp::Common. validate: myfield: javascript: /^[sS]{2,50}$/ perl: !!perl/code: >- { My::Funk::fb_perl_validate({ min => 2, max => 50, profanity => 'check' })->(shift); } POST PROCESSING
There are two exceptions to "pure YAML syntax" where this module does some post-processing of the result. REFERENCES (ala CGI::FormBuilder::Source::File) You can specify references as string values that start with &, $, @, or \% in the same way you can with CGI::FormBuilder::Source::File. If you have a full direct package reference, it will look there, otherwise it will traverse up the caller stack and take the first it finds. For example, say your code serves multiple sites, and a menu gets different options depending on the server name requested: # in My::Funk: our $food_options = { www.meats.com => [qw( beef chicken horta fish )], www.veggies.com => [qw( carrot apple quorn radish )], }; # in source file: options: @{ $My::Funk::food_options->{ $ENV{SERVER_NAME} } } EVAL STRINGS You can specify an eval statement. You could achieve the same example a different way: options: eval { $My::Funk::food_options->{ $ENV{SERVER_NAME} }; } The cost either way is about the same -- the string is eval'd. EXAMPLE
method: GET header: 0 title: test name: test action: /test submit: test it linebreaks: 1 required: - test1 - test2 fields: - test1 - test2 - test3 - test4 fieldopts: test1: type: text size: 10 maxlength: 32 test2: type: text size: 10 maxlength: 32 test3: type: radio options: - - 1 - Yes - - 0 - No test4: options: @test4opts sort: &Someother::Package::sortopts validate: test1: /^w{3,10}$/ test2: javascript: EMAIL perl: eq 'test@test.foo' test3: - 0 - 1 test4: @test4opts You get the idea. A bit more whitespace, but it works in a standardized way. METHODS
new() Normally not used directly; it is called from CGI::FormBuilder. Creates the "CGI::FormBuilder::Source::YAML" object. Arguments from the 'source' hash passed to CGI::FormBuilder->new() will become defaults, unless specified in the file. parse($source) Normally not used directly; it is called from CGI::FormBuilder. Parses the specified source file. No fancy params -- just a single file- name is accepted. If the file isn't acceptable to YAML::Syck, I suppose it will die. SEE ALSO
CGI::FormBuilder, CGI::FormBuilder::Source AUTHOR
Copyright (c) 2006 Mark Hedges <hedges@ucsd.edu>. All rights reserved. LICENSE
This module is free software; you may copy it under terms of the Perl license (GNU General Public License or Artistic License.) http://www.opensource.org/licenses/index.html perl v5.8.8 2007-12-09 CGI::FormBuilder::Source::YAML(3pm)
All times are GMT -4. The time now is 09:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy