Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Need script to generate file. Post 302892632 by drl on Thursday 13th of March 2014 09:27:52 PM
Old 03-13-2014
Hi.

My goal in this was to allow the most straight-forward awk statement { print $1,$2,$5,$6 } to solve the problem Of course, as MadeInGermany observed, there are quoted strings in which commas are embedded.

So one thing we could do is to reformat the csv file so that the field separators are something other than commas. This can be done relatively easily with extant perl modules, and we just need to learn how to use them.

So here is a shell script that allows, at the heart, the simple awk print statement noted above:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate separator replacement with perl.

echo
what ~/bin/divepm
# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C divepm perl

pl " Modules and versions used by demo perl script p1:"
divepm -q -i=p1

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

pl " Expected output:"
cat expected_output.txt

pl " Results:"
./p1 ',|' $FILE |
tee f1 |
awk 'BEGIN { FS="|"; OFS="|" }{ print $1,$2,$5,$6 }' |
tee f2 |
./p1 '|,'

pl " Demo perl script:"
cat p1

exit 0

producing:
Code:
$ ./s1

divepm	Display version of perl modules.

Environment: LC_ALL = , LANG = en_US.UTF-8
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian 5.0.8 (lenny, workstation) 
bash GNU bash 3.2.39
divepm (local) 1.4
perl 5.10.0

-----
 Modules and versions used by demo perl script p1:
 1.06	Text::CSV
 1.08	Carp

-----
 Input data file data1:
1,APRIL,NEW,"New market,delhi,pune,India",RECENT, 254664
2,MARCH,OLD,"New Area,Mumbai,UP,India",CURRENT, 152483

-----
 Expected output:
1,APRIL,RECENT,254664
2,MARCH,CURRENT,152483

-----
 Results:
1,APRIL,RECENT,254664
2,MARCH,CURRENT,152483

-----
 Demo perl script:
#!/usr/bin/env perl

# @(#) p1	Demonstrate reconfigure separator for CSV file.
# $Id: p1,v 1.2 2014/03/13 21:47:41 drl Exp drl $

# No checking for errors in this demo version.
# Input separators from command-line,
# Initialize the CSV input and output objects,
# Loop until EOF:
# Read line,
# Parse line with first separator,
# Combine fields into line,
# Print line with new separator.

use Text::CSV;
use Carp;

$comma = ",";
$bell  = "";
$bell  = "|";

$t1   = shift;
$inps = substr( $t1, 0, 1 );
$outs = substr( $t1, 1, 1 );

$csv = Text::CSV->new( { sep_char => $inps, allow_whitespace => 1 } );
$out = Text::CSV->new( { sep_char => $outs } );

while (<>) {
  chomp;
  $line   = $_;
  $status = $csv->parse($line);
  if ( not $status ) {
    carp("problem with csv-parse at $.");
    print "Line $. -- $line";
  }
  @columns = $csv->fields();

  $status = $out->combine(@columns);
  if ( not $status ) {
    carp("problem with csv-combine at $.");
    print "Line $. -- $line";
  }
  $line = $out->string();
  print "$line\n";
}

Obviously there is little in the way of error checking, but the main body of code works. The intermediate files f1, f2 can be examined to see the results of the transformations.

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
 

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Modify script to generate a log file

I've seen several examples of scripts in thise forum about having a script generate a log file. I have a script that is run from cron and that monitors a file system for a specfic filename(s) and then performs some actions on them. Normally I call this script from another script (which the one... (2 Replies)
Discussion started by: heprox
2 Replies

2. Shell Programming and Scripting

To generate the FTP Script file

Hi, I am new to the shell programming., My requirement is , I have an shell file, which call internally the sql file, which generates 4 files on the directory., and then shell has to create the file which contains all the ftp commands to extract the files to different server for later... (1 Reply)
Discussion started by: konankir
1 Replies

3. Shell Programming and Scripting

how to generate html file using script?

Hi Friends I have an requirement that i need to generate html file using script. and the script output shold keep adding to that html file like tablewise. can anyone please help me out in this. thanks Krish. (2 Replies)
Discussion started by: kittusri9
2 Replies

4. Shell Programming and Scripting

KSH - help needed for creating a script to generate xml file from text file

Dear Members, I have a table in Oracle DB and one of its column name is INFO which has data in text format which we need to fetch in a script and create an xml file of a new table from the input. The contents of a single cell of INFO column is like: Area:app - aam Clean Up Criteria:... (0 Replies)
Discussion started by: Yoodit
0 Replies

5. Shell Programming and Scripting

Script to generate csv file

Hello; I need to generate a csv file that contains a list of all the files in a particular server (from the root directory ie: \) that have a permission stamp of 777. I would like to create the csv so that it contains the following: server name, file name, full path name where file exists,... (17 Replies)
Discussion started by: gvolpini
17 Replies

6. Shell Programming and Scripting

Needed script to FTP a File and generate a quality checksum file

hi all i want a script to FTP a file and should generate a quality checksum file means when I FTP a file from one server to another server it should generate a QC file which should contain timestamp,no.of records in that file Thanks in advance saikumar (3 Replies)
Discussion started by: hemanthsaikumar
3 Replies

7. Shell Programming and Scripting

Script to generate csv file

Dears, I am new in shell world and I need your help in this, I have to create a report based on the output file generated by another program. I want to write a shell script for this. The output file generated every 15 minutes but i can’t open it until the end of day so the script will get the... (3 Replies)
Discussion started by: abdul2020
3 Replies

8. Shell Programming and Scripting

Script to generate .csv file

Dears,I need your help in this, I have to create a report based on the output file generated by another program. I want to write a shell script for this. The output file generated every 15 minutes but i can’t open it until the end of day so the script will get the file as an input the file will be... (8 Replies)
Discussion started by: abdul2020
8 Replies
All times are GMT -4. The time now is 05:47 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy