Parse csv files by their names Post: 302861209

Sponsored Content

Top Forums Shell Programming and Scripting Parse csv files by their names Post 302861209 by Jhon.c on Tuesday 8th of October 2013 11:16:43 AM

10-08-2013

Registered User

Parse csv files by their names

HI all

I have multiple csv files with the names

VAR1_VAR2_VAR3_VAR4.csv

All the files have the same structure inside just values change.
I am trying to retrieve data from those files by fixing at each time one or more VAR.
I tried to write a script but I have 2 problems:

2- When I fix one variable (i.e, *_VAR2_VAR3_VAR4.csv ) on when VAR1 have 3 entries how to iterate this variable inside a loop ?

My script is here (My first tentative)

Code:

 #!/bin/bash          
      #OLDFIS=$IFS
      #IFS=","
      awk -F "," '
      #-------------------------- Table 1  ----------------------- -------#
 # Header begin    

 BEGIN {

      print"=========================================================="

      printf"%-8s %-8s %-8s %-8s %-8s\n","Network size", "P_Energy", "E_Energy",      
     "All_energy","Latency"

      print"=========================================================="
      }

 # Header end  


     NR==3 {printf "%-20s", $1 } NR==25 {printf "%-20s", $2 }   NR==12 {printf "%-20s", $2 }   
   NR==29 {printf "%-20s", $2 } NR==49 {printf "%-20s", $4 }' /...PATH/ *.csv >test.txt

      #-----------------------------------------------------------------------------#

Output:

Code:

======================================================================== 
    Network size       P_Energy           E_Energy           All_energy         Latency             
    =========================================================================
                    0.0402597           0.00767312          0.0479328           0.294311

With this code I have two problems:
1- Even I specify a large number of files I get only one row, should I add a loop?
2-I want to parse a set of files by fixing one variable (VAR1), and use its different value as the first column of my output.

Any ideas please?

Jhon.c

View Public Profile for Jhon.c

Find all posts by Jhon.c

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Hi I have a CSV file with me in this format Currency, USD, EUR, USD, 1.00, 1.32, EUR, 0.66, 1.00, How do I transpose the file to get to the format below. currency, currency, rate USD, USD, 1.00 USD, EUR, 1.32 EUR, USD, 0.66 EUR, EUR, 1.00 Thanks for your help We are using...

2. Shell Programming and Scripting

Re-usable function to parse csv files with different number of fields

Hi there, been pondering how to deal with this and hoping someone would give me an insight on this. I need help on creating a reusable bash funtion to parse csv files containing different number of fields (comma-seperated). My initial thought is to create function for each input csv file (20+...

3. Shell Programming and Scripting

CSV File parse help in Perl

Folks, I have a bit of an issue trying to obtain some data from a csv file using PERL. I can sort the file and remove any duplicates leaving only 4 or 5 rows containing data. My problem is that the data contained in the original file contains a lot more columns and when I try ro run this script...

4. Shell Programming and Scripting

Unix Script to parse a CSV

I am writing a unix script that will parse a CSV and edit the values. My CSV looks like this 0,0,0,0,1,0,1,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0 10,11,7,0,4,12,2,3,7,0,11,3,12,4,0,5,5,4,5,0,8,6,12,0,9,3,3,0,2,7,8...

5. Shell Programming and Scripting

substrings from all files incl subdirs into csv with dir names

Greetings! I have multiple files, one per subdirectory, all with the same file name. All subdirectories are one level deep from the main directory. The data in the files is tab delimited between fields and record delimited with a newline. The subdirectory names have the date in the...

6. Shell Programming and Scripting

Parse csv file

Hi, Our requirement is to parse the input file(.csv format). The each column in the file is delimited with comma. We need to take each column and apply some business validation rule. If data itself contains comma, then those fields are enclosed with double quotes ("). We can see this double...

7. Shell Programming and Scripting

How to parse csv format?

Hi, I have a file with 3 fields in csv format: /tmp/foo/,MODIFY,bar/toto "/tmp/foo, bar/","ATTRIB,ISDIR","toto, tata/foobar"I would like to split fields in order to obtain the following: Line1: /tmp/foo/ MODIFY bar/totoLine2: /tmp/foo, bar/ ATTRIB,ISDIR toto, tata/foobarCan't find my way...

8. Shell Programming and Scripting

parse file names

trying to parse out all parts of this file name. REC=`echo "CAMXI.F0150.txt" | sed 's/.*$*$.*/\1/'` export "FLRECL=$REC" FLECL=0150 I can get the numbers 0150 out of the file name. But need to capture first 5 bytes and extension. So i would export 3 variables (name, length, extension)...

9. Shell Programming and Scripting

I have strigns below, in which we have files names. Can we parse them in one algorithm? If not, the

I have several strings with files names roundrobin_report_report_ALL_dbagadm_${Today}-${TM}_U.csv I want to say That I found a way to have prefix roundrobin_report_report_ALL_dbagadmU. letter=`echo $var | cut -d'.' -f 1 | rev | cut -d'_' -f 1 | rev` echo "NAME = $letter" prefix=`echo $var...

10. Shell Programming and Scripting

Need a script to parse data and output to csv

I am not too savvy with arrays and am assuming that what I am looking for needs arrays. This is my requirement. So I have the raw data that gets updated to a log as shown below StudentInfo: FullInfo = { Address = Newark Age = 20 Name= John } StudentInfo:...

LEARN ABOUT CENTOS

processcsv.py

PROCESSCSV.PY(1)					      Virtualization Support						  PROCESSCSV.PY(1)

NAME

       processcsv.py - process virt-top CSV files

SUMMARY

	virt-top --csv data.csv
	processcsv.py < data.csv

DESCRIPTION

       virt-top is a top(1)-like utility for showing stats of virtualized domains.

       processcsv.py is a simple Python script that post-processes the output of "virt-top --csv".

       It is used like this:

	virt-top --csv data.csv
	processcsv.py < data.csv

       The second command will overwrite the following files in the current directory:

       "global.csv"
	   This contains the global (host) statistics columns from the CSV file.

       "domainNN.csv" (multiple files)
	   For each libvirt domain ID NN, a file is created containing the per-domain statistics from the CSV file.

SEE ALSO

       virt-top(1)

AUTHORS

       Richard W.M. Jones <rjones @ redhat . com>

COPYRIGHT

       (C) Copyright 2007-2012 Red Hat Inc., Richard W.M. Jones http://libvirt.org/

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
       Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

virt-top-1.0.8							    2013-12-29							  PROCESSCSV.PY(1)

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to Parse a CSV file into a Different Format

Discussion started by: cdesiks

2. Shell Programming and Scripting

Re-usable function to parse csv files with different number of fields

Discussion started by: jy2k7ca

3. Shell Programming and Scripting

CSV File parse help in Perl

Discussion started by: lodey

4. Shell Programming and Scripting

Unix Script to parse a CSV

Discussion started by: RJ17