Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Filter using awk in CSV files
Top Forums Shell Programming and Scripting Filter using awk in CSV files Post 303008027 by pradyumnajpn10 on Saturday 25th of November 2017 03:24:49 PM
Old 11-25-2017
Awk read from shell environment variable
Hi Scrutinizer,

Actually I want to have that sort from suggestions.liked it. I ve no control over the commands.It executes for 1 hr or so for one command.
As long as it serves the purpose and is out of danger I love to adopt any suggestion from unix.com as a newcomer to linux.

Issue:
In this case the problem is only processing the first line from AWS.CSV file and exits.
Actually It should process all the steps for a single line from AWS.CSV input file and
fetch next line from AWS.csv file and repeat all the 5 steps.
Which I solved with below code using while loop, Is there any risk in doing like this or any better idea please suggest.
Code:
#!/bin/bash

while read -r line
do

  #Step 1
  cmd = $(echo $line|awk -F, '{gsub(/^"|"" "$/, "", $16); print $16}')

  #Step 2
  start_time=$(date +"%Y-%m-%dT%T" -d '-1 minute')
  $cmd; $cmd; $cmd
  end_time=$(date +"%Y-%m-%dT%T" -d '+1 minute')

  #Step 3
  #Process all the CSV files starting with exec=1 run and run those files with java command
  find . -maxdepth 1 -type f -name "exec=1*.csv" -exec java -classpath ./:${CLASSPATH} execGetResult {} \;

  #Step 4
  ./AWS_get_statistics.sh -t Manager -s ${start_time} -e ${end_time}

  #Step 5
  cd ..
  tar cvf AWS.${date}.tar ./AWS

done < AWS.csv


Rgds,
PD
Last edited by pradyumnajpn10; 11-25-2017 at 06:31 PM.. Reason: read Awk read from shell environment variable
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

csv files (with quoted commas) and awk

I have a file as follows: 1,"This is field 2",3,4,5 2,"This is field 2 it can contain one , comma",3,4,5 3,"This is field 2 it also, can, contain, more",3,4,5 4,"This is field 2 without extra commas",3,4,5 and i pass this through to awk: awk -F, ' { if (... (3 Replies)
Discussion started by: Cranie
3 Replies

2. Shell Programming and Scripting

validation of data using filter (awk or other that works...) in csv files

Hello People I have the following file.csv: date,string,float,number,boolean 20080303,abc,1.5,123,Y 20080304,abc,1.2,345,N 20080229,nvh,1.4,098,Y 20080319,ugy,1.9,586,N 20080315,gyh,2.4,345,Y 20080316,erf,3.1,932,N I need to filter the date field where I have a data bigger than I... (1 Reply)
Discussion started by: Rafael.Buria
1 Replies

3. Shell Programming and Scripting

how to give multiple csv files as input in awk

Hi All, I am new to shell scripting..My problem is i want to give multiple csv files as input to awk script and process the data into one file.. My input file is File1 File2 File3 Product Location Period SalesPrice A x 8/11/2010 ... (7 Replies)
Discussion started by: kvth
7 Replies

4. Shell Programming and Scripting

Merge 2 csv files with awk

I have 2 files pipe delimted and want to merge them based on a key e.g file 1 123$aaa$yyy$zzz 345$xab$yzy$zyz 456$sss$ttt$foo 799$aaa$ggg$dee file 2 123$hhh 345$ddd 456$xxx 888$zzz so if the key is the first field, and the result should be the common key between file 1 and 2 (6 Replies)
Discussion started by: loloAix
6 Replies

5. Shell Programming and Scripting

Using AWK to match CSV files with duplicate patterns

Dear awk users, I am trying to use awk to match records across two moderately large CSV files. File1 is a pattern file with 173,200 lines, many of which are repeated. The order in which these lines are displayed is important, and I would like to preserve it. File2 is a data file with 456,000... (3 Replies)
Discussion started by: isuewing
3 Replies

6. Shell Programming and Scripting

awk filter by columns of file csv

Hi, I would like extract some lines from file csv using awk , below the example: I have the file test.csv with in content below. FLUSSO;COD;DATA_LAV;ESITO ULL;78;17/09/2013;OL ULL;45;05/09/2013;Apertura NP;45;13/09/2013;Riallineamento ULLNP;78;17/09/2013;OL NPG;14;12/09/2013;AperturaTK... (6 Replies)
Discussion started by: giankan
6 Replies

7. UNIX for Beginners Questions & Answers

awk assistance - Comparing 2 csv files

Hello all, I have searched high and low for a solution to this, many have come really close but not quite what I'm after. I have 2 files. One contains GUID's, for example: 8121E002-96FE-4C9C-BC5A-6AFF20DACECD 84468F30-F3B7-418B-81F0-0908E80792BF A second file, contains a path to the... (8 Replies)
Discussion started by: tirmUK
8 Replies

8. Shell Programming and Scripting

Modify csv-files with awk

Hello everyone! I have thousands of csv files I have to import into a Database table. As usually the files aren't perfect. For example they have a different number of columns and some weird columns. The second problem is, that I have to add 3 parts of the filename into 3 rows in the... (6 Replies)
Discussion started by: elRonaldo
6 Replies

9. UNIX for Beginners Questions & Answers

awk on csv files

wk on csv files Hi I have two csv files 1 ) keys.csv with data as below XX1,offsite XX2,offsite XX8,none XX3,offsite XX4,onsite XX7,none XX5,onsite XX6,onsite XX15,none 2) source.csv file with data like 1,0123,ppp,XX1 2,0122,sss,XX2 3,1239,yyy,XX8 4,567,kkk,XX5 (2 Replies)
Discussion started by: zozoo
2 Replies

10. Shell Programming and Scripting

Can we filter the below log data into CSV format?

HI , I m looking for help here!!! Can we filter the below log data into CSV format ? timestamp INFO <text > - Some text Drive .. Need a format of separate field such as 1 2 3 4 ... (2 Replies)
Discussion started by: MohSalNiz
2 Replies

LEARN ABOUT DEBIAN

mongoexport

MONGOEXPORT(1)							  Mongo Database						    MONGOEXPORT(1)

NAME

       mongoexport - the Mongo export tool

SYNOPSIS

       mongoexport [OPTIONS]

DESCRIPTION

       mongoexport  is a tool to export a MongoDB collection to either JSON or CSV. The query can be filtered or a list of fields to output can be
       given.

       If the output is CSV, the fields must be specified in order.

EXAMPLES

       mongoexport -d test -c test1 --csv -f name,num
	      export documents from test.test1 in CSV format

OPTIONS

       --help show usage information

       -h, --host HOST
	      server to connect to (default HOST=localhost)

       -d, --db DATABASE
	      database to use

       -c, --c COLLECTION
	      collection to use

       -q, --query QUERY
	      query filter

       -f, --fields FIELDS
	      comma-separated list of field names

       --csv  export to CSV instead of JSON

       -o, --out FILE
	      output file, if not specified, stdout is used

       --dbpath PATH
	      directly access mongod data files in this path, instead of connecting to a mongod instance

COPYRIGHT

       Copyright 2007-2009 10gen

SEE ALSO

       For more information, please refer to the MongoDB wiki, available at http://www.mongodb.org.

AUTHOR

       Kristina Chodorow

10gen								     June 2009							    MONGOEXPORT(1)
All times are GMT -4. The time now is 03:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy