Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Filter using awk in CSV files
Top Forums Shell Programming and Scripting Filter using awk in CSV files Post 303008016 by pradyumnajpn10 on Saturday 25th of November 2017 10:59:39 AM
Old 11-25-2017
parse all csv files in directory and process it for Java command
Hi Rudic,

I know this must be silly for you but somewhere I might've over-explained or misguided you. Sorry about that.
Now I tried with all your suggestion and Step 1 and Step 2 and overcame it. Of-course your expertise one liner which I love is most welcome
Code:
#!/bin/bash

#Step 1
awk -F, '{gsub(/^"|"" "$/, "", $16); print $16}' AWS.CSV > out

#Step 2

while IFS= read -r line; do
  #Process each line 3 times with xargs(first try failed)
  # seq 3 | xargs -I{} $line

  for i in `seq 1 3`;
   do
    #Record the start time
    if [ $i -eq 1 ]; then
	start_time=$(date);# need to change later to get certain date format
    fi
    #Record the end time
    if [ $i -eq 3 ]; then
        end_time=$(date); 
    fi
    
    $line
   done < out

#Step 3
#Process all the CSV files starting with exec=1 run and run those files with java command
find . -maxdepth 1 -type f -name "exec=1*.csv" -exec 'java -classpath ./:${CLASSPATH} execGetResult {}' \;

Somehow managed to do step 3.But not sure about the syntax.
Step 3:
Find all the CSV files which is starting with exec=1*.csv and execute java command for each file found like below
Ex:
Code:
java -classpath ./:${CLASSPATH}  execGetResult exec=1_one.csv
java -classpath ./:${CLASSPATH}  execGetResult exec=1_two.csv

I am doing something like below to search the csv files in current diretory but struggling with syntax to embed java command for each csv files
Code:
find . -maxdepth 1 -type f -name "exec=1*.csv" -exec 'java -classpath ./:${CLASSPATH} execGetResult {}' \;

Rgds,
PD
Last edited by pradyumnajpn10; 11-25-2017 at 01:38 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

csv files (with quoted commas) and awk

I have a file as follows: 1,"This is field 2",3,4,5 2,"This is field 2 it can contain one , comma",3,4,5 3,"This is field 2 it also, can, contain, more",3,4,5 4,"This is field 2 without extra commas",3,4,5 and i pass this through to awk: awk -F, ' { if (... (3 Replies)
Discussion started by: Cranie
3 Replies

2. Shell Programming and Scripting

validation of data using filter (awk or other that works...) in csv files

Hello People I have the following file.csv: date,string,float,number,boolean 20080303,abc,1.5,123,Y 20080304,abc,1.2,345,N 20080229,nvh,1.4,098,Y 20080319,ugy,1.9,586,N 20080315,gyh,2.4,345,Y 20080316,erf,3.1,932,N I need to filter the date field where I have a data bigger than I... (1 Reply)
Discussion started by: Rafael.Buria
1 Replies

3. Shell Programming and Scripting

how to give multiple csv files as input in awk

Hi All, I am new to shell scripting..My problem is i want to give multiple csv files as input to awk script and process the data into one file.. My input file is File1 File2 File3 Product Location Period SalesPrice A x 8/11/2010 ... (7 Replies)
Discussion started by: kvth
7 Replies

4. Shell Programming and Scripting

Merge 2 csv files with awk

I have 2 files pipe delimted and want to merge them based on a key e.g file 1 123$aaa$yyy$zzz 345$xab$yzy$zyz 456$sss$ttt$foo 799$aaa$ggg$dee file 2 123$hhh 345$ddd 456$xxx 888$zzz so if the key is the first field, and the result should be the common key between file 1 and 2 (6 Replies)
Discussion started by: loloAix
6 Replies

5. Shell Programming and Scripting

Using AWK to match CSV files with duplicate patterns

Dear awk users, I am trying to use awk to match records across two moderately large CSV files. File1 is a pattern file with 173,200 lines, many of which are repeated. The order in which these lines are displayed is important, and I would like to preserve it. File2 is a data file with 456,000... (3 Replies)
Discussion started by: isuewing
3 Replies

6. Shell Programming and Scripting

awk filter by columns of file csv

Hi, I would like extract some lines from file csv using awk , below the example: I have the file test.csv with in content below. FLUSSO;COD;DATA_LAV;ESITO ULL;78;17/09/2013;OL ULL;45;05/09/2013;Apertura NP;45;13/09/2013;Riallineamento ULLNP;78;17/09/2013;OL NPG;14;12/09/2013;AperturaTK... (6 Replies)
Discussion started by: giankan
6 Replies

7. UNIX for Beginners Questions & Answers

awk assistance - Comparing 2 csv files

Hello all, I have searched high and low for a solution to this, many have come really close but not quite what I'm after. I have 2 files. One contains GUID's, for example: 8121E002-96FE-4C9C-BC5A-6AFF20DACECD 84468F30-F3B7-418B-81F0-0908E80792BF A second file, contains a path to the... (8 Replies)
Discussion started by: tirmUK
8 Replies

8. Shell Programming and Scripting

Modify csv-files with awk

Hello everyone! I have thousands of csv files I have to import into a Database table. As usually the files aren't perfect. For example they have a different number of columns and some weird columns. The second problem is, that I have to add 3 parts of the filename into 3 rows in the... (6 Replies)
Discussion started by: elRonaldo
6 Replies

9. UNIX for Beginners Questions & Answers

awk on csv files

wk on csv files Hi I have two csv files 1 ) keys.csv with data as below XX1,offsite XX2,offsite XX8,none XX3,offsite XX4,onsite XX7,none XX5,onsite XX6,onsite XX15,none 2) source.csv file with data like 1,0123,ppp,XX1 2,0122,sss,XX2 3,1239,yyy,XX8 4,567,kkk,XX5 (2 Replies)
Discussion started by: zozoo
2 Replies

10. Shell Programming and Scripting

Can we filter the below log data into CSV format?

HI , I m looking for help here!!! Can we filter the below log data into CSV format ? timestamp INFO <text > - Some text Drive .. Need a format of separate field such as 1 2 3 4 ... (2 Replies)
Discussion started by: MohSalNiz
2 Replies

LEARN ABOUT DEBIAN

jh_manifest

JH_MANIFEST(1)							    Javahelper							    JH_MANIFEST(1)

NAME

       jh_manifest - Adds or/and modifies manifests for jars

SYNOPSIS

       jh_manifest  [debhelperoptions]	[options]  jar1 ... jarN

       jh_manifest  [debhelperoptions]	[options]

DESCRIPTION

       Javahelper tool to add or update manifests in a jar file. It can be used in two modes. If passed jar files, it will only process these jar
       files. Otherwise it will update all jar files in the packages it acts on.

       When processing a package, the debhelper(7) exclude option will make jh_manifest ignore matching jar files.

FILES

       debian/package.manifest (or debian/manifest)
	   This file consist of a list of jar files and values to add to their manifests. Values in this file will take precedence over values in
	   the original manifest (and command line arguments over values in this file).

	   It is allowed to list a link in this file instead of an actual jar file, provided that the link can be resolved when jh_manifest
	   processes it.

	   If a jar file is listed here cannot be found, jh_manifest will print a warning, unless the jar file has been excluded.

	   As of javahelper >= 0.32, you may add comments in this file. If the line starts with a "#" it is completely ignored.

	   This file is ignored if jh_manifest is passed jar files via command line.

OPTIONS

       -c classpath, --classpath=classpath
	   Sets the Class-Path attribute of all processed jar files to classpath.

	   If not passed, then the CLASSPATH environment variable will be used in the given jar file do not have a Class-Path attribute.

       -m class, --main=class
	   Sets the Main-Class attribute to class in all processed jar files.

       -o options, --javaopts=options
	   Sets the Debian-Java-Parameters to options in all processed jar files. This attribute is used by jarwrapper to start java with extra
	   options (e.g. to make more memory available).

       -j /path/to/java/home, --java-home=/path/to/java/home
	   Sets the Debian-Java-Home attribute to /path/to/java/home in all processed jars. This attribute is used by jarwrapper to determine
	   which JVM to use.

EXAMPLES

       An example debian/manifest file:

	# use the symlink so we do not have to update with the next upstream release.
	usr/share/java/my.jar:
	 Class-Path: dep1.jar dep2.jar
	 Main-Class: some.awesome.Class
	usr/share/java/dep2.jar:
	 Class-Path: dep1.jar

SEE ALSO

       debhelper(7)

       This program is a part of javahelper and uses debhelper as backend. There are also tutorials in /usr/share/doc/javahelper.

AUTHOR

       Niels Thykier <niels@thykier.net>

COPYRIGHT AND LICENSE

       Copyright 2010 by Niels Thykier

       This tool is free software; you may redistribute it and/or modify it under the terms of GNU GPL 2.

0.43								    2012-03-12							    JH_MANIFEST(1)
All times are GMT -4. The time now is 02:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy