Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Filter using awk in CSV files
Top Forums Shell Programming and Scripting Filter using awk in CSV files Post 303008020 by Scrutinizer on Saturday 25th of November 2017 12:34:14 PM
Old 11-25-2017
For step 2, how about something like:
Code:
cmd=$(awk -F, '{gsub(/^"|"" "$/, "", $16); print $16}' AWS.CSV)
start_time=$(date -d '-1 minute')
$cmd; $cmd; $cmd
end_time=$(date -d '+1 minute')

Note 1: Running a command like that is dangerous, unless you have total control over the content of $cmd.

Note 2: The date command is for GNU date..
Last edited by Scrutinizer; 11-25-2017 at 01:42 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

csv files (with quoted commas) and awk

I have a file as follows: 1,"This is field 2",3,4,5 2,"This is field 2 it can contain one , comma",3,4,5 3,"This is field 2 it also, can, contain, more",3,4,5 4,"This is field 2 without extra commas",3,4,5 and i pass this through to awk: awk -F, ' { if (... (3 Replies)
Discussion started by: Cranie
3 Replies

2. Shell Programming and Scripting

validation of data using filter (awk or other that works...) in csv files

Hello People I have the following file.csv: date,string,float,number,boolean 20080303,abc,1.5,123,Y 20080304,abc,1.2,345,N 20080229,nvh,1.4,098,Y 20080319,ugy,1.9,586,N 20080315,gyh,2.4,345,Y 20080316,erf,3.1,932,N I need to filter the date field where I have a data bigger than I... (1 Reply)
Discussion started by: Rafael.Buria
1 Replies

3. Shell Programming and Scripting

how to give multiple csv files as input in awk

Hi All, I am new to shell scripting..My problem is i want to give multiple csv files as input to awk script and process the data into one file.. My input file is File1 File2 File3 Product Location Period SalesPrice A x 8/11/2010 ... (7 Replies)
Discussion started by: kvth
7 Replies

4. Shell Programming and Scripting

Merge 2 csv files with awk

I have 2 files pipe delimted and want to merge them based on a key e.g file 1 123$aaa$yyy$zzz 345$xab$yzy$zyz 456$sss$ttt$foo 799$aaa$ggg$dee file 2 123$hhh 345$ddd 456$xxx 888$zzz so if the key is the first field, and the result should be the common key between file 1 and 2 (6 Replies)
Discussion started by: loloAix
6 Replies

5. Shell Programming and Scripting

Using AWK to match CSV files with duplicate patterns

Dear awk users, I am trying to use awk to match records across two moderately large CSV files. File1 is a pattern file with 173,200 lines, many of which are repeated. The order in which these lines are displayed is important, and I would like to preserve it. File2 is a data file with 456,000... (3 Replies)
Discussion started by: isuewing
3 Replies

6. Shell Programming and Scripting

awk filter by columns of file csv

Hi, I would like extract some lines from file csv using awk , below the example: I have the file test.csv with in content below. FLUSSO;COD;DATA_LAV;ESITO ULL;78;17/09/2013;OL ULL;45;05/09/2013;Apertura NP;45;13/09/2013;Riallineamento ULLNP;78;17/09/2013;OL NPG;14;12/09/2013;AperturaTK... (6 Replies)
Discussion started by: giankan
6 Replies

7. UNIX for Beginners Questions & Answers

awk assistance - Comparing 2 csv files

Hello all, I have searched high and low for a solution to this, many have come really close but not quite what I'm after. I have 2 files. One contains GUID's, for example: 8121E002-96FE-4C9C-BC5A-6AFF20DACECD 84468F30-F3B7-418B-81F0-0908E80792BF A second file, contains a path to the... (8 Replies)
Discussion started by: tirmUK
8 Replies

8. Shell Programming and Scripting

Modify csv-files with awk

Hello everyone! I have thousands of csv files I have to import into a Database table. As usually the files aren't perfect. For example they have a different number of columns and some weird columns. The second problem is, that I have to add 3 parts of the filename into 3 rows in the... (6 Replies)
Discussion started by: elRonaldo
6 Replies

9. UNIX for Beginners Questions & Answers

awk on csv files

wk on csv files Hi I have two csv files 1 ) keys.csv with data as below XX1,offsite XX2,offsite XX8,none XX3,offsite XX4,onsite XX7,none XX5,onsite XX6,onsite XX15,none 2) source.csv file with data like 1,0123,ppp,XX1 2,0122,sss,XX2 3,1239,yyy,XX8 4,567,kkk,XX5 (2 Replies)
Discussion started by: zozoo
2 Replies

10. Shell Programming and Scripting

Can we filter the below log data into CSV format?

HI , I m looking for help here!!! Can we filter the below log data into CSV format ? timestamp INFO <text > - Some text Drive .. Need a format of separate field such as 1 2 3 4 ... (2 Replies)
Discussion started by: MohSalNiz
2 Replies

LEARN ABOUT CENTOS

trace-cmd-extract

TRACE-CMD-EXTRACT(1)													      TRACE-CMD-EXTRACT(1)

NAME

       trace-cmd-extract - extract out the data from the Ftrace Linux tracer.

SYNOPSIS

       trace-cmd extract [OPTIONS]

DESCRIPTION

       The trace-cmd(1) extract is usually used after trace-cmd-start(1) and trace-cmd-stop(1). It can be used after the Ftrace tracer has been
       started manually through the Ftrace pseudo file system.

       The extract command creates a trace.dat file that can be used by trace-cmd-report(1) to read from. It reads the kernel internal ring buffer
       to produce the trace.dat file.

OPTIONS

       -p plugin
	   Although extract does not start any traces, some of the plugins require just reading the output in ASCII format. These are the latency
	   tracers, since the latency tracers have a separate internal buffer. The plugin option is therefore only necessary for the wakeup,
	   wakeup-rt, irqsoff, preemptoff and preemptirqsoff plugins.

	       With out this option, the extract command will extract from the internal
	       Ftrace buffers.

       -O option
	   If a latency tracer is being extracted, and the -p option is used, then there are some Ftrace options that can change the format. This
	   will update those options before extracting. To see the list of options see trace-cmd-list. To enable an option, write its name, to
	   disable the option append the characters no to it. For example: noprint-parent will disable the print-parent option that prints the
	   parent function in printing a function event.

       -o outputfile
	   By default, the extract command will create a trace.dat file. This option will change where the file is written to.

SEE ALSO

       trace-cmd(1), trace-cmd-record(1), trace-cmd-report(1), trace-cmd-start(1), trace-cmd-stop(1), trace-cmd-reset(1), trace-cmd-split(1),
       trace-cmd-list(1), trace-cmd-listen(1)

AUTHOR

       Written by Steven Rostedt, <rostedt@goodmis.org[1]>

RESOURCES

       git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

COPYING

       Copyright (C) 2010 Red Hat, Inc. Free use of this software is granted under the terms of the GNU Public License (GPL).

NOTES

	1. rostedt@goodmis.org
	   mailto:rostedt@goodmis.org

								    06/11/2014						      TRACE-CMD-EXTRACT(1)
All times are GMT -4. The time now is 04:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy