Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: validation of data using filter (awk or other that works...) in csv files
Top Forums Shell Programming and Scripting validation of data using filter (awk or other that works...) in csv files Post 302173059 by Rafael.Buria on Wednesday 5th of March 2008 12:38:23 PM
Old 03-05-2008
validation of data using filter (awk or other that works...) in csv files
Hello People

I have the following file.csv:

date,string,float,number,boolean
20080303,abc,1.5,123,Y
20080304,abc,1.2,345,N
20080229,nvh,1.4,098,Y
20080319,ugy,1.9,586,N
20080315,gyh,2.4,345,Y
20080316,erf,3.1,932,N

I need to filter the date field where I have a data bigger than I pass (this date IŽll receive and store into a variable) reject the data and remove it from my file. So today is 20080305 then I receive this date and store in a variable. My file need to have only the rows with the dates 20080303, 20080304, 20080229 according to my file above.

IŽve tried to remove it using the command :
awk -F, 'BEGIN {OFS=","} { if ($1 = ${date}) print $0}' file.csv > ex.csv

However it looks like that the awk do not take variables into account.

Does anyone have any ideia?

Regards
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl - extracting data from .csv files

PROJECT: Extracting data from an employee timesheet. The timesheets are done in excel (for user ease) and then converted to .csv files that look like this (see color code key below): ,,,,,,,,,,,,,,,,,,, 9/14/2003,<-- Week Ending,,,,,,,,,,,,,,,,,, Craig Brennan,,,,,,,,,,,,,,,,,,,... (3 Replies)
Discussion started by: kregh99
3 Replies

2. UNIX for Advanced & Expert Users

awk data validation

Hi , This is a general doubt.... Is there any way to do data validation inside an awk script.. Let me make it more clear.... I have a string variable inside awk script .. Is there any way to check whether it is number or string etc... Thanks in advance. Shihab (1 Reply)
Discussion started by: shihabvk
1 Replies

3. Shell Programming and Scripting

CSV file parsing and validation

I have a CSV file that needs to through two seperate processes (in the end there will be 2 files (Dload.unl and Tload.unl and we'll say the input file name is mass.csv). I have a processfile() function that will call the process Dload funtion. In Dload I want to read mass.csv into Dload and then... (1 Reply)
Discussion started by: dolo21taf
1 Replies

4. UNIX for Advanced & Expert Users

Time validation in a csv file

Hi, Im having a hard time in creating a script with the following conditions below. I have a csv file generated which is updated every 5 mins and it contains a timestamp in it. for example: time data 00:00 1 00:05 0 00:10 6 00:15 3 however, there is a time that... (4 Replies)
Discussion started by: mdap
4 Replies

5. UNIX for Dummies Questions & Answers

Extracting data from PDF files into CSV file

Hi, I have several hundreds of PDFfiles number 01.pdf, 02.pdf, 03.pdf, etc in one folder. These are vey long documentd with a lot of information (text, tables, figures, etc). I need to extract the information asociated with one disease in particular (Varicella). The information I need to... (5 Replies)
Discussion started by: Xterra
5 Replies

6. Shell Programming and Scripting

awk : Filter a set of data to parse header line and last field of multiple same match.

Hi Experts, I have a data with multiple entry , I want to filter PKG= & the last column "00060110" or "00088150" in the output file: ############################################################################################### PKG= P8SDB :: VGS = vgP8SOra vgP8SDB1 vgP8S001... (5 Replies)
Discussion started by: rveri
5 Replies

7. Shell Programming and Scripting

awk filter by columns of file csv

Hi, I would like extract some lines from file csv using awk , below the example: I have the file test.csv with in content below. FLUSSO;COD;DATA_LAV;ESITO ULL;78;17/09/2013;OL ULL;45;05/09/2013;Apertura NP;45;13/09/2013;Riallineamento ULLNP;78;17/09/2013;OL NPG;14;12/09/2013;AperturaTK... (6 Replies)
Discussion started by: giankan
6 Replies

8. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

9. Shell Programming and Scripting

Filter using awk in CSV files

Hello Gentlemen, Finding difficulties to play with my Input files:confused: . Your guidance will certainly help as always. After converting to csv file from XLSM file, I am getting some extra ""(double quote) characters which I want to terminate inside shell script and process it further. ... (6 Replies)
Discussion started by: pradyumnajpn10
6 Replies

10. Shell Programming and Scripting

Can we filter the below log data into CSV format?

HI , I m looking for help here!!! Can we filter the below log data into CSV format ? timestamp INFO <text > - Some text Drive .. Need a format of separate field such as 1 2 3 4 ... (2 Replies)
Discussion started by: MohSalNiz
2 Replies

LEARN ABOUT DEBIAN

csv2mipe

CSV2MIPE(1)						User Contributed Perl Documentation					       CSV2MIPE(1)

NAME

       csv2mipe.pl - Generates MIPE file based on 3 tab-delimited files
	 based on MIPE version v1.1
	 arguments: * tab-delimited file with PCR-level data
		    * tab-delimited file with SNP-level data
		    * tab-delimited file with assay-level data

	 Columns in file with PCR-level data:
	   pcr_id
	   pcr_modified (might be multiple, divided by semi-colon ";")
	   pcr_project (might be multiple, divided by semi-colon ";")
	   pcr_researcher (might be multiple, divided by semi-colon ";")
	   pcr_species
	   source_type
	   source_id
	   design_seq
	   primer1_oligo
	   primer1_seq
	   primer1_tm
	   primer2_oligo
	   primer2_seq
	   primer2_tm
	   design_remark (might be multiple, divided by semi-colon ";")
	   use_seq
	   use_revcomp
	   use_remark (might be multiple, divided by semi-colon ";")
	   pcr_remark (might be multiple, divided by semi-colon ";")

	 Columns in file with SNP-level data:
	   pcr_id
	   snp_id
	   snp_pos
	   snp_amb
	   snp_remark (might be multiple, divided by semi-colon ";")

	 Columns in file with assay-level data:
	   pcr_id
	   snp_id
	   assay_id
	   assay_type
	   assay_enzyme
	   assay_oligo
	   assay_specific
	   assay_tail
	   assay_strand
	   assay_remark (might be multiple, divided by semi-colon ";")

SYNOPSIS

       csv2mipe.pl <pcr_file.csv> <snp_file.csv> <assay_file.csv>

ADDITIONAL INFO

       See http://mipe.sourceforge.net

AUTHOR

       Jan Aerts (jan.aerts@bbsrc.ac.uk)

perl v5.14.2							    2005-07-20							       CSV2MIPE(1)
All times are GMT -4. The time now is 12:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy