Check for null values in a columns. I have dozen of CSV files in a directory.


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Check for null values in a columns. I have dozen of CSV files in a directory.
# 1  
Old 12-07-2017
Hammer & Screwdriver Check for null values in a columns. I have dozen of CSV files in a directory.

Hi Folks,

I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email.

I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and that the file contains 6 fields no more no less than 6, if all of this conditions are true nothing happens but if any of these conditions failed, re-name the file to .bad. Here is what i have done so far.

Code:
for f in *.csv;
do 
  awk -F, '!(NF==6 && $1+0==$1 && $6 == ""){f=1; exit} END{exit f}' "$f" || mv "$f" "$f".bad;
done


Moderator's Comments:
Mod Comment
Please wrap all code, data, files input & output/errors in CODE tags.
It makes them easier to read and preserves spacing for indentation and fixed-width data.

Last edited by rbatte1; 12-07-2017 at 12:39 PM.. Reason: Added CODE tags
# 2  
Old 12-07-2017
So - ?
# 3  
Old 12-07-2017
i am not sure what you mean by ?. in any case never mind.
# 4  
Old 12-07-2017
What ? means is:
Is what you have so far working as you expect it to work? If not, in what way is it failing to work?
  • Are you getting diagnostic messages? (What are they?)
  • Are files being renamed that should not be renamed? (Under what conditions?)
  • Are files not being renamed that should be renamed? (Under what conditions?)
And, if so, where are you stuck in adding code for the remaining checks?
This User Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Multiple columns replace with null values.

I am trying to replace the partcular columns(Col3,col5,col20,col44,col55,co56,col59,col60,col61,col62,col74,col75,col88,col90,col91,col93,col94,col95) with empty Input file Col1,col2,col3,col4,col5------,col100 1,2,3,4,5,---------,100 3,4,5,6,7,---------,300 Output : ... (3 Replies)
Discussion started by: onesuri
3 Replies

2. Shell Programming and Scripting

Replace null values in csv with zero

Hi, i have another issue: i have three files: FILE 1 ServiceEventHandler, Processed,Percentage 5285337,100% FILE 2 Wallet, Processed,Percentage 5285337,100% (1 Reply)
Discussion started by: reignangel2003
1 Replies

3. UNIX for Dummies Questions & Answers

Find Null values in Columns and fail execution by displaying error message

Hi All, I am new to shell scripting. I have a requirement as part of my job to find out null/empty values in column 2 and column 3 from a CSV file and exit the further execution of script by displaying a simple error message. I have developed a script to do this by reading various articles... (7 Replies)
Discussion started by: tpk
7 Replies

4. Shell Programming and Scripting

Match columns from two csv files and update field in one of the csv file

Hi, I have a file of csv data, which looks like this: file1: 1AA,LGV_PONCEY_LES_ATHEE,1,\N,1,00020460E1,0,\N,\N,\N,\N,2,00.22335321,0.00466628 2BB,LES_POUGES_ASF,\N,200,200,00006298G1,0,\N,\N,\N,\N,1,00.30887539,0.00050312... (10 Replies)
Discussion started by: djoseph
10 Replies

5. Shell Programming and Scripting

Script to check for null values in a column

Hi Guys, I am new to shell script.I need your help to write a shell script. I have a csv file which has 13 columns separated by , I have written below script to fetch required 5 columns. awk -F, '(NR==1){h3=$3;h4=$4;h8=$8;h9=$9;h13=$13;next}(NF>1) \ {print... (5 Replies)
Discussion started by: Vivekit82
5 Replies

6. Shell Programming and Scripting

Check null values column

hi, I had a small question.I had a file from which i need to extract data. I have written the below script to check if the file exists and if it exists extract requierd columns from the file. IFILE=/home/home01/Report_1.csv OFILE=/home/home01/name.csv.out1 if #Checks if file exists... (1 Reply)
Discussion started by: Vivekit82
1 Replies

7. Shell Programming and Scripting

Check for null values in columns

Hi , I have below data with fixed with of 52 bytes having three columns value data. 01930 MA GLOUCESTER 02033 02025 COHASSET 01960 MA ... (3 Replies)
Discussion started by: sonu_pal
3 Replies

8. Shell Programming and Scripting

csv 4 columns values comparison!

Hi all, i have a csv file which as the following data: 294;F03;2000;40441 294;F03;2000;40443 284;F01;5400;44051 284;F01;5700;45666 the file holds 11689 lines. I was trying to get a script running to output results from this file that for each line with the condition: if a line is found... (9 Replies)
Discussion started by: stryng
9 Replies

9. UNIX for Dummies Questions & Answers

Check for null values in a column

Hi All, I have a file with 10 columns and get the required data for nine columns properly except 8th. In 8th column i have both NULL and NON NULL values...i.e certain records have values for all the columns including 8th column and certain records have 8th column as NULL.My requisite is,without... (20 Replies)
Discussion started by: ganesh_248
20 Replies

10. Shell Programming and Scripting

How to check Null values in a file column by column if columns are Not NULLs

Hi All, I have a table with 10 columns. Some columns(2nd,4th,5th,7th,8th and 10th) are Not Null columns. I'll get a tab-delimited file and want to check col by col and generate seperate error code for each col eg:102 if 2nd col value is NULL and 104 if 4th col value is NULL so on... I am a... (7 Replies)
Discussion started by: Mandab
7 Replies
Login or Register to Ask a Question