Matching the header of a .CSV file with dynamic field names


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Matching the header of a .CSV file with dynamic field names
# 1  
Old 07-29-2014
Matching the header of a .CSV file with dynamic field names

I have a .CSV file (frequency - weekly) whose header contains the year-week value in two of the columns which keeps changing every week. For an instance please see below.
Code:
Column1,Column2,Column3,Column4,Column5,Column6,Column7,Column8,Column9,Column10,Column11,Column12,Column13,201420 Column14,201420 Column15

I have to check if the data is placed under the right column and then proceed loading that data into the datamart. I created a standard header to compare it with the header of this file, but due to the dynamic nature of the field 14 and 15, I am not able to match the header every week. There is no date in the field name and I cannot use sysdate. Could you please suggest me how can I achieve this either using cat/awk/sed or any wildcards? Also let me know if you need more information.
# 2  
Old 07-29-2014
Try
Code:
awk 'NR==1 {getline X < "stdhead"; split (X, ST); for (i=1; i<=NF; i++) if (ST[i] != $i) print i " - " ST[i] " - " $i}' FS="," file.csv
14 - Column14 - 201420 Column14
15 - Column15 - 201420 Column15

# 3  
Old 07-29-2014
Appreciate your response RudiC. Your code prints the columns with their columns numbers as shown below.
Code:
1 -  - Column1
2 -  - Column2
3 -  - Column3
4 -  - Column4
5 -  - Column5
6 -  - Column6
7 -  - Column7
8 -  - Column8
9 -  - Column9
10 -  - Column10
11 -  - Column11
12 -  - Column12
13 -  - Column13
14 -  - 201420 Column14
15 -  - 201420 Column15

For this, what should be the standard header and how do I compare the header of my file file with that of the standard header?

Thank you in advance...
# 4  
Old 07-29-2014
Replace the "stdhead" after the getline with your actual standard header file name.
# 5  
Old 07-29-2014
That was one of my concerns RudiC. I am unable to decide what the standard header should be as the header of the file keeps changing every week. This check should be part of my shell script.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Check for empty line at end of a dynamic header in each file

Hi Folks, I have a requirement to develop a shell script. PFB my requirement, Requirement: I need to check an empty line after the end of each header in respective file and if a empty line is present simply echo file OK and if empty line is not present echo "Adding empty line" and add an... (6 Replies)
Discussion started by: tpk
6 Replies

2. Shell Programming and Scripting

Match columns from two csv files and update field in one of the csv file

Hi, I have a file of csv data, which looks like this: file1: 1AA,LGV_PONCEY_LES_ATHEE,1,\N,1,00020460E1,0,\N,\N,\N,\N,2,00.22335321,0.00466628 2BB,LES_POUGES_ASF,\N,200,200,00006298G1,0,\N,\N,\N,\N,1,00.30887539,0.00050312... (10 Replies)
Discussion started by: djoseph
10 Replies

3. Linux

How do I format a Date field of a .CSV file with multiple commas in a string field?

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 "12","B000QRIGJ4","4432","string with quotes, and with a comma, and colon: in... (3 Replies)
Discussion started by: dhruuv369
3 Replies

4. Shell Programming and Scripting

How to get sqlplus column header once in csv file?

Hi All, Could anyoone please let me know how do I get sqlplus column header once in csv file Scripts are below: cat concreq.sh #!/bin/bash . $HOME/.profile while ; do sqlplus apps/pwd <<-EOF set lines 100 pages 100 col "USER_CONCURRENT_QUEUE_NAME" format a40; --set termout off... (5 Replies)
Discussion started by: a1_win
5 Replies

5. Shell Programming and Scripting

Matching and Merging csv data fields based on a common field

Dear List, I have a file of csv data which has a different line per compliance check per host. I do not want any omissions from this csv data file which looks like this: date,hostname,status,color,check 02-03-2012,COMP1,FAIL,Yellow,auth_pass_change... (3 Replies)
Discussion started by: landossa
3 Replies

6. UNIX for Dummies Questions & Answers

Merge all csv files in one folder considering only 1 header row and ignoring header of all others

Friends, I need help with the following in UNIX. Merge all csv files in one folder considering only 1 header row and ignoring header of all other files. FYI - All files are in same format and contains same headers. Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies

7. Shell Programming and Scripting

Add header to a .csv file

Hi, I am trying to add a header record to all the .csv files in a directory. I am using the below sed commnad sed -i '1 i \abc,sam,xyz,tip,pep,rip' xyz.csv but this is not adding the header and I am not getting any error,pls tell me if any thing is wrong in the code. Thanks, Shruthi (2 Replies)
Discussion started by: shruthidwh
2 Replies

8. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

9. Shell Programming and Scripting

Append Header in CSV file

Hi, I create a csv file and the output looks like below Arun,E001 Sathish,E003 Now i need to include the below header and the output should like below Name,Number Arun,E001 Sathish,E003 Please guide me. Thanks (4 Replies)
Discussion started by: Sekar1
4 Replies

10. Shell Programming and Scripting

How can I add a header to a csv file

I have a csv file which has no header. the file has 15 fields and needs to go out with a header of 8 fields. The header content needs to have some variables and some fixed that i have set up: variable header fields OUTFILE_YEAR=`date '+%y'` DATE=`date '+%d%m%y'` TIME=`date '+%H:%M:%S'`... (6 Replies)
Discussion started by: Pablo_beezo
6 Replies
Login or Register to Ask a Question