Search Results

Since both are flat files with data patterns, it is possible with awk

It wouldn't degrade the performance...but performing an operation on 100 GB file at a time vs performing the task on 1GB files parallelly...
That's the beauty of Hadoop
But upto you to choose where...

If you are dumping the data into HDFS, I assume the data is very big.
In that case you can run the awk code as streams and it would run faster
Anyways I shall provide the code in some time as I...

I don't see any issue when I tested the above given code
I would suggest you check the input files correctly and retry

awk -F "," 'NR == FNR {h = (h == "") ? $1 : (h FS $1); gsub("[)(]", "-", $2); split($2, a, "-"); d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}
FNR == 1 {print h >...

These 2 lines are the login part...
1st line is assignment part for schema file
2nd line is the validation part from schema

You had issues splitting the data type field in schema file using the delimited '(' or ')' (issue depends on unix flavor).
Hence, I have converted the braces to '-'
now, your schema file datatype...

I ran the same code and below is the output
$ cat conf
id,Numeric(2),NOT NULL
name,String(20)
state,String(10),NOT NULL
street_No,Numeric(4)
$ cat data
abc,john,MI,201
22,Lilly,CA,405...

awk -F "," 'NR == FNR {gsub("[)(]", "-", $2); split($2, a, "-"); d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}
{for(i = 1; i <= NF; i++)
{if(((d[i] == "Numeric" && (($i +...

Verified this code and it provided the expected output for the given input
awk -F "," 'NR == FNR{split($2, a, "\)|\("); d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}
...

post your questions here

Are you sure cf and df has the same content as provided in the previous post

Hey Shree, you cant keep changing the requirement.
In your first code, there was a header and hence written the code to meet that requirement

With the data you gave recently, I ran the below...

The above given code is updated for Numeric and String.
Tell me if that doesn't work

In the first example, you have given, "integer" and "string"
The code is written in such a way to cehck integer and string but not "Numeric" and "String". Even case is important.
you can change the...

In the below code, conf.csv is the name of schema file and file.csv is the name of data file
awk -F "," 'NR == FNR {a[NR] = $2; next}
FNR == 1 {print $0 > "goodrec"; print $0 > "badrec"}
FNR >...

You will need to both the files
Schema file first and data file next

If both schjema file (conf) and data file (file) as comma separated
Below code will do the job and send eligible records to a file "goodrec" and bad data to "badrec"

awk -F "," 'NR == FNR {a[NR]...