Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Checking for the file existence

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Checking for the file existence
# 8  
Old 05-08-2014
Code:
awk -F "," -vDT="$(date +%m%d%Y%H%M)" 'BEGIN {GOOD = "goodrec_" DT;
BAD = "badrec_" DT;
putG = "hadoop fs -put /home/hduser/saptha/validate/" GOOD " /user/hduser/Dataparse/";
putB = "hadoop fs -put /home/hduser/saptha/validate/" BAD " /user/hduser/Dataparse/"}
NR == FNR{
h = (h == "") ? $1 : (h FS $1); 
gsub("[)(]", "-", $2);
split($2, a, "-"); 
d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}  
FNR == 1 {print h > GOOD; print h > BAD} 
{
  for(i = 1; i <= NF; i++)
  {
   
     if(((d[i] == "Integer" && (($i + 0) == $i || $i == "")) ||  (d[i] == "String" && ($i + 0) != $i) || (d[i] == "Char"  && ($i + 0) != $i)) && (length($i) <= l[i])  && (length($i) >= n[i]))
      {f = 1} else {f = 0};
        if(f == 0) {print $0 > BAD; b++; next}
  }
    print $0 > GOOD; g++
}
END {
print "Parsing Success!";
print "Validated records are found on the Hadoop Path \"/user/hduser/Dataparse\""
system(putG);
system(putB);
    }' configfile.txt datafile.txt d1.txt

# 9  
Old 05-08-2014
Hi thank you Srini, it's working perfectly fine!!!

---------- Post updated at 01:28 AM ---------- Previous update was at 12:28 AM ----------

Hi Srini,

I have one more query to ask you.
Below is the script :
Code:
#!/bin/bash

awk -F "," -vDT="$(date +%m%d%Y%H%M)" 'BEGIN {GOOD = "goodrec_" DT;
BAD = "badrec_" DT;
putG = "hadoop fs -put /home/hduser/saptha/validate/" GOOD " /user/hduser/Dataparse/";
putB = "hadoop fs -put /home/hduser/saptha/validate/" BAD " /user/hduser/Dataparse/"}
NR == FNR{
h = (h == "") ? $1 : (h FS $1); 
gsub("[)(]", "-", $2);
split($2, a, "-"); 
d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}  
FNR == 1 {print h > GOOD; print h > BAD} 
{
  for(i = 1; i <= NF; i++)
  {
   
     if(((d[i] == "Integer" && (($i + 0) == $i || $i == "")) ||  (d[i] == "String" && ($i + 0) != $i) || (d[i] == "Char"  && ($i + 0) != $i)) && (length($i) <= l[i])  && (length($i) >= n[i]))
      {f = 1} else {f = 0};
        if(f == 0) {print $0 > BAD; b++; next}
  }
    print $0 > GOOD; g++
}
END {
print "Parsing Success!";
print "Count of Good Records : " g;
print "Count of Bad Records : " b;
print "Validated records are found on the Hadoop Path \"/user/hduser/Dataparse\""
system(putG);
system(putB);
 }' configfile.txt d1.txt

hive <<! > hive_out.log
create table GoodRecords(
id int,
name string,
state string,
phone_no int,
gender string) row format delimited fields terminated by ',' stored as textfile;
LOAD DATA INPATH '/user/hduser/Dataparse/goodrec' INTO TABLE GoodRecords;
exit;
!

In the above code I'm finally moving/copying the data from HDFS to HIVE.
In the hive query i wanted create the hive table in such a way that , for the field name and datatypes it should refer to the config file. Ho wthis can be done ?
My config.txt file is
Code:
id,Integer(2),NOT NULL
name,String(20)
state,String(5),NOT NULL
phone_no,Integer(4)
gender,Char(1)

Thnaks,
Shree
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Command script for checking a file existence

Hello, I have a directory where sometimes appear a certain file name - and I'd like to be notified by email when that happens... so what command or script I may use? e.g. if there's a file named "adam" in the directory named "dir1" then send a mail to "abc@abc.com".. it needs to permanently... (5 Replies)
Discussion started by: netrom
5 Replies

2. Shell Programming and Scripting

Checking file existence along with condition

Hi am trying to write a script which find the existence of a file from a find command output and perform a task if the file exists. Help me out with the correct syntax . Am trying with the following one but unable to get the output. if then <some tasks> else echo "file not exists" fi (5 Replies)
Discussion started by: rogerben
5 Replies

3. Shell Programming and Scripting

Checking existence of file using awk

Hi, I need to check whether a particular file exists ot not using awk. Can anyone help me please? For Example:script that i am using: awk '{filename =$NF; rc=(system("test -r filename")) print $rc;}' "$1" is not working. Here I am passing a text file as input whose last word contains a... (6 Replies)
Discussion started by: manish007
6 Replies

4. Shell Programming and Scripting

checking the file existence using ssh

Hi Can any body say me the reason for below error ssh -o 'StrictHostKeyChecking no' user@client ' && print "1"' I am getting error as "Missing ]":wall: (6 Replies)
Discussion started by: ramesh12621
6 Replies

5. Shell Programming and Scripting

Checking the existence of a file before getting last modified

Hi, I am trying to check the existence of a file, from a list of possible filenames: status-A status-B status-C before retrieving the last modified datetime using ls, I want to check it exists or ls will throw an error. So I have tried this: if ; then ls status-* fi But the if... (3 Replies)
Discussion started by: LostInTheWoods
3 Replies

6. Shell Programming and Scripting

Multiple file existence and checking file size

I want to check the files in particular directory are more that 0 Bytes i.e, Non zero byte file. The script should print a msg if all the files in that directory are empty( 0 Byte). (2 Replies)
Discussion started by: lathish
2 Replies

7. Shell Programming and Scripting

Checking Multiple file existence

Hi, I want to check multiple files exist or not in a single if statement in korn Shell:confused:. Please help me Thanks (1 Reply)
Discussion started by: lathish
1 Replies

8. Shell Programming and Scripting

Checking for existence of a flat file in UNIX !

Hi All, I have a requirement where in i need to check for existence of a file and later execute some pmcmd commands related to informatica. I tried many ways but was unsuccessful could you please throw some light. Below are the sample codes i wrote. Example 1: #!/bin/ksh... (4 Replies)
Discussion started by: Ariean
4 Replies

9. Shell Programming and Scripting

Checking the existence of a file..

Hi, I am trying to check for the existence of a file using the 'test' and the file existence options. When trying to check for a file with a space in between e.g 'Team List', it gives the following error. learn1: line 3: test: `Team: binary operator expected I am pasting my code below as... (7 Replies)
Discussion started by: igandu
7 Replies

10. Shell Programming and Scripting

checking file existence

Hi, My requirement was to check the existence of a file having a specified pattern.The way i tried to achieve this was if ; then echo "File found" fi an example file having this pattern was 'ilvs_trace01.0124'. it will vary... (3 Replies)
Discussion started by: DILEEP410
3 Replies
Login or Register to Ask a Question