Checking for the file existence Post: 302900486

Sponsored Content

Top Forums Shell Programming and Scripting Checking for the file existence Post 302900486 by shree11 on Wednesday 7th of May 2014 06:09:12 AM

05-07-2014

Registered User

Checking for the file existence

Hi,

I have written a script to validate the data file by referreing to the configurtion file. And moving the validated good records and bad records into HDFS.
Suppose after 15 mins if i receive one more data fie,then after validation the good and bad records shold be stored in hadoop with the timestamp attached to it.
So whenever the new data file comes ,after valifdation the bad and good records are stored in the hadoop with the timestamp attached to the file name.
How this can be done using the unix script?

Below is the code written so for

Code:

#!/bin/bash 

awk -F "," 'NR == FNR{
h = (h == "") ? $1 : (h FS $1); 
gsub("[)(]", "-", $2);
split($2, a, "-"); 
d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}  
FNR == 1 {print h > "goodrec"; print h > "badrec"} 
{
  for(i = 1; i <= NF; i++)
  {
   
     if(((d[i] == "Integer" && (($i + 0) == $i || $i == "")) ||  (d[i] == "String" && ($i + 0) != $i) || (d[i] == "Char"  && ($i + 0) != $i)) && (length($i) <= l[i])  && (length($i) >= n[i]))
      {f = 1} else {f = 0};
        if(f == 0) {print $0 > "badrec"; b++; next}
  }
    print $0 > "goodrec"; g++
}
END {
print "Parsing Success!";
print "Validated records are found on the Hadoop Path \"/user/hduser/Dataparse\""
    }' configfile.txt datafile.txt d1.txt

#Loading good and bad records on HDFS
hadoop fs -put /home/hduser/saptha/validate/badrec /user/hduser/Dataparse/
hadoop fs -put /home/hduser/saptha/validate/goodrec /user/hduser/Dataparse/

So here i want goodrec and badrec file should be like for each data file.

Code:

goodrec_timestamp
badrec_timestamp

Thanks,
Shree

shree11

View Public Profile for shree11

Find all posts by shree11

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

checking file existence

Hi, My requirement was to check the existence of a file having a specified pattern.The way i tried to achieve this was if ; then echo "File found" fi an example file having this pattern was 'ilvs_trace01.0124'. it will vary...

2. Shell Programming and Scripting

Checking the existence of a file..

Hi, I am trying to check for the existence of a file using the 'test' and the file existence options. When trying to check for a file with a space in between e.g 'Team List', it gives the following error. learn1: line 3: test: `Team: binary operator expected I am pasting my code below as...

3. Shell Programming and Scripting

Checking for existence of a flat file in UNIX !

Hi All, I have a requirement where in i need to check for existence of a file and later execute some pmcmd commands related to informatica. I tried many ways but was unsuccessful could you please throw some light. Below are the sample codes i wrote. Example 1: #!/bin/ksh...

4. Shell Programming and Scripting

Checking Multiple file existence

Hi, I want to check multiple files exist or not in a single if statement in korn Shell:confused:. Please help me Thanks

5. Shell Programming and Scripting

Multiple file existence and checking file size

I want to check the files in particular directory are more that 0 Bytes i.e, Non zero byte file. The script should print a msg if all the files in that directory are empty( 0 Byte).

6. Shell Programming and Scripting

Checking the existence of a file before getting last modified

Hi, I am trying to check the existence of a file, from a list of possible filenames: status-A status-B status-C before retrieving the last modified datetime using ls, I want to check it exists or ls will throw an error. So I have tried this: if ; then ls status-* fi But the if...

7. Shell Programming and Scripting

checking the file existence using ssh

Hi Can any body say me the reason for below error ssh -o 'StrictHostKeyChecking no' user@client ' && print "1"' I am getting error as "Missing ]":wall:

8. Shell Programming and Scripting

Checking existence of file using awk

Hi, I need to check whether a particular file exists ot not using awk. Can anyone help me please? For Example:script that i am using: awk '{filename =$NF; rc=(system("test -r filename")) print $rc;}' "$1" is not working. Here I am passing a text file as input whose last word contains a...

9. Shell Programming and Scripting

Checking file existence along with condition

Hi am trying to write a script which find the existence of a file from a find command output and perform a task if the file exists. Help me out with the correct syntax . Am trying with the following one but unable to get the output. if then <some tasks> else echo "file not exists" fi

10. Shell Programming and Scripting

Command script for checking a file existence

Hello, I have a directory where sometimes appear a certain file name - and I'd like to be notified by email when that happens... so what command or script I may use? e.g. if there's a file named "adam" in the directory named "dir1" then send a mail to "abc@abc.com".. it needs to permanently...

LEARN ABOUT LINUX

xfs_logprint

xfs_logprint(8) 					      System Manager's Manual						   xfs_logprint(8)

NAME

       xfs_logprint - print the log of an XFS filesystem

SYNOPSIS

       xfs_logprint [ options ] device

DESCRIPTION

       xfs_logprint prints the log of an XFS filesystem (see xfs(5)).  The device argument is the pathname of the partition or logical volume con-
       taining the filesystem. The device can be a regular file if the -f option is used. The  contents  of  the  filesystem  remain  undisturbed.
       There are two major modes of operation in xfs_logprint.

       One  mode  is  better  for  filesystem  operation debugging.  It is called the transactional view and is enabled through the -t option. The
       transactional view prints only the portion of the log that pertains to recovery. In  other  words,  it  prints  out  complete  transactions
       between the tail and the head. This view tries to display each transaction without regard to how they are split across log records.

       The  second mode starts printing out information from the beginning of the log.	Some error blocks might print out in the beginning because
       the last log record usually overlaps the oldest log record. A message is printed when the physical end of the log is reached and  when  the
       logical	end  of  the  log  is  reached. A log record view is displayed one record at a time. Transactions that span log records may not be
       decoded fully.

OPTIONS

       -b     Extract and print buffer information. Only used in transactional view.

       -c     Attempt to continue when an error is detected.

       -C filename
	      Copy the log from the filesystem to the file filename.  The log itself is not printed.

       -d     Dump the log from front to end, printing where each log record is located on disk.

       -D     Do not decode anything; just print data.

       -e     Exit when an error is found in the log. Normally, xfs_logprint tries to continue and unwind from bad logs.   However,  sometimes	it
	      just dies in bad ways.  Using this option prevents core dumps.

       -f     Specifies  that  the  filesystem	image  to be processed is stored in a regular file at device (see the mkfs.xfs(8) -d file option).
	      This might happen if an image copy of a filesystem has been made into an ordinary file with xfs_copy(8).

       -l logdev
	      External log device. Only for those filesystems which use an external log.

       -i     Extract and print inode information. Only used in transactional view.

       -q     Extract and print quota information. Only used in transactional view.

       -n     Do not try and interpret log data; just interpret log header information.

       -o     Also print buffer data in hex.  Normally, buffer data is just decoded, so better information can be printed.

       -s start-block
	      Override any notion of where to start printing.

       -t     Print out the transactional view.

SEE ALSO

       mkfs.xfs(8), mount(8).

																   xfs_logprint(8)