Checking for the file existence Post: 302900486

Sponsored Content

Top Forums Shell Programming and Scripting Checking for the file existence Post 302900486 by shree11 on Wednesday 7th of May 2014 06:09:12 AM

05-07-2014

Registered User

Checking for the file existence

Hi,

I have written a script to validate the data file by referreing to the configurtion file. And moving the validated good records and bad records into HDFS.
Suppose after 15 mins if i receive one more data fie,then after validation the good and bad records shold be stored in hadoop with the timestamp attached to it.
So whenever the new data file comes ,after valifdation the bad and good records are stored in the hadoop with the timestamp attached to the file name.
How this can be done using the unix script?

Below is the code written so for

Code:

#!/bin/bash 

awk -F "," 'NR == FNR{
h = (h == "") ? $1 : (h FS $1); 
gsub("[)(]", "-", $2);
split($2, a, "-"); 
d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next}  
FNR == 1 {print h > "goodrec"; print h > "badrec"} 
{
  for(i = 1; i <= NF; i++)
  {
   
     if(((d[i] == "Integer" && (($i + 0) == $i || $i == "")) ||  (d[i] == "String" && ($i + 0) != $i) || (d[i] == "Char"  && ($i + 0) != $i)) && (length($i) <= l[i])  && (length($i) >= n[i]))
      {f = 1} else {f = 0};
        if(f == 0) {print $0 > "badrec"; b++; next}
  }
    print $0 > "goodrec"; g++
}
END {
print "Parsing Success!";
print "Validated records are found on the Hadoop Path \"/user/hduser/Dataparse\""
    }' configfile.txt datafile.txt d1.txt

#Loading good and bad records on HDFS
hadoop fs -put /home/hduser/saptha/validate/badrec /user/hduser/Dataparse/
hadoop fs -put /home/hduser/saptha/validate/goodrec /user/hduser/Dataparse/

So here i want goodrec and badrec file should be like for each data file.

Code:

goodrec_timestamp
badrec_timestamp

Thanks,
Shree

shree11

View Public Profile for shree11

Find all posts by shree11

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

checking file existence

Hi, My requirement was to check the existence of a file having a specified pattern.The way i tried to achieve this was if ; then echo "File found" fi an example file having this pattern was 'ilvs_trace01.0124'. it will vary...

2. Shell Programming and Scripting

Checking the existence of a file..

Hi, I am trying to check for the existence of a file using the 'test' and the file existence options. When trying to check for a file with a space in between e.g 'Team List', it gives the following error. learn1: line 3: test: `Team: binary operator expected I am pasting my code below as...

3. Shell Programming and Scripting

Checking for existence of a flat file in UNIX !

Hi All, I have a requirement where in i need to check for existence of a file and later execute some pmcmd commands related to informatica. I tried many ways but was unsuccessful could you please throw some light. Below are the sample codes i wrote. Example 1: #!/bin/ksh...

4. Shell Programming and Scripting

Checking Multiple file existence

Hi, I want to check multiple files exist or not in a single if statement in korn Shell:confused:. Please help me Thanks

5. Shell Programming and Scripting

Multiple file existence and checking file size

I want to check the files in particular directory are more that 0 Bytes i.e, Non zero byte file. The script should print a msg if all the files in that directory are empty( 0 Byte).

6. Shell Programming and Scripting

Checking the existence of a file before getting last modified

Hi, I am trying to check the existence of a file, from a list of possible filenames: status-A status-B status-C before retrieving the last modified datetime using ls, I want to check it exists or ls will throw an error. So I have tried this: if ; then ls status-* fi But the if...

7. Shell Programming and Scripting

checking the file existence using ssh

Hi Can any body say me the reason for below error ssh -o 'StrictHostKeyChecking no' user@client ' && print "1"' I am getting error as "Missing ]":wall:

8. Shell Programming and Scripting

Checking existence of file using awk

Hi, I need to check whether a particular file exists ot not using awk. Can anyone help me please? For Example:script that i am using: awk '{filename =$NF; rc=(system("test -r filename")) print $rc;}' "$1" is not working. Here I am passing a text file as input whose last word contains a...

9. Shell Programming and Scripting

Checking file existence along with condition

Hi am trying to write a script which find the existence of a file from a find command output and perform a task if the file exists. Help me out with the correct syntax . Am trying with the following one but unable to get the output. if then <some tasks> else echo "file not exists" fi

10. Shell Programming and Scripting

Command script for checking a file existence

Hello, I have a directory where sometimes appear a certain file name - and I'd like to be notified by email when that happens... so what command or script I may use? e.g. if there's a file named "adam" in the directory named "dir1" then send a mail to "abc@abc.com".. it needs to permanently...

LEARN ABOUT DEBIAN

bup-margin

bup-margin(1)						      General Commands Manual						     bup-margin(1)

NAME

       bup-margin - figure out your deduplication safety margin

SYNOPSIS

       bup margin [options...]

DESCRIPTION

       bup margin  iterates  through  all  objects  in	your  bup repository, calculating the largest number of prefix bits shared between any two
       entries.  This number, n, identifies the longest subset of SHA-1 you could use and still encounter a collision between your object ids.

       For example, one system that was tested had a collection of 11 million objects (70 GB), and bup margin returned 45.  That  means  a  46-bit
       hash  would be sufficient to avoid all collisions among that set of objects; each object in that repository could be uniquely identified by
       its first 46 bits.

       The number of bits needed seems to increase by about 1 or 2 for every doubling of the number of objects.  Since SHA-1 hashes have 160 bits,
       that  leaves 115 bits of margin.  Of course, because SHA-1 hashes are essentially random, it's theoretically possible to use many more bits
       with far fewer objects.

       If you're paranoid about the possibility of SHA-1 collisions, you can monitor your repository by running bup margin occasionally to see	if
       you're getting dangerously close to 160 bits.

OPTIONS

       --predict
	      Guess  the offset into each index file where a particular object will appear, and report the maximum deviation of the correct answer
	      from the guess.  This is potentially useful for tuning an interpolation search algorithm.

       --ignore-midx
	      don't use .midx files, use only .idx files.  This is only really useful when used with --predict.

EXAMPLE

	      $ bup margin
	      Reading indexes: 100.00% (1612581/1612581), done.
	      40
	      40 matching prefix bits
	      1.94 bits per doubling
	      120 bits (61.86 doublings) remaining
	      4.19338e+18 times larger is possible

	      Everyone on earth could have 625878182 data sets
	      like yours, all in one repository, and we would
	      expect 1 object collision.

	      $ bup margin --predict
	      PackIdxList: using 1 index.
	      Reading indexes: 100.00% (1612581/1612581), done.
	      915 of 1612581 (0.057%)

SEE ALSO

       bup-midx(1), bup-save(1)

BUP

       Part of the bup(1) suite.

AUTHORS

       Avery Pennarun <apenwarr@gmail.com>.

Bup unknown-															     bup-margin(1)