Files extraction - any help ?

Login or Register to Ask a Question and Join Our Community

Files extraction - any help ?

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting Files extraction - any help ?

04-30-2010

Registered User

60, 0

Join Date: Jul 2009

Last Activity: 28 January 2011, 7:37 AM EST

Location: India [Great India] hey hey .. urs also greaty country frnd

Posts: 60

Thanks Given: 0

Thanked 0 Times in 0 Posts

Files extraction - any help ?

Hi Friends,
i am new to unix,i have a big doubt/help.

I have files in folders SER1 and SER2 with naming convention as below
file_2010-03-19.txt and so on
the file naming format is file_<date>.txt. I would like to copy the files to directory "Landing"

I have entries in a log file log.txt as below

2010-02-19 SER1 SER2 failed
2010-02-10 SER2 failed
2010-04-14 SER1 failed
2010-04-26 SER1 SER2 failed

Its indicate that the files ,named using the date(first filed) not extracted(copied)successfully from the folders SER1 (2nd) and SER2 (3rd)

Now using this log file I want look into the failed folders and try to extract the files that named with date in1st filed of log file. And the script that I we are going to develop should use this log.txt file and extract all files from respective failed folders(SER1/SER2 or from Both).

If the file extraction once again failed for any date/SER1/SER2 , my script should prepare same log file (log.txt) at the end.
If the extraction success from both folders, no need of entry in log file. If any one one folder or both failed, I need a entry in th log file.

Can any one help me out on this?

Gopal_Engg

View Public Profile for Gopal_Engg

Find all posts by Gopal_Engg

04-30-2010

Registered User

6,575, 572

Join Date: Sep 2007

Last Activity: 5 November 2019, 9:08 AM EST

Location: St. Gallen, Switzerland

Posts: 6,575

Thanks Given: 179

Thanked 572 Times in 484 Posts

You have and had several infractions cause of code tags by different mods; if you have 30 points, you are out. Writing this here in the thread since it seems you ignore PMs about it.

Your post is totally messed up with formatting tags (where there is no reason for it - there is not only the bold tags) - you'll have to replace them with code tags your own.

zaxxon

View Public Profile for zaxxon

Find all posts by zaxxon

04-30-2010

Registered User

1,714, 63

Join Date: Apr 2004

Last Activity: 15 May 2020, 11:27 AM EDT

Location: Bordeaux, France

Posts: 1,714

Thanks Given: 2

Thanked 63 Times in 59 Posts

You can do something like that :

Code:

LogFile=log.txt
TargetDir=Landing

> ${LogFile}.new

while read date infos
do
   set -- ${infos}
   eval status=\$$#

   # Keeps entries with status not equal to failed

   if [ "${status}" != "failed" ]
   then
      echo "${date} ${infos}" >> ${LogFile}.new
      continue
   fi

   # Extract file in each failed directory

   new_failed_dirs=""
   while [ $# -gt 1 ]
   do
      dir=$1
      shift
      file="${dir}/file_${date}.txt"
      echo "\nExtracting file ${file} ..."
      if cp ${file} ${TargetDir}/
      then
          echo "Ok."
      else
          echo "Failed."
          new_failed_dirs="${new_failed_dirs} ${dir}"
      fi
   done

   # Write failed directory into new log

   if [ -n "${new_failed_dirs}" ]
   then
      echo "${date} ${new_failed_dirs} failed" >> ${LogFile}.new
   fi

done < log.txt
mv ${LogFile} ${LogFile}.old && mv ${LogFile}.new ${LogFile}

Input file log.txt :

Code:

2010-02-19 SER1 SER2 failed
2010-02-10 SER2 failed
2010-04-14 SER1 failed
2010-04-26 SER1 SER2 failed

Script output :

Code:

Extracting file SER1/file_2010-02-19.txt ...
cp: SER1/file_2010-02-19.txt: No such file or directory
Failed.

Extracting file SER2/file_2010-02-19.txt ...
Ok.

Extracting file SER2/file_2010-02-10.txt ...
cp: SER2/file_2010-02-10.txt: No such file or directory
Failed.

Extracting file SER1/file_2010-04-14.txt ...
cp: SER1/file_2010-04-14.txt: No such file or directory
Failed.

Extracting file SER1/file_2010-04-26.txt ...
Ok.

Extracting file SER2/file_2010-04-26.txt ...
cp: SER2/file_2010-04-26.txt: No such file or directory
Failed.

Files in Landing directory ;

Code:

file_2010-02-19.txt  file_2010-04-26.txt

New log file log.txt :

Code:

2010-02-19  SER1 failed
2010-02-10  SER2 failed
2010-04-14  SER1 failed
2010-04-26  SER2 failed

The old log file is saved in log.txt.old

Jean-Pierre.

aigles

View Public Profile for aigles

Find all posts by aigles

05-03-2010

Registered User

60, 0

Join Date: Jul 2009

Last Activity: 28 January 2011, 7:37 AM EST

Location: India [Great India] hey hey .. urs also greaty country frnd

Posts: 60

Thanks Given: 0

Thanked 0 Times in 0 Posts

Hi friend,

Thank you very much. Can you explain how this script working. Especially i dont understand the while loop.

while read date infos
whats s this infos. whay r u using it in while loop. whats the use. and

set -- ${infos} - i dont understand this

eval status=\$$# - i dont understand this

Please explain me Jean-Pierre.

Gopal_Engg

View Public Profile for Gopal_Engg

Find all posts by Gopal_Engg

05-03-2010

Registered User

1,714, 63

Join Date: Apr 2004

Last Activity: 15 May 2020, 11:27 AM EDT

Location: Bordeaux, France

Posts: 1,714

Thanks Given: 2

Thanked 63 Times in 59 Posts

Code:

read date infos

This command reads a line from the input file, and stores the first word in the variable date and the remaining in the variable infos.

Code:

set -- ${infos}
eval status=\$$#

This command parses the contents of the variable info,words are stored into variables $1 $2 $3... the number of words is given by $#
The value of the last word is stored in the variable status

Code:

$ echo "2010-02-19 SER1 SER2 failed" | read date infos
$ echo "date=[$date] infos=[$infos]"
date=[2010-02-19] infos=[SER1 SER2 failed]
$ set -- ${infos}
$ echo "\$#=$# \$1=[$1] \$2=[$2] \$3=[$3]"$#=3 $1=[SER1] $2=[SER2] $3=[failed]
$ eval status=\$$#
$ echo "status=[$status]"
status=[failed]
$ set -x
$ eval status=\$$#
+ eval status=$3
+ status=failed
$

Jean-Pierre.

aigles

View Public Profile for aigles

Find all posts by aigles

05-03-2010

Registered User

60, 0

Join Date: Jul 2009

Last Activity: 28 January 2011, 7:37 AM EST

Location: India [Great India] hey hey .. urs also greaty country frnd

Posts: 60

Thanks Given: 0

Thanked 0 Times in 0 Posts

thank you very much

Gopal_Engg

View Public Profile for Gopal_Engg

Find all posts by Gopal_Engg

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX - 2 tab delimited files, conditional column extraction

Please know that I am very new to unix and trying to learn 'on the job'. I'm only manipulating large tab-delimited files (millions of rows), but I'm stuck and don't know how to proceed with the following. Hoping for some friendly advice :) I have 2 tab-delimited files - with differing column &...

2. Shell Programming and Scripting

Speed up extraction od tar.bz2 files using bash

The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2. Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process? I am using a xeon processor with 12 cores. Thank you :). ...

3. Shell Programming and Scripting

Extraction of sequences from files

hey!!! I have 2 files file1 is as ids.txt and is >gi|546473186|gb|AWWX01630222.1| >gi|546473233|gb|AWWX01630175.1| >gi|546473323|gb|AWWX01630097.1| >gi|546474044|gb|AWWX01629456.1| >gi|546474165|gb|AWWX01629352.1| file2 is sequences.fasta and is like >gi|546473233|gb|AWWX01630175.1|...

4. Shell Programming and Scripting

Removal of extra spaces in *.log files to allow extraction of frequencies

Our university has upgraded its version of a computational chemistry program that our group uses quite regularly. In the past we have been able to extract frequency spectra from log files that are generated. Since the upgrade, the viewing program errors out. I've been able to trace down the changes...

5. Shell Programming and Scripting

Column extraction from multiple files to multiple files

I have roughly ~30 .txt files in a directory which all have unique names. These files all contain text arranged in columns separated by whitespace (example file: [#YY MM DD hh mm WDIR WSPD GST WVHT DPD APD MWD PRES ATMP WTMP DEWP VIS TIDE #yr mo dy hr mn degT m/s m/s m sec ...

6. UNIX for Dummies Questions & Answers

Need help for data extraction if files

Hello all, I want to extract some particular data from a files and than add all the values . but i m not able to cut the particular word(USU-INOCT and USU-OUTOCT) as it is coming not in column. and than able to add values coming in it . can anyone help me Please cat <file name> ...

7. Shell Programming and Scripting

Extraction of data from multiple text files, and creation of a chart

Hello dear friends, My problem as explained below seems really basic. Fact is that I'm totally new to programming, and have only a week to produce a script ( CShell or Perl ? ) to perform this action. While searching on the forums, I found a command that could help me, but I don't know...

8. Shell Programming and Scripting

Selective extraction of data from a files

Hi, I would like to seek for methods to do selective extraction of line froma file. The scenario as follows: I have a file with content: message a received on 11:10:00 file size: 10 bytes send by abc message b received on 11:20:00 file size: 10 bytes send by abc

9. UNIX for Dummies Questions & Answers

merged 10 files with column extraction into one

Hi, I have 600 text files. In each txt file, I have 3 columns, e.g: File 1 a 0.21 0.003 b 0.34 0.004 c 0.72 0.002 File 2 a 0.25 0.0083 b 0.38 0.0047 c 0.79 0.00234 File 3 a 0.45 0.0063 b 0.88 0.0027 c 0.29 0.00204 ... my filename as "sc2408_0_5278.txt sc2408_0_5279.txt...

10. Shell Programming and Scripting

Extraction of latest files from cvs repository

Hi everyone.. Anybody having idea to get the latest file from CVS repository through schell scripts. Thanks in advance. Regards shahid Bakshi

Login or Register to Ask a Question