Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Files extraction - any help ?
Top Forums Shell Programming and Scripting Files extraction - any help ? Post 302417715 by aigles on Friday 30th of April 2010 10:15:42 AM
Old 04-30-2010
You can do something like that :
Code:
LogFile=log.txt
TargetDir=Landing

> ${LogFile}.new

while read date infos
do
   set -- ${infos}
   eval status=\$$#

   # Keeps entries with status not equal to failed

   if [ "${status}" != "failed" ]
   then
      echo "${date} ${infos}" >> ${LogFile}.new
      continue
   fi

   # Extract file in each failed directory

   new_failed_dirs=""
   while [ $# -gt 1 ]
   do
      dir=$1
      shift
      file="${dir}/file_${date}.txt"
      echo "\nExtracting file ${file} ..."
      if cp ${file} ${TargetDir}/
      then
          echo "Ok."
      else
          echo "Failed."
          new_failed_dirs="${new_failed_dirs} ${dir}"
      fi
   done

   # Write failed directory into new log

   if [ -n "${new_failed_dirs}" ]
   then
      echo "${date} ${new_failed_dirs} failed" >> ${LogFile}.new
   fi

done < log.txt
mv ${LogFile} ${LogFile}.old && mv ${LogFile}.new ${LogFile}

Input file log.txt :
Code:
2010-02-19 SER1 SER2 failed
2010-02-10 SER2 failed
2010-04-14 SER1 failed
2010-04-26 SER1 SER2 failed

Script output :
Code:
Extracting file SER1/file_2010-02-19.txt ...
cp: SER1/file_2010-02-19.txt: No such file or directory
Failed.

Extracting file SER2/file_2010-02-19.txt ...
Ok.

Extracting file SER2/file_2010-02-10.txt ...
cp: SER2/file_2010-02-10.txt: No such file or directory
Failed.

Extracting file SER1/file_2010-04-14.txt ...
cp: SER1/file_2010-04-14.txt: No such file or directory
Failed.

Extracting file SER1/file_2010-04-26.txt ...
Ok.

Extracting file SER2/file_2010-04-26.txt ...
cp: SER2/file_2010-04-26.txt: No such file or directory
Failed.

Files in Landing directory ;
Code:
file_2010-02-19.txt  file_2010-04-26.txt

New log file log.txt :
Code:
2010-02-19  SER1 failed
2010-02-10  SER2 failed
2010-04-14  SER1 failed
2010-04-26  SER2 failed

The old log file is saved in log.txt.old


Jean-Pierre.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extraction of latest files from cvs repository

Hi everyone.. Anybody having idea to get the latest file from CVS repository through schell scripts. Thanks in advance. Regards shahid Bakshi (4 Replies)
Discussion started by: shahidbakshi
4 Replies

2. UNIX for Dummies Questions & Answers

merged 10 files with column extraction into one

Hi, I have 600 text files. In each txt file, I have 3 columns, e.g: File 1 a 0.21 0.003 b 0.34 0.004 c 0.72 0.002 File 2 a 0.25 0.0083 b 0.38 0.0047 c 0.79 0.00234 File 3 a 0.45 0.0063 b 0.88 0.0027 c 0.29 0.00204 ... my filename as "sc2408_0_5278.txt sc2408_0_5279.txt... (2 Replies)
Discussion started by: libenhelen
2 Replies

3. Shell Programming and Scripting

Selective extraction of data from a files

Hi, I would like to seek for methods to do selective extraction of line froma file. The scenario as follows: I have a file with content: message a received on 11:10:00 file size: 10 bytes send by abc message b received on 11:20:00 file size: 10 bytes send by abc (3 Replies)
Discussion started by: dwgi32
3 Replies

4. Shell Programming and Scripting

Extraction of data from multiple text files, and creation of a chart

Hello dear friends, My problem as explained below seems really basic. Fact is that I'm totally new to programming, and have only a week to produce a script ( CShell or Perl ? ) to perform this action. While searching on the forums, I found a command that could help me, but I don't know... (2 Replies)
Discussion started by: ackheron
2 Replies

5. UNIX for Dummies Questions & Answers

Need help for data extraction if files

Hello all, I want to extract some particular data from a files and than add all the values . but i m not able to cut the particular word(USU-INOCT and USU-OUTOCT) as it is coming not in column. and than able to add values coming in it . can anyone help me Please cat <file name> ... (7 Replies)
Discussion started by: anamdev
7 Replies

6. Shell Programming and Scripting

Column extraction from multiple files to multiple files

I have roughly ~30 .txt files in a directory which all have unique names. These files all contain text arranged in columns separated by whitespace (example file: [#YY MM DD hh mm WDIR WSPD GST WVHT DPD APD MWD PRES ATMP WTMP DEWP VIS TIDE #yr mo dy hr mn degT m/s m/s m sec ... (5 Replies)
Discussion started by: aozgaa
5 Replies

7. Shell Programming and Scripting

Removal of extra spaces in *.log files to allow extraction of frequencies

Our university has upgraded its version of a computational chemistry program that our group uses quite regularly. In the past we have been able to extract frequency spectra from log files that are generated. Since the upgrade, the viewing program errors out. I've been able to trace down the changes... (16 Replies)
Discussion started by: wsuchem
16 Replies

8. Shell Programming and Scripting

Extraction of sequences from files

hey!!! I have 2 files file1 is as ids.txt and is >gi|546473186|gb|AWWX01630222.1| >gi|546473233|gb|AWWX01630175.1| >gi|546473323|gb|AWWX01630097.1| >gi|546474044|gb|AWWX01629456.1| >gi|546474165|gb|AWWX01629352.1| file2 is sequences.fasta and is like >gi|546473233|gb|AWWX01630175.1|... (9 Replies)
Discussion started by: harpreetmanku04
9 Replies

9. Shell Programming and Scripting

Speed up extraction od tar.bz2 files using bash

The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2. Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process? I am using a xeon processor with 12 cores. Thank you :). ... (7 Replies)
Discussion started by: cmccabe
7 Replies

10. UNIX for Beginners Questions & Answers

UNIX - 2 tab delimited files, conditional column extraction

Please know that I am very new to unix and trying to learn 'on the job'. I'm only manipulating large tab-delimited files (millions of rows), but I'm stuck and don't know how to proceed with the following. Hoping for some friendly advice :) I have 2 tab-delimited files - with differing column &... (10 Replies)
Discussion started by: GTed
10 Replies

LEARN ABOUT DEBIAN

parallel-slurp

PARALLEL-SLURP(1)														 PARALLEL-SLURP(1)

NAME

       parallel-slurp - copy files from listed hosts

SYNOPSIS

       parallel-slurp [OPTIONS] -h hosts.txt -L destdir remote local

DESCRIPTION

       pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of
       homogenously-configured hosts.

       parallel-slurp gathers specified files from hosts you listed.

OPTIONS

       -r --recursive
	   recusively copy directories (OPTIONAL)

       -L --localdir
	   output directory for remote file copies

       -h --hosts
	   hosts file (each line "host[:port] [user]")

       -l --user
	   username (OPTIONAL)

       -p --par
	   max number of parallel threads (OPTIONAL)

       -o --outdir
	   output directory for stdout files (OPTIONAL)

       -e --errdir
	   output directory for stderr files (OPTIONAL)

       -t --timeout
	   timeout (secs) (-1 = no timeout) per host (OPTIONAL)

       -O --options
	   SSH options (OPTIONAL)

       -v --verbose
	   turn on warning and diagnostic messages (OPTIONAL)

EXAMPLE

       An example to copy /home/irb2/foo.txt from each host. Files gathered will be stored in /tmp/outdir/hostname/foo.txt.

	   # prallel-slurp -h hosts.txt -L /tmp/outdir -l irb2 
		    /home/irb2/foo.txt foo.txt

ENVIRONMENT

       All four programs take similar sets of options. All of these options can be set using the following environment variables:

       o   PSSH_HOSTS
       o   PSSH_USER
       o   PSSH_PAR
       o   PSSH_OUTDIR
       o   PSSH_VERBOSE
       o   PSSH_OPTIONS

SEE ALSO

       parallel-ssh(1), parallel-scp(1), parallel-nuke(1), parallel-rsync(1), ssh(1)

AUTHOR

       Brent N. Chun <bnc@theether.org>

COPYING

       Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun

NOTES

	1. bnc@theether.org
	   mailto:bnc@theether.org

								    03/30/2009							 PARALLEL-SLURP(1)
All times are GMT -4. The time now is 05:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy