Copy all the files with time stamp and remove header,trailer from file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Copy all the files with time stamp and remove header,trailer from file
# 1  
Old 07-23-2007
Copy all the files with time stamp and remove header,trailer from file

All,

I am new to unix and i have the following requirement.

I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file..

Could anybody please throw some light on this how to write a script for above requirement ..

Thanks in advance..
# 2  
Old 07-23-2007
Ksrams,
Please give us more information, such as sample of file name with the
timestamp and the expected output.
# 3  
Old 07-23-2007
Thanks for quick reply Shell_Life.
Quote:
Originally Posted by Shell_Life
Ksrams,
Please give us more information, such as sample of file name with the
timestamp and the expected output.
Here is the info.
Input file(s) located under directory /input and file names are:
1) input1_20070723_171556.txt
2) input2_20070723_171556.txt
3) input3_20070723-171556.txt
input1_20070723_171556.txt sample data:

HEADERRECORD 20070723 DESC
aaa 111 222 333 bbb
bbb 222 111 444 ccc
ccc 333 444 111 ddd
TRAILERRECORD 3


I want the do the following process in one script..
a) First thing is i want to copy all the files into /ouput directory
b) rename the files under /out directory to
1) rename input1_20070723_171557.txt file to input1.txt and also remove HEADERRECORD 20070723 DESC(header record i.e always first line) and TRAILERRECORD 3(Detail record i.e always last line in file) from input1.txt
input1.txt file data should be:

aaa 111 222 333 bbb
bbb 222 111 444 ccc
ccc 333 444 111 ddd


2) rename input2_20070723_171557.txt to input2.txt and also remove first and last lines from input2.txt
3) rename input3_20070723_171556.txt to input3.txt and also remove first and last lines from input3.txt

I hope this is much better compare to my previous post.
I would appreciate if you guide me on this..

Thanks in advance..
# 4  
Old 07-24-2007
Script

I think this script should help you

Code:
#!/bin/ksh

Input_Dir="$HOME/input"
Output_Dir="$HOME/out"

rm -f $Output_Dir/*
for file in `ls $Input_Dir`  # If the input directory contains only files
# for file in `ls -l $Input_Dir | grep '^-' | awk '{print $9}'` #Only files in Dir
do
        new_file=${file%%_*}
        wc -l $Input_Dir/$file | awk '{ print $1}' | read line_count
        sed "1d;$line_count""d" $Input_Dir/$file > $Output_Dir/$new_file.txt
        touch -r $Input_Dir/$file $Output_Dir/$new_file.txt
done
exit

# 5  
Old 07-24-2007
Code:
mInDir='/Input/'
mOutDir='/Output/'
mFileExp='input*.txt'
for mFile in `find ${mInDir} -type f -name "${mFileExp}"`
do
  mFName=`echo ${mFile} | sed 's/.*\(input.*\)_20.*txt/\1/'`
  mOutFile=${mOutDir}${mFName}'.txt'
  sed '1d;$d' ${mFile} > ${mOutFile}
  rm -f ${mFile}
done

# 6  
Old 07-24-2007
Quote:
Originally Posted by Shell_Life
Code:
mInDir='/Input/'
mOutDir='/Output/'
mFileExp='input*.txt'
for mFile in `find ${mInDir} -type f -name "${mFileExp}"`
do
  mFName=`echo ${mFile} | sed 's/.*\(input.*\)_20.*txt/\1/'`
  mOutFile=${mOutDir}${mFName}'.txt'
  sed '1d;$d' ${mFile} > ${mOutFile}
  rm -f ${mFile}
done

Hi Shell_Life,

Input files not always starts with input*.txt,input files could start with any name.I want to process all the files under input directory and filename last 20 characters are always timestamp.txt..Could you please modify this script to process all the files instead of input*.txt files.Would it possible if we pass input and output directories as parameters to this script.

Thanks in advance..
# 7  
Old 07-24-2007
Code:
#!/bin/ksh
mInDir=$1       ## Parameter one
mOutDir=$2     ## Parameter two
mDTimeExp='_[0-9]\{8\}.[0-9]\{6\}.txt'
ls ${mInDir} | grep $mDTimeExp | \
while read mFile
do
  mFName=`echo ${mFile} | sed "s/\(.*\)${mDTimeExp}/\1/"`
  mOutFile=${mOutDir}${mFName}'.txt'
  sed '1d;$d' ${mFile} > ${mOutFile}
  rm -f ${mFile}
done


Last edited by Shell_Life; 07-24-2007 at 03:54 PM.. Reason: Correct parameters.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Append pipe | at the end of all the rows except header n trailer for all the files under a directory

Hi Experts Need help... I am looking for a Unix script to append pipe | at the end of all the rows (except header and trailer)in all the files placed under the directory /interfaces/Temp e.g. Header row1 row2 row3 Trailer The script should read all the files under... (3 Replies)
Discussion started by: phani333
3 Replies

2. Shell Programming and Scripting

Verify the header and trailer in file

please see my requirement, I hope I am clear. (9 Replies)
Discussion started by: mirwasim
9 Replies

3. Shell Programming and Scripting

Script to validate file header and trailer

Hi, I need a script that validates a file header/detail/trailer. File layout is: Header - Rec_Type|File_name|File_Date Detail - Rec_Type|field1|field2|field3... Trailder - Rec_Type|File_name|File_Date|Record_count Sample Data: HDR|customer_data.dat|20120709... (7 Replies)
Discussion started by: ash_sh
7 Replies

4. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh... (2 Replies)
Discussion started by: nvuradi
2 Replies

5. UNIX for Dummies Questions & Answers

Adding header and trailer into a file

Hi, I want to add the below Header to all the files in sequence File1,File2,File3...etc "ABC,<number of chracter in the file>" e,g - If File1 is as below pqrstuvdt abcdefgh then I want to add the above header into it ,So that File1 becomes as below ABC,17 pqrstuvdt abcdefgh ... (9 Replies)
Discussion started by: spari2
9 Replies

6. Shell Programming and Scripting

Identify log files based on time stamp,zip and then copy..HELP

Hi All, PFB is a requirement. I am new to shell scripting. So plz help. It would be highly appreciated. 1. choose all the log files based on a particular date (files location is '/test/domain')--i.e,we should choose all the files that are modified on 29th November, neither 28th nor 30th 2.... (3 Replies)
Discussion started by: skdas_niladri
3 Replies

7. Shell Programming and Scripting

improve performance - replace $\| with $#@ and remove header and trailer records

Hi All, In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'. I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance? ... (1 Reply)
Discussion started by: HemaV
1 Replies

8. Shell Programming and Scripting

how to copy an alertlog file from a particular time stamp?

Hi, Can any one tell me how to copy an alertlog file from a particular time stamp using shell script? Thanks (3 Replies)
Discussion started by: suman_dba1
3 Replies

9. Shell Programming and Scripting

Merge text files while combining the multiple header/trailer records into one each.

Situation: Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to... (4 Replies)
Discussion started by: oordonez
4 Replies

10. Shell Programming and Scripting

Remove header(first line) and trailer(last line) in ANY given file

Hi, I need some help in removing the header (first line) and the trailer (last line) in a give file... The data file actually comes in EBCDIC format and I converted it into ASCII.. Now I need to strip off the first line and the last line.. I think we can use sed to do something like this:... (2 Replies)
Discussion started by: madhunk
2 Replies
Login or Register to Ask a Question