Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

File processing one after another file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File processing one after another file
# 1  
Old 06-24-2014
File processing one after another file
Hi,

I have requirement to process my script to process one after another

Example

directory /opt/ftplogs it contains 10 t0 100 files like below

Code:
 adn_DF9D_20140618_0001.log
 adn_DF9D_20140618_0002.log
 adn_DF9D_20140618_0003.log
 adn_DF9D_20140618_0004.log
 adn_DF9D_20140618_0005.log
 adn_DF9D_20140618_0006.log
 upto
  
 adn_DF9D_20140618_0100.log

I need to process first adn_DF9D_20140618_0001.log after finishing process needs to be go anthor file adn_DF9D_20140618_0002.log.

I tried below
Code:
 source = /opt/ftplogs
  
 target =/opt/target
  
 cp -p $source $target
  
 while read -r line
 do
 if (condition -le 390)
 then
 echo {line} >>/opt/lesser.log
 else
 echo {line} >>/opt/greater.log
 fi
 done <$file
  
 mv $source /opt/backup

The above code is processing at a time all file. but I need to be process from one file copy then process then move next take 2nd file same continue with upto 100 files

Please suggest to me how can I write a shell script on this scenario. will appreciate your suggestions.
Last edited by Scrutinizer; 06-24-2014 at 08:52 AM.. Reason: CODE tags
# 2  
Old 06-24-2014
Do you want to process all the files in the directory? You could do something like:
Code:
for file in $(ls $source)
do
    process $file
done

Your example code doesn't seem to be complete though - there may be better ways to do it, depending on exactly what you want to do with the files, and whether it's all of them or a subset.
# 3  
Old 06-24-2014
Try:

Code:
cd "$source"
for logfile in adn_*.log
do
  while read -r 
  do
    # process content of individual logfile
  done < "$logfile"
done

Last edited by Scrutinizer; 06-25-2014 at 04:37 AM..
# 4  
Old 06-25-2014
Thanks for your suggestion the above script is for all file processing at a time.

My requirement is
First file i.e abc.txt copy then process and then move after take second file def.txt copy then process and then move

Please suggest to me
# 5  
Old 06-25-2014
You could do the copy from the source directory to the target directory within the for loop. If that fails you could break the loop or use exit 1 to leave the script. If the copying is succesful, then process that file in de target directory by prepending the target directory to the file name: "$target/$logfile" .

So, something like:
Code:
cd "$source"
for logfile in adn_*.log
do
  cp -p "$logfile" "$target" || exit 1
  while read -r 
  do
    # process content of individual logfile
  done < "$target/$logfile"
done

# 6  
Old 06-25-2014
Is it possible to push to loop one after another from bulk list of file ?

Please suggest to me how ?
# 7  
Old 06-25-2014
What do you mean with "bulk list of file" ? Can you give an example?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with file processing

I have a fixed width file coming from source system. The total characters on each record is 786. I am getting records in the file with less than and greater than this number and the process is failing bcs of this length of the record Is there a way to spit out the bad records (length <>786)... (5 Replies)
Discussion started by: dsravanam
5 Replies

2. UNIX for Dummies Questions & Answers

awk - Rename output file, after processing, same as input file

I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this? ... (12 Replies)
Discussion started by: High-T
12 Replies

3. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

4. Shell Programming and Scripting

Recursive file processing from a path and printing output in a file

Hi All, The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the std... (1 Reply)
Discussion started by: Optimus81
1 Replies

5. Shell Programming and Scripting

processing an input file and then the output file

Hi. I am new to scripting and could really do with some advice on the best way to put a script together. Here is the scenario I am working to; - i will get files via ftp to a tmp directory on the server - all files will have a unique file name but with the same extension (.USM) - for each... (5 Replies)
Discussion started by: yabai
5 Replies

6. Shell Programming and Scripting

How to processing the log file within certain dates based on the file name

Hi I am working on the script parsing specific message "TEST" from multiple file. The log file name looks like: N3.2009-11-26-03-05-02.console.log.tar.gz N4.2009-11-29-00-25-03.console.log.tar.gz N6.2009-12-01-10-05-02.console.log.tar.gz I am using the following command: zgrep -a --text... (1 Reply)
Discussion started by: shyork2001
1 Replies

7. Shell Programming and Scripting

File processing

Hello, I am trying to find a solution which could be simple rather than coding unnecessary code. Scenario I have is :- I have a line sequential text file and while reading the file I need to call some kind of program which does data transfer, and read next so on... The issue I have is I have... (4 Replies)
Discussion started by: sentak
4 Replies

8. Shell Programming and Scripting

how to change the current file processing to some other random file in awk ?

Hello, say suppose i am processing an file emp.dat the field of which are deptno empno empname etc now say suppose i want to change the file to emp.lst then how can i do it? Here i what i attempted but in vain BEGIN{ system("sort emp.dat > emp.lst") FILENAME="emp.lst" } { print... (2 Replies)
Discussion started by: salman4u
2 Replies

9. Shell Programming and Scripting

Checking for a control file before processing a data file

Hi All, I am very new to Shell scripting... I got a requirement. I will have few text files(data files) in a particular directory. they will be with .txt extension. With same name, but with a different extension control files also will be there. For example, Sample_20081001.txt is the data... (4 Replies)
Discussion started by: purna.cherukuri
4 Replies

10. Shell Programming and Scripting

Have a shell script check for a file to exist before processing another file

I have a shell script that runs all the time looking for a certain type of file and then it processes the file through a series of other scripts. The script is watching a directory that has files uploaded to it via SFTP. It already checks the size of the file to make sure that it is not still... (3 Replies)
Discussion started by: heprox
3 Replies
Login or Register to Ask a Question