Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Shell script to identify the number of files and to append data

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to identify the number of files and to append data
# 1  
Old 05-24-2010
Shell script to identify the number of files and to append data
Hi

I am having a question where I have to
1) Identify the number of files in a directory with a specific format
and if the count is >1 we need to concatenate those two files into one file and remember that in the second file the header should not be copied. it should be form first file.
Could you please give some tips on this.

sample :

input files
Code:
abc123

abc234

def123



output files
Code:
abc

def



for example

file: abc123

Code:
col1|col2|col3

1|2|3

file: abc234
Code:
col1|col2|col3

4|5|6

output

file: abc
Code:
col1|col2|col3

1|2|3

4|5|6

thanks
Sunil
Last edited by vgersh99; 05-25-2010 at 07:24 PM.. Reason: code tags, please!
# 2  
Old 05-25-2010
You didn't mention what is that specific format (for filename)?
Anyway,

say for abc files, you can do something like:

Code:
i=0
for file in abc*
do
  [ $i -eq 0 ] && cat $file > final_abc || tail  +2 $file >> final_abc
  i=$(( $i + 1 ))
done

# 3  
Old 05-25-2010
the file formats are as below
Code:
20070101_20070630_20100524112157_DW_EXPORT_PAYMENTS.txt
20060101_20060630_20100524112157_DW_EXPORT_PAYMENTS.txt

20070101_20070630_20100524112157_DW_EXPORT_PRODUCTS.txt

the first two files should be written to asingle output file called DW_EXPORT_PAYMENTS.txt
and the last file will be renamed to DW_EXPORT_PRODUCTS.txt

"If I find that there is more than one file then only the concatenation should take place else simple rename should occur"

Hope I am clear
Last edited by vgersh99; 05-25-2010 at 07:26 PM.. Reason: code tags, please!
# 4  
Old 05-26-2010
try something like..

Code:
for pattern in $(ls -1 | awk -F "[_.]" '{print $6}'| sort -u)
do
	i=0
	for file in *${pattern}.txt
	do
		[ $i -eq 0 ] && cat $file > ${pattern}.final || tail  +2 $file >> ${pattern}.final
		i=$(( $i + 1 ))
	done
done

# 5  
Old 05-26-2010
Try:
Code:
awk 'FILENAME~/[0-9]{3}$/ {
     p=substr(FILENAME,1,3)
     if ( o == p  && $1 ~ /col1|col2|col3/ ) {getline}
     print >p
     o=p }' *

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Append data by looking up 2 tables for multiple files

I want to lookup values from two different tables based on common columns and append. The trick is the column to be looked up is not fixed and varies , so it has to be detected from the header. How can I achieve this at once, for multiple data files, but lookup tables fixed. The two lookup... (5 Replies)
Discussion started by: ritakadm
5 Replies

2. Shell Programming and Scripting

Standardization of input source data files using shell script

Hi there, I'm a newbie in unix and am fishing for options related to how raw input data files are handled. The scenario, as I'm sure y'all must be very familiar with, is this : we receive upwards of 50 data files in ASCII format from various source systems - now each file has its own structure... (3 Replies)
Discussion started by: Prat Khos
3 Replies

3. Shell Programming and Scripting

Read multiple files, parse data and append to a file

Hi..Can anyone suggest a simple way of achieving this. I have several files which ends with extension .vcf . I will give example with two files In the below files, we are interested in File 1: 38 107 C 3 T 6 C/T 38 241 C 4 T 5 C/T 38 247 T 4 C 5 T/C 38 259 T 3 C 6 T/C... (8 Replies)
Discussion started by: empyrean
8 Replies

4. Shell Programming and Scripting

Bash script to find the number of files and identify which ones are 0 bytes.

I am writing a bash script to find out all the files in a directory which are empty. I am running into multiple issues. I will really appreciate if someone can please help me. #!/bin/bash DATE=$(date +%m%d%y) TIME=$(date +%H%M) DIR="/home/statsetl/input/civil/test" ... (1 Reply)
Discussion started by: monasharma13
1 Replies

5. Shell Programming and Scripting

Shell script for reading number of files on server

Hi, I have requirement: of reading number of files on perticular server for current date, in perticular folder and send out emails to business folks. I'm not unix person; but i can use this script in another tool to get the desired results. I appriciate your help. Thanks. (8 Replies)
Discussion started by: premaboodi
8 Replies

6. UNIX for Dummies Questions & Answers

Count Number Of lines in text files and append values to beginning of file

Hello, I have 50 text files in a directory called "AllFiles" I want to make a program that will go inside of the "AllFiles" Directory and count the number of lines in each individual text file. Then, the program will calculate how many more lines there are over 400 in each text file and... (7 Replies)
Discussion started by: motoxeryz125
7 Replies

7. Shell Programming and Scripting

Shell Script to append files

I have several content (text) files in a folder called "content" I have several google ads (text) files in a folder called "google_ads" Example: /content /java websphere.txt android.txt /microsoft framework.txt /c_sharp linq.txt ... (4 Replies)
Discussion started by: rac.mike
4 Replies

8. Shell Programming and Scripting

Shell Script to identify the line number containing a particular "string"

Hi, I have a log file, where i am required to identify the line number, where a particular string/line appears in the log file. And then copy 200 lines above that line number to a new file. Can someone provide pointers on how to write this script or what command to be used ? Any... (2 Replies)
Discussion started by: kk2202
2 Replies

9. Shell Programming and Scripting

Compare semicolon seperated data in 2 files using shell script

hello members, I have some data ( seperated by semicolon ) with close to 240 rows in a text file temp1. temp2.txt stores 204 rows of data ( seperated by semicolon ). I want to : Sort the data in both files by field1.i.e first data field in every row. compare the data in both files and print... (6 Replies)
Discussion started by: novice82
6 Replies

10. Shell Programming and Scripting

shell script for log files data!

Hi Team, please write a shell script. It is for veritas netbackup logs. The result we currently have is a single file for each day's backups. we will keep the files in the directory and the file names are like below mentioned. example :/opt/openv/netbackup/reports/Daily/NB_success*. The No... (6 Replies)
Discussion started by: rvrao77
6 Replies
Login or Register to Ask a Question