Process split files with same name by group


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Process split files with same name by group
# 1  
Old 03-21-2017
Process split files with same name by group

File examples

Code:
f17_mar_01_02_03_04_fsw1.xml
f17_mar_01_02_03_04_fsw2.xml
f17_mar_01_02_03_04_fsw3.xml

Code:
f17_feb_13_20_49_06_fsw1.xml
f17_feb_13_20_49_06_fsw2.xml
f17_feb_13_20_49_06_fsw3.xml

I have many xml files that are grouped with same file name, but are numbered from 1-to-many (fsw1..fsw2..fsw3 etc..). These example files are very short snippets of the thousands of grouped xmls’ I have.

Problem: I need to process these files by filename group in a loop (basically combining them into one file per group), but I cannot seem to restart the loop every time the filename changes and it processes all the files together into one file.

Question: I need to process each group separately (fsw1..fsw2..fsw3 ) and then the next filename (fsw1..fsw2..fsw3). Is there an easy conditional way to loop on filename pattern and restart loop every time the filename changes?

Thank You

Last edited by Scrutinizer; 03-21-2017 at 01:46 PM.. Reason: code tags
# 2  
Old 03-21-2017
Try:
Code:
for first in *_fsw1.xml
do
  group=${first%_*}
  echo "Processing group ${group}"
  for file in "${group}"_fsw*.xml
  do
    echo "Processing ${file}"
  done 
done

To combine into a file, try:
Code:
for first in *_fsw1.xml
do
  group=${first%_*}
  echo "Processing group ${group}"
  for file in "${group}"_fsw*.xml
  do
    cat "${file}"
  done > "${group}.out"
done

If there are not so many files in a group that they would exceed line length limitations, then you could also try this:
Code:
for first in *_fsw1.xml
do
  group=${first%_*}
  echo "Processing group ${group}"
  cat "${group}"_fsw*.xml > "${group}.out"
done


Last edited by Scrutinizer; 03-22-2017 at 12:47 AM..
# 3  
Old 03-21-2017
Thank you! So far the very first example you gave me seems to work much better than what I had. I just put my "combine" code within your code and it separates the groups. I will do some more thorough testing and let you know if I run into an issue.

Thanks again!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Replace Stub Values In One Group Of Files With Actual Values From Another Group Of Files

I have two directories of files (new-config-files and old-config-files): new-config-files/this-db/config.inc.php new-config-files/that-db/config.inc.php new-config-files/old-db/config.inc.php new-config-files/new-db/config.inc.php new-config-files/random-database/config.inc.php etc. ... (4 Replies)
Discussion started by: spacegoose
4 Replies

2. UNIX for Beginners Questions & Answers

Split and Rename Split Files

Hello, I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number. What I have tried is the below command with 2 digit numeric value split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02) How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies

3. Shell Programming and Scripting

ksh : split hex number group

Hi, sry for poor english I have a group of hex number as : 4D40:4D42 I want so split this group in a list as : 4D40,4D41,4D42 i don't know how i can do this in ksh Thanks (5 Replies)
Discussion started by: jocazh
5 Replies

4. Programming

Create a group of process

Hi all ! :) What I want? 1. All child process must be in the same group of process. Parent is a leader of the group. How to do this? I would be greatfull of some example of code, I read about setsid but I can't even start... My code so far: #include <stdio.h> #include <stdlib.h>... (2 Replies)
Discussion started by: mattdj
2 Replies

5. UNIX for Dummies Questions & Answers

Split binary file every occurrence of a group of characters

Hello I am new to scripts, codes, bash, terminal, etc. I apologize this my be very scattered because I frankly don't have any idea where to begin and I have had trouble sleeping lately. I have several 2GB files I wish to split. This Code 00 00 01 BA ** ** ** ** ** ** ** ** C3 F8 00 00 01 BB 00... (17 Replies)
Discussion started by: PatrickE
17 Replies

6. Shell Programming and Scripting

Split, Search and Reformat by Data Group

Hi, I am writing just to share my appreciation for help I have received from this site in the past. In a previous post Split File by Data Group I received a lot of help with a troublesome awk script to reformat some complicated data blocks. What I learned really came in hand recently when I... (1 Reply)
Discussion started by: mkastin
1 Replies

7. Shell Programming and Scripting

split file based on group count

Hi, can some one please help me to split the file based on groups. like in the below scenario x indicates the begining of the group and the file should be split each with 2 groups below there are 10 groups it should create 5 files. could you please help? (4 Replies)
Discussion started by: hitmansilentass
4 Replies

8. Shell Programming and Scripting

Split file by data group

Hi all, I'm having a little trouble solving a file split I need to get done. I have the following data: 1. Light 1A. Light Soft texture: it's soft color: the color value is that of something light vital statistics: srm: 23 og: 1.035 sp: 1.065 comment: this is nice if you like... (8 Replies)
Discussion started by: mkastin
8 Replies

9. Shell Programming and Scripting

Split these into many ...(/etc/group)!!

Guys Following input line is from /etc/group file.As we know last entry in a line of /etc/group is userlist (all the users belonging to that group). I need to splilt this one line into 3 lines as shown below (3 because userlist has 3 names in it). Input: lp:!:11:root,lp,printq ... (13 Replies)
Discussion started by: ak835
13 Replies

10. Shell Programming and Scripting

split process files in parallel and rejoin

Hi I need to split a huge file into multiple smaller files using split command. After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle . Check the status of each of these sqlloaders and then after sucessfull... (6 Replies)
Discussion started by: xiamin
6 Replies
Login or Register to Ask a Question