Grouping files into tars


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grouping files into tars
# 1  
Old 03-26-2009
Grouping files into tars

Hi all,

I have a problem where i have several files in a directory which I SCP from a server to my local machine and i would like to periodically tar/gzip them based on their naming convention.

Here is the scenario:

I SCP files (which all end with the same ending) periodically across to a '/stored' location:

stat_t_123
stat_p_123
stat_c_123

stat_t_999
stat_p_999
stat_c_999

As you can see they all end in '123' or '999' (as an example).

So what I'm looking for is a way to query the '/stored' directory and look for all files ending with '123' or '999' and then grouping them together into a single tar/gzip'd file.

the tarred files will then remain in the '/stored' location.
So you would end up with:

/stored/files123.tar.gz
/stored/files999.tar.gz

I know i can get all files which are not tars or gzips like this:

`ls /stored | grep -v ".tar" | grep -v ".tar.gz"`

but how would i achieve the grouping? SmilieSmilie

any helps guys?
# 2  
Old 03-26-2009
Quote:
Originally Posted by muay_tb
Hi all,
So what I'm looking for is a way to query the '/stored' directory and look for all files ending with '123' or '999' and then grouping them together into a single tar/gzip'd file.
...

`ls /stored | grep -v ".tar" | grep -v ".tar.gz"`

but how would i achieve the grouping? SmilieSmilie

any helps guys?
Hi,
one approach could be

Code:
tar czvf files123.tar.gz *123

And about Your listing command, the first -v .tar will also exclude tar.gz so You don't need that -v tar.gz

Best regards,
Lakris
# 3  
Old 03-26-2009
Code:
for i in `ls -1 stat_?_???|cut -d"_" -f3|sort -u` ; do
     tar czvf /stored/files${i}.tar.gz stat_?_$i
done

# 4  
Old 03-27-2009
Thanks guys, i will give both a go.

Once again many thanks! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grouping files on pattern

I have this Requirement where i have to group the files, I have a folder say "temp" where many files resides...files are like this; 010020001_S-ABC-Sort-DEFAW_YYYYMMDD_HHMMSS.txt 010020004_S-PQR-Sort-DRTON_YYYYMMDD_HHMMSS.txt 010020009_S-JKL-Sort_MNOLO_YYYYMMDD_HHMMSS.txt... (8 Replies)
Discussion started by: gnnsprapa
8 Replies

2. Shell Programming and Scripting

Grouping and Calculating

Hi All, I want to read the input file and store the output in the Output file. I pasted the sample Input and Output file below. Help me with this. Input file ================================= ITEM1 AAAAA 1 ITEM1 BBBBB 1 ITEM1 CCCCC 1 ITEM2 AAAAA 5 ITEM2 CCCCC 4... (1 Reply)
Discussion started by: humaemo
1 Replies

3. Shell Programming and Scripting

Name grouping

awk 'FNR==NR {a; next} $NF in a' genes.txt refseq_exons.txt > output.txt I can not figure out how to group the same name in $4 together. Basically, all the SKI together in separate rows and all the TGFB2. Thank you :). chr1 2160133 2161174 SKI chr1 218518675 218520389 TGFB2... (1 Reply)
Discussion started by: cmccabe
1 Replies

4. Shell Programming and Scripting

Grouping

Hi all, I am using following command: perl program.pl input.txt output.txt CUTOFF 3 > groups_3.txt containing program.pl, two files (input.txt, output.txt) and getting output in groups_3.txt: But, I wish to have 30 files corresponding to each CUTOFF ranging from 0 to 30 using the same... (1 Reply)
Discussion started by: bioinfo
1 Replies

5. Shell Programming and Scripting

grouping log files based on counter

I have my log file as below 00:18:02 - Nothing normal; Garbage Collection kicked off & running from last 3 min... 00:19:02 - Nothing normal; Garbage Collection kicked off & running from last 4 min... 00:19:02 - Nothing normal; Garbage Collection kicked off & running from last 4 min...... (11 Replies)
Discussion started by: manas_ranjan
11 Replies

6. Shell Programming and Scripting

Grouping files according to certain fields in their name

I have a list of fils stored insortedLst, and want to select certain fields to group specific files together: Example of the files would be as below: n02-z30-dsr65-ndelt0.25-varp0.002-16x12drw-run1.log n02-z30-dsr65-ndelt0.25-varp0.002-16x12drw-run2.log... (2 Replies)
Discussion started by: kristinu
2 Replies

7. Shell Programming and Scripting

combine 3 files by grouping

I have a file, which is really large but i shortened it: A3059GVS 1 A 01 Plate_1 40 25.37016 14.6298 A3059GVS 2 A 01 Plate_2 40 26.642002 13.3583 A3059GVS 3 A 02 Plate_1 40 25.381462 ... (4 Replies)
Discussion started by: mykey242
4 Replies

8. UNIX for Advanced & Expert Users

Size of a tarball without untarring - Catch parent tar ball has sub tars

hi, I am in a weird situation. I have a parent tarball which contains 2 sub tarballs. The structure is such : Parent.tar.gz ---- > child1.tar.gz and child2.tar.gz I need to get the size of the parent tarball without untaring it I know that the command is gunzip -c parent.tar.gz | wc -c ... (1 Reply)
Discussion started by: mnanavati
1 Replies

9. Shell Programming and Scripting

parsing file names and then grouping similar files

Hello Friends, I have .tar files which exists under different directories after the below code is run: find . -name "*" -type f -print | grep .tar > tmp.txt cat tmp.txt ./dir1/subdir1/subdir2/database-db1_28112009.tar ./dir2/subdir3/database-db2_28112009.tar... (2 Replies)
Discussion started by: EAGL€
2 Replies

10. UNIX for Dummies Questions & Answers

grouping and sorting??

how would I write a command line that creates a new file named stuff.txt in the current working directory which contains the number of directories in the current working directory, followed by the number of empty files in the current working directory, followed by the name of the working directory? (3 Replies)
Discussion started by: jorogon0099
3 Replies
Login or Register to Ask a Question