Sponsored Content
Top Forums Shell Programming and Scripting Find and split the list of files with suffiz of seg** Post 302762777 by Satish Shettar on Tuesday 29th of January 2013 03:40:59 AM
Old 01-29-2013
Find and split the list of files with suffiz of seg**

Hi,.

I am writing a script to get the new files and split them.


Requirement
Find the new files under the path "/wload/scmp/app/data/OAS" (There are 5 sub folders).
Gunzip the files which are having .gz suffix.
Put the list of files in the filename in the format "fiels_to_split_2013012911300.txt where as the numeric part is date and time.
Perform the split command on the files to split each file containing 50000 lines.
The output of the split filename should have the same file with suffix seg** where ** can be either numbers or alphabets.


I got the code from Pikk45 and tried to add/modify as per the need.
Code:
find /wload/scmp/app/data/OAS -name "*.gz" -size +100000000 -type f -mtime -1 -exec gunzip {} \;
FILENAME=new_fiels_to_split$(date +"%Y%m%d%H%M").txt
find /wload/scmp/app/data/OAS -type f -mtime -1 > /wload/scmp/app/data/OAS/$FILENAME
FILES=/wload/scmp/app/data/OAS/$FILENAME
FIRST=1
FILE=1
SPL=50000
while read file; do
 split -l 50000 "$FILENAME" "$FILENAME".seg
 done < ${FILES}
FIRST=`expr $FIRST + 50000`
SPL=`expr $SPL + 50000`
FILE=`expr $FILE + 1`

for example
the source files

Code:
-rw-rw-r-- 1 scmpadm scmpuser 755861762 Jan 26 20:04 activity.log.20130125.gz
-rw-rw-r-- 1 scmpadm scmpuser 0 Jan 26 20:05 trigger.activity.log.20130125
-rw-rw-r-- 1 scmpadm scmpuser 913911244 Jan 27 20:24 activity.log.20130126.gz
-rw-rw-r-- 1 scmpadm scmpuser 0 Jan 27 20:25 trigger.activity.log.20130126
-rw-rw-r-- 1 scmpadm scmpuser 329351168 Jan 28 19:10 activity.log.20130127.gz

After gunzip
Code:
-rw-rw-r-- 1 scmpadm scmpuser 3012939863 Jan 28 19:10 activity.log.20130127

Files after split
Code:
-rw-rw---- 1 scmpadm scmpuser 49877675 Jan 29 07:04 activity.log.20130127.segaa
-rw-rw---- 1 scmpadm scmpuser 47552800 Jan 29 07:07 activity.log.20130127.segab
-rw-rw---- 1 scmpadm scmpuser 45867421 Jan 29 07:08 activity.log.20130127.segac
-rw-rw---- 1 scmpadm scmpuser 48985303 Jan 29 07:09 activity.log.20130127.segad
-rw-rw---- 1 scmpadm scmpuser 50846638 Jan 29 07:10 activity.log.20130127.segae
-rw-rw---- 1 scmpadm scmpuser 50946829 Jan 29 07:13 activity.log.20130127.segaf
-rw-rw---- 1 scmpadm scmpuser 50968144 Jan 29 07:14 activity.log.20130127.segag
-rw-rw---- 1 scmpadm scmpuser 51758156 Jan 29 07:18 activity.log.20130127.segah
-rw-rw---- 1 scmpadm scmpuser 52290589 Jan 29 07:23 activity.log.20130127.segai
-rw-rw---- 1 scmpadm scmpuser 19329024 Jan 29 07:26 activity.log.20130127.segaj


Last edited by Satish Shettar; 01-29-2013 at 07:20 AM.. Reason: Changed several icode tags to code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

list read only files using find

hi, how can i list read only files (for u,g,o) using find command? Thanks and Regards Vivek.s (1 Reply)
Discussion started by: vivekshankar
1 Replies

2. Shell Programming and Scripting

How to find the list of files

How to find the list of files of particular type present in Directory and sub-directory. (2 Replies)
Discussion started by: senthilk615
2 Replies

3. Shell Programming and Scripting

I need a script to find socials in files and output a list of those files

I am trying to find socail security numbers in files in (and under) a specific directory and output a list of the files where they are found... the format would be with no dashes just 9 numeric characters in a row. I have tried this: find /DirToLookIn -exec grep '\{9\}' /dev/null {} \; >>... (1 Reply)
Discussion started by: NewSolarisAdmin
1 Replies

4. UNIX for Dummies Questions & Answers

Find files and display only directory list containing those files

I have a directory (and many sub dirs beneath) on AIX system, containing thousands of file. I'm looking to get a list of all directory containing "*.pdf" file. I know basic syntax of find command, but it gives me list of all pdf files, which numbers in thousands. All I need to know is, which... (4 Replies)
Discussion started by: r7p
4 Replies

5. UNIX for Dummies Questions & Answers

Creating a List of Files With Find

Hi, I need to go through all the files on my system and build a list/output file with the paths of all files where the first two characters within the file match an expression. I know I can use something like find . | xargs cut -b1-2 or find . -exec cut -b1-2 {} \; to get the characters... (3 Replies)
Discussion started by: 008_
3 Replies

6. Shell Programming and Scripting

find list of files from a list and copy to a directory

I will be very grateful if someone can help me with bash shell script that does the following: I have a list of filenames: A01_155716 A05_155780 A07_155812 A09_155844 A11_155876 that are kept in different sub directories within my current directory. I want to find these files and copy... (3 Replies)
Discussion started by: manishabh
3 Replies

7. HP-UX

Find list of files that uses password.

Hi, I am looking for a command or script which should list out the files that uses database password 'June1232011'. I know where the files are located (path) .Since there are too many files exist in the directory, i want to find out the files that uses this password. It would be a... (10 Replies)
Discussion started by: Maddy123
10 Replies

8. Shell Programming and Scripting

Split list of files into an array and pass to function

There are two parts to this. In the first part I need to read a list of files from a directory and split it into 4 arrays. I have done that with the following code, # collect list of file names STATS_INPUT_FILENAMES=($(ls './'$SET'/'$FOLD'/'*'in.txt')) # get number of files... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

9. Shell Programming and Scripting

Find command to find a word from list of files

I need to find a word '% Retail by State' in the folder /usr/sas/reports/RetailSalesTaxallocation. When I tried like below, -bash-4.1$ cd /usr/sas/reports/RetailSalesTaxallocation -bash-4.1$ find ./ -name % Retail by State find: paths must precede expression: Retail Usage: find ... (10 Replies)
Discussion started by: Ram Kumar_BE
10 Replies

10. UNIX for Beginners Questions & Answers

Find for files within a list of subfolders

Helo Is there a better way to search within a list of subfolders : A_START_PATH="/data_1/data_2" # # dir2, dir3, dir6, ..... dir59 exists # A_LIST="$A_START_PATH/dir1 $A_START_PATH/dir4 $A_START_PATH/dir5" find "$A_LIST" -type f -name"*.txt" Now searching for all files in any subdirs... (2 Replies)
Discussion started by: jcdole
2 Replies
All times are GMT -4. The time now is 11:33 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy