Help splitting files greater than 250mb


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Help splitting files greater than 250mb
# 8  
Old 03-05-2013
Did you try the code I suggested in message #5 in this thread. If it will work for you, it will eliminate having to post-process all of your big files and it will guarantee that all of your intermediate files are text files. (That may be important unless all that you do with the intermediate files is move them someplace else and recombine them into the original big files.)
# 9  
Old 03-05-2013
Miri,

we are on AIX Version 5.3.
I am very close using yor script just not able to append the .txt properly or use the number. -d is not recognized.

---------- Post updated at 01:24 PM ---------- Previous update was at 01:23 PM ----------

Don,

We take the files & move them into FileNet for archiving. I did attempt using your script but it created multiple files of 0 bytes. Not sure it was doing what I wanted. Thank you though.
# 10  
Old 03-05-2013
Quote:
Originally Posted by tomj5141
Miri,

we are on AIX Version 5.3.
I am very close using yor script just not able to append the .txt properly or use the number. -d is not recognized.

---------- Post updated at 01:24 PM ---------- Previous update was at 01:23 PM ----------

Don,

We take the files & move them into FileNet for archiving. I did attempt using your script but it created multiple files of 0 bytes. Not sure it was doing what I wanted. Thank you though.
I copied the copied and pasted the script from message #5 back into the directory where I tested it, changed the input file pathname to the pathname of a test file I created and got the results I had gotten before.

Are you by any chance working on a Solaris system? If so, please try again using /usr/xpg4/bin/awk or nawk instead of awk. If you aren't using a Solaris system, what type of system are you using?
# 11  
Old 03-05-2013
You can use this logic to do the renaming from lexicographic suffix to numeric:
Code:
#!/bin/bash 
for file in P8.*txt ; do
    size=$(ls -l $file | awk '{print $5}')              #get the size of file
    if [[ $size -gt $((250*1024*1024)) ]] ; then        #file bigger than 250M
       split -b 250M -a1 $file ${file%.txt}_            #do the spliting, -a1: use only one digit
       suff=0  
       for i in ${file%.txt}_[a-z] ; do                 #change _[a-z] to _[0-9] and add .txt suffix:
           if [[ $suff = 0 ]] ; then  
             newname=${i%_*}.txt 
           else 
             newname=${i%_*}_${suff}.txt 
           fi
          ((suff++))
      
          echo mv $i  $newname                           #take out 'echo' if it works as you want
       done 
   fi
done

You can simplify by omitting the inner if ; else ; fi clause if you don't care that the first split file will have a number in it.
You may need to quote variables if the filenames have spaces in them.

Last edited by mirni; 03-05-2013 at 02:56 PM..
# 12  
Old 03-05-2013
ksh[7]: suff++: 0403-053 Expression is not complete; more tokens expected. Not sure what is missing but I guess I can just leave it end with the letter then apply the filetype later. I do have a question. When we have a split -b 250m why do files split larger than 250mb???? :

-rw-rw-r-- 1 dfelp dfelpdev 262144000 Mar 05 13:52 P8.PECO_Energy.Residential.PA.12.2010_.txta

-rw-rw-r-- 1 dfelp dfelpdev 184936511 Mar 05 13:52 P8.PECO_Energy.Residential.PA.12.2010_.txtb

I just decreased the size to 200m to compensate but was wondering.
# 13  
Old 03-05-2013
They don't. 250 MB = 262144000 bytes: 262144000/(1024*1024)=250
If you want 250 million bytes instead, use "MB" instead of "M", or just the number. Look at man split for details.
You can look at the file size expressed in k,M etc bytes by doing
Code:
ls -lh

Quote:
ksh[7]: suff++:
Seems like you are using ksh; do you have the shebang line as I wrote it?:
#!/bin/bash

In any case, you can try to change
Code:
((suff++))

to
Code:
let suff += 1

# 14  
Old 03-06-2013
Seems as though I should have taken some more technical classes in college & less business considering my career. I am currently learning Unix, .Net (VB & C#), & SQL on the fly as well as the applications. These forums have been very useful. Thanks for your help! My solution is, I use the awk script as I had before & just added your code to the .sh file its in with a few adjustments:

Code:
#!/bin/bash
for file in P8.* ; do 
  size=$(ls -l $file | awk '{print $5}')              #get the size of file
  if [[ $size -gt $((250*1024*1024)) ]] ; then        #file bigger than 250M
      split -b 230m -a1 $file ${file%}.               #do the spliting, numeric suffix, -a1: use only one digit
  fi
done
     for file in P8.* ; do 
  size=$(ls -l $file | awk '{print $5}')              
  if [[ $size -gt $((250*1024*1024)) ]] ; then     
      rm $file 
  fi
done

These files get moved from the server using webmethods so I will let our webmethods developer change them to .txt.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. Shell Programming and Scripting

Please help list/find files greater 1G move to different directory

I have have 6 empty directory below. I would like write bash scipt if any files less "1000000000" bytes then move to "/export/home/mytmp/final" folder first and any files greater than "1000000000" bytes then move to final1, final2, final3, final4, final4, final5 and that depend see how many files,... (6 Replies)
Discussion started by: dotran
6 Replies

3. Shell Programming and Scripting

Need to delete the log files when the disk used% greater than 85% using df -k

Hi, I am new to Shell scripts. I have an urgent requirement to find the disk space using "df -k". from that output,I need to check the used% whether greater than 85%. if it is greater than 85% then need to delete my log files. It is very urgent please some one help me. Thanks in Advance... (2 Replies)
Discussion started by: Anandbarnabas
2 Replies

4. How to Post in the The UNIX and Linux Forums

Very Urgent ---Need to delete the log files when the disk used% greater than 85% using df -k*

Hi, I am new to Shell scripts. I have an urgent requirement to find the disk space using "df -k". from that output,I need to check the used% whether greater than 85%. if it is greater than 85% then need to delete my log files. It is very urgent please some one help me. Thanks in Advance... (1 Reply)
Discussion started by: Anandbarnabas
1 Replies

5. Shell Programming and Scripting

Find files greater than a particular date in filename.

I need a unix command which will find all the files greater that a particular date in the file name. say for example I have files like(filenaming cov : filename.YYDDMMSSSS.txt) abc.201206015423.txt abc.201207013456.txt abc.201202011234.txt abc.201201024321.txt efg.201202011234.txt... (11 Replies)
Discussion started by: lijjumathew
11 Replies

6. UNIX for Dummies Questions & Answers

List of Files which are Greater then a specific date

A newbie question... I need to get a list of the Files and folders which are greater then a specific date. I want write the output to a Text file. What I know ls -lrt gives me list of all the files ordered by date. Also ls > fileName will write the results to a text file. Please help (6 Replies)
Discussion started by: rkaif
6 Replies

7. Shell Programming and Scripting

Trying to find files equal to and greater than

Hi Guys and Gals, I'm having some difficulty putting this check into a shell script. I would like to search a particular directory for a number of files. The logic I have is pretty simple: Find file named *.txt that are newer than <this file> and count them If the number of files is equal to... (4 Replies)
Discussion started by: bbbngowc
4 Replies

8. Shell Programming and Scripting

Important finding --- find files greater than 1 MB

as we can find file greater than 1 MB with find command as: find /dir -name '*' -size +1M find /dir/* -name '*' -size +1M but wats its doing is , its finding files only in current directory not in sub-directories. i want files from sub-directories too. Please help... Thanx in... (3 Replies)
Discussion started by: manoj_dahiya22
3 Replies

9. UNIX for Advanced & Expert Users

Problem creating files greater than 2GB

With the C code I am able to create files greater than 2GB if I use the 64 bit compile option -D_FILE_OFFSET_BITS=64. There I am using the function fprintf to write into the file. But when I use C++ and ofstream the file is getting truncated when the size grows beyond 2GB. Is there any special... (1 Reply)
Discussion started by: bobbyjohnz
1 Replies

10. Shell Programming and Scripting

Deleting specific files greater then 90 days

I have a directory that contains a number of history files for a process. Every file starts with the string "EVACK". FOr example EVACK0101.000001 EVACK0102.095940 EVACKmmdd.hhmiss I want to erase in the specific directory ONLY(no sub-directories) all EVACK files older then 90 days. I... (2 Replies)
Discussion started by: beilstwh
2 Replies
Login or Register to Ask a Question