"Help finding/adding up file size..."

Post #302166804 by Stove13 on Wednesday 13th of February 2008 12:14:19 AM

Help finding/adding up file size...

Ok, I am new to UNIX and Shell scripting and I am trying to do the following using a korn shell script:
I have a large data file that is loaded into a database, this data file is split into parts to make it more manageable based on a size parameter. The file contains many account records, the account number shows up in the 4th field if that is of any help. To split it, I do a grep to a temporary file, when a new account record set begins, there is an identifying record number in the first field. So, the grep looks for that and sends those to a temp file, then I count the #of accounts in the temp file and divide the # of accounts by the number of splits to determine how many accounts will be in each file. Then my script continues with its loading process... my problem is, there is a maxsize for the splits, sometimes an account can have thousands of records, resulting in it being very large for that one account. Since I am only splitting by the # of accounts it does not know that a large account may possibly send it over the limit and cause it to fail. Somehow, I need to be able to add up the size of the account records for each account and use that to determine if I need to create an extra split (if it hits the maxsize). I'm not sure if maybe I should try to get an "average record size" and somehow use that or if there is something right in front of me like wc that will work, any help is greatly appreciated. If this helps, here is a piece of the code:

datafilesz=`ls -l $datafile | awk '{print $5}'`
recsinfile=`grep -c '^' $datafile`

datasplits=$(( datafilesz / $maxsize ))
datafiles=$(( $datasplits + 1 ))

if (( $datafiles > 1 )) then
grep -n '^01' $datafile > /tmp/datarecs.$$
accounts=`grep -c '^' /tmp/datarecs.$$`

recsperfile=$(( $accounts / $datafiles ))
 
Test Your Knowledge in Computers #963
Difficulty: Medium
Conficker, also known as Downup, Downadup and Kido, was a computer worm targeting the Microsoft Windows operating system that was first detected in November 2008.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding EOL char is there or not in a big size file

Hi, I am using ksh. I have to find wether data file has EOL or not. as per my knowledge we can easily find by checking each character. But this is a tedious job as per my requirement because my data file size is very big . It may be in 25-30 MB. So please advice me how i can check wether... (4 Replies)
Discussion started by: HariRaju
4 Replies

2. Shell Programming and Scripting

finding duplicate files by size and finding pattern matching and its count

Hi, I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern. Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies

3. UNIX for Dummies Questions & Answers

Need help in finding Folder Size

Hi, I would like to find the size of a folder. When I run the command du -k It is going through all the sub-folder and files and taking really much time. Is there any command to get the complete directory size without showing the sub-folder and file size. Appreciate your response. ... (3 Replies)
Discussion started by: TonySolarisAdmi
3 Replies

4. Solaris

Finding out the memory size via the iLOM

I would like to know if it is possible to find out how much memory is in a machine from the iLOM prompt on an x86 box? I have retrieved the MAC address details from the iLOM promt before using show /SYS/MB/NETx and wondering if I can do the same for the Memory although I can't seem to find anything... (4 Replies)
Discussion started by: Chains
4 Replies

5. Shell Programming and Scripting

adding new line after finding specific text

hello i need some help here are the contents of my file. test.txt this is filename 1.mp3 http://www.url.com/filenamehashed filename 2.mp3 http://www.url.com/fileamehashed something_else.zip http://www.url.com/filenamehashed so this file has 100 of these lines filename url I would... (9 Replies)
Discussion started by: mscice
9 Replies

6. UNIX for Dummies Questions & Answers

Finding size of all directories

Alright so I've tried a couple different things that at first glance, looked like they worked. find . -maxdepth 5 -type d -daystart -mtime 1 | xargs du -h Which seems to ignore the previous commands such as depth and modified time. find .. -maxdepth 2 -type d -daystart -ctime 1 | xargs... (8 Replies)
Discussion started by: Aussiemick
8 Replies

7. Shell Programming and Scripting

finding max size

Hi I have a list of 2000 records with multiple entries and I want to get the max size for each entry ABC 1 ABC 2 ABC 3 ABC 4 DEF 1 DEF 2 DEF 2 DEF 2 DEF 2 ... (9 Replies)
Discussion started by: Diya123
9 Replies

8. Shell Programming and Scripting

Help in finding & comparing avg file size

I have some files in a directory with two different extensions which get created everyday. Can you please help me out in getting the average file size for both these extensions and checking it with the last two file sizes of the same file extension ? To be more clear.. Lets say I have 10 files... (1 Reply)
Discussion started by: kiran1112
1 Replies

9. Shell Programming and Scripting

Finding size of files with spaces in their file names

I am running a UNIX script to get unused files and their sizes from the server. The issue is arising due to the spaces present in the filename/folder names.Due to this the du –k command doesn’t work properly.But I need to calculate the size of all files including the ones which have spaces in them.... (4 Replies)
Discussion started by: INNSAV1
4 Replies

10. Debian

Finding out size of sub-directory

Hi, Is there a way to find out the size of a sub-directory? Eg subidrectory in main directory /data, like this : /data/solr-5.3.1. When I do a df -h /data/solr-5.3.1, it still gives the size of /data: root@L28condor:/data/solr-5.3.1# df -h . Filesystem Size Used... (1 Reply)
Discussion started by: anaigini45
1 Replies
ARRAY_PAD(3)								 1							      ARRAY_PAD(3)

array_pad - Pad array to the specified length with a value

SYNOPSIS
array array_pad (array $array, int $size, mixed $value) DESCRIPTION
array_pad(3) returns a copy of the $array padded to size specified by $size with value $value. If $size is positive then the array is padded on the right, if it's negative then on the left. If the absolute value of $size is less than or equal to the length of the $array then no padding takes place. It is possible to add at most 1048576 elements at a time. PARAMETERS
o $array - Initial array of values to pad. o $size - New size of the array. o $value - Value to pad if $array is less than $size. RETURN VALUES
Returns a copy of the $array padded to size specified by $size with value $value. If $size is positive then the array is padded on the right, if it's negative then on the left. If the absolute value of $size is less than or equal to the length of the $array then no padding takes place. EXAMPLES
Example #1 array_pad(3) example <?php $input = array(12, 10, 9); $result = array_pad($input, 5, 0); // result is array(12, 10, 9, 0, 0) $result = array_pad($input, -7, -1); // result is array(-1, -1, -1, -1, 12, 10, 9) $result = array_pad($input, 2, "noop"); // not padded ?> SEE ALSO
array_fill(3), range(3). PHP Documentation Group ARRAY_PAD(3)

Featured Tech Videos

All times are GMT -4. The time now is 03:07 PM.
Unix & Linux Forums Content Copyright 1993-2019. All Rights Reserved.
Privacy Policy