02-13-2008
Help finding/adding up file size...
Ok, I am new to UNIX and Shell scripting and I am trying to do the following using a korn shell script:
I have a large data file that is loaded into a database, this data file is split into parts to make it more manageable based on a size parameter. The file contains many account records, the account number shows up in the 4th field if that is of any help. To split it, I do a grep to a temporary file, when a new account record set begins, there is an identifying record number in the first field. So, the grep looks for that and sends those to a temp file, then I count the #of accounts in the temp file and divide the # of accounts by the number of splits to determine how many accounts will be in each file. Then my script continues with its loading process... my problem is, there is a maxsize for the splits, sometimes an account can have thousands of records, resulting in it being very large for that one account. Since I am only splitting by the # of accounts it does not know that a large account may possibly send it over the limit and cause it to fail. Somehow, I need to be able to add up the size of the account records for each account and use that to determine if I need to create an extra split (if it hits the maxsize). I'm not sure if maybe I should try to get an "average record size" and somehow use that or if there is something right in front of me like wc that will work, any help is greatly appreciated. If this helps, here is a piece of the code:
datafilesz=`ls -l $datafile | awk '{print $5}'`
recsinfile=`grep -c '^' $datafile`
datasplits=$(( datafilesz / $maxsize ))
datafiles=$(( $datasplits + 1 ))
if (( $datafiles > 1 )) then
grep -n '^01' $datafile > /tmp/datarecs.$$
accounts=`grep -c '^' /tmp/datarecs.$$`
recsperfile=$(( $accounts / $datafiles ))
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I am using ksh.
I have to find wether data file has EOL or not. as per my knowledge we can easily find by checking each character. But this is a tedious job as per my requirement because my data file size is very big . It may be in 25-30 MB.
So please advice me how i can check wether... (4 Replies)
Discussion started by: HariRaju
4 Replies
2. Shell Programming and Scripting
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
3. UNIX for Dummies Questions & Answers
Hi,
I would like to find the size of a folder. When I run the command
du -k
It is going through all the sub-folder and files and taking really much time.
Is there any command to get the complete directory size without showing the sub-folder and file size.
Appreciate your response.
... (3 Replies)
Discussion started by: TonySolarisAdmi
3 Replies
4. Shell Programming and Scripting
hello i need some help here are the contents of my file.
test.txt
this is filename 1.mp3 http://www.url.com/filenamehashed
filename 2.mp3 http://www.url.com/fileamehashed
something_else.zip http://www.url.com/filenamehashed
so this file has 100 of these lines
filename url
I would... (9 Replies)
Discussion started by: mscice
9 Replies
5. UNIX for Dummies Questions & Answers
Alright so I've tried a couple different things that at first glance, looked like they worked.
find . -maxdepth 5 -type d -daystart -mtime 1 | xargs du -h Which seems to ignore the previous commands such as depth and modified time.
find .. -maxdepth 2 -type d -daystart -ctime 1 | xargs... (8 Replies)
Discussion started by: Aussiemick
8 Replies
6. Shell Programming and Scripting
Hi
I have a list of 2000 records with multiple entries and I want to get the max size for each entry
ABC 1
ABC 2
ABC 3
ABC 4
DEF 1
DEF 2
DEF 2
DEF 2
DEF 2
... (9 Replies)
Discussion started by: Diya123
9 Replies
7. UNIX for Advanced & Expert Users
I am trying to find RAM size in my HP-UNIX server. what command I should use for this?
I am using top command but not clear about below line from top o/p
Memory: 1517080K (471284K) real, 1877692K (751256K) virtual, 8078944K free Page# 1/6 (3 Replies)
Discussion started by: venkatababu
3 Replies
8. Shell Programming and Scripting
I have some files in a directory with two different extensions which get created everyday. Can you please help me out in getting the average file size for both these extensions and checking it with the last two file sizes of the same file extension ?
To be more clear..
Lets say I have 10 files... (1 Reply)
Discussion started by: kiran1112
1 Replies
9. Shell Programming and Scripting
I am running a UNIX script to get unused files and their sizes from the server. The issue is arising due to the spaces present in the filename/folder names.Due to this the du -k command doesn't work properly.But I need to calculate the size of all files including the ones which have spaces in them.... (4 Replies)
Discussion started by: INNSAV1
4 Replies
10. Debian
Hi,
Is there a way to find out the size of a sub-directory?
Eg subidrectory in main directory /data, like this : /data/solr-5.3.1.
When I do a df -h /data/solr-5.3.1, it still gives the size of /data:
root@L28condor:/data/solr-5.3.1# df -h .
Filesystem Size Used... (1 Reply)
Discussion started by: anaigini45
1 Replies
LEARN ABOUT DEBIAN
ledgersmb::dbobject::account
LedgerSMB::DBObject::Account(3pm) User Contributed Perl Documentation LedgerSMB::DBObject::Account(3pm)
NAME
LedgerSMB::DBObject::Account - Base class for chart of accounts entries
SYNOPSYS
This class contains methods for managing chart of accounts entries (headings and accounts).
INERITS
LedgerSMB::DBObject
METHODS
save()
This method saves the chart of accounts entry.
The hash component of the object may contain an id attribute, used to overwrite an account if that one exists.
Hash entries Used:
id: (optional): If set, overwrite existing account. accno: the text used to specify the account number description: Text to describe
the account category: A = asset, L = liability, Q = Equity, I = Income, E = expense gifi_accno: The GIFI account entry control code
heading: (Optional) The integer representing the heading.id desired contra: If true, the account balances on the opposite side. tax:
If true, is a tax account link: a list of strings representing text box identifier.
get()
This method gets a chart of accounts entry. It requires that the $account->{id} value must be properly set.
check_transactions()
Returns true if the account has transactions, false if not. Also sets the $account->{has_transactions} value to the return value.
$account->{id} must be set.
is_recon
Returns true if is set up for reconciliation. False otherwise.
delete()
Attempts to delete the account. This will NOT succeed if the account is referenced in any way by any transactions, credit accounts,
etc.
$account->{id} must be set.
list()
Returns a list of all accounts.
generate_links()
A mostly-private method for generating and checking whether link data is valid.
This is usually done (automatically) in preparation for saving the information to the database.
list_headings
Returns a list of account_heading's. No inputs required.
SEE ALSO
LedgerSMB::DBObject, LedgerSMB
COPYRIGHT
Copyright (c) 2009, the LedgerSMB Core Team. This is licensed under the GNU General Public License, version 2, or at your option any later
version. Please see the accompanying License.txt for more information.
perl v5.14.2 2012-03-25 LedgerSMB::DBObject::Account(3pm)