loop through files splitting up and totals


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting loop through files splitting up and totals
# 1  
Old 03-08-2010
loop through files splitting up and totals

I need to go through files in a folder and names that match write to a new file and have counts of the seperate names. fairly new to scripting and know what I want but do not know how to script it. example of what I need follows

PACKAGE='ls -A1 | tr -s '-' '^' | cut -f2 -d"^" | sort -n'
loop through packages spliting up and totals
COUNT=0
for NAME in PACKAGE

1st time
save name

do
if curr name same as prev name
count
save name
if different name
total count
send count and prev name to file
reset count
done



Example DATA:

PKG.Mortgage3DayDisclosure
PKG.Mortgage3DayDisclosure
PKG.Mortgage3DayDisclosure

***total****

PKG.MortgageAppraisalDelivery
PKG.MortgageAppraisalDelivery
PKG.MortgageAppraisalDelivery

***total****

PKG.MortgageCommitment
PKG.MortgageCommitment

***total****

PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure
PKG.MortgageCreditScoreDisclosure

***total****

PKG.MortgageDeclination
PKG.MortgageDeclination
PKG.MortgageDeclination
PKG.MortgageDeclination

***total****

hopefully this makes sense......
# 2  
Old 03-08-2010
Code:
ls -A1 | tr -s '-' '^' | cut -f2 -d"^" | sort -n | 
 awk '{arr[$0]++} END {for(i in arr) { print i, arr[i]}} ' > newfile

By total I assume you meant a file count. If you need to sum values we need file layout information.
# 3  
Old 03-08-2010
awesome! this works for the most part but how do you have the totals print on the following line versus the same line as the the file name?

---------- Post updated at 02:38 PM ---------- Previous update was at 02:27 PM ----------

Had another question also.... need to pull from a range of dates (ex. 6:00 AM March 3 through 6:00 AM March 4) from a folder (same folder as prev question) and put those file names in a new file to process later. Dates would not be hard dates but dates from the folder directory, how would I do that?

---------- Post updated at 03:18 PM ---------- Previous update was at 02:38 PM ----------

I figured out how to put totals on a new line... Thanks for your help Jim.... that was awesome!!!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. Shell Programming and Scripting

Execution of loop :Splitting a single file into multiple .dat file

hdr=$(cut -c1 $path$file|head -1)#extract header”H” trl=$(cut -c|path$file|tail -1)#extract trailer “T” SplitFile=$(cut -c 50-250 $path 1$newfile |sed'$/ *$//' head -1')# to trim white space and extract table name If; then # start loop if it is a header While read I #read file Do... (4 Replies)
Discussion started by: SwagatikaP1
4 Replies

3. Shell Programming and Scripting

Splitting files into 100 files with field value

I want a script to split my file upon the last field (15) As file A,b,c,.......,01 C,v,n,.......,02 C,r,v,........,01 F,s,a,........,03 X,y,d,........,99 To make output 01.txt A,b,c,.......,01 C,r,v,........,01 02.txt C,v,n,.......,02 (12 Replies)
Discussion started by: teefa
12 Replies

4. Shell Programming and Scripting

Splitting files

Hello all I have a file which has around 80 million records, I want to split it to 12 equal files, I tried using the split command but it is allowing me to split according to number of lines or by size. Is there a way i can split the file into 12 files without worrying about the number of lines... (7 Replies)
Discussion started by: Sri3001
7 Replies

5. UNIX for Dummies Questions & Answers

Splitting Files Help

Hi Gurus, Lets say i have a file with some 30 records... How can i split that file into 3 files Also it shud be dynamic in the sense.. I wouldnt mind if file 1 has 15, file 2 has 10 and file 3 has 5.... Please help.. Thanks (6 Replies)
Discussion started by: saggiboy10
6 Replies

6. Shell Programming and Scripting

Report Totals

Hello, I have written a script in a previous server and its being migrated to a new server. I'm trying to debug my script since i've had to make minor changes to it to get it to work. I'm having a hard time getting my totals to populate here is the syntax DUMP_COUNT=`sqlplus -S... (4 Replies)
Discussion started by: senormarquez
4 Replies

7. UNIX for Dummies Questions & Answers

splitting the files

Hi, I have some files with 2 million odd records which i need to split into chunks of 0.5 millions. I have the file sorted with a key column in order. The same key value can appear as 4 or 5 records in the file. Hence after splitting we are checking whether all the key values are present in the... (5 Replies)
Discussion started by: dnat
5 Replies

8. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies

9. UNIX for Advanced & Expert Users

splitting the files

Hi, How can i split the big file by the lines?. For eg. I wanna split the file from the line 140 to 1700. (9 Replies)
Discussion started by: sharif
9 Replies

10. Shell Programming and Scripting

splitting the files

Hi, I'am using HP-UX.I have a input file which has 102 drop statements in it.I'am using csplit to split the files.The upper limit is 99 only.I'am using the -n 102 option.It says "suffix size not vaild".Any suggestions how to do it using csplit? Thanx in advance, sounder. (1 Reply)
Discussion started by: sounder123
1 Replies
Login or Register to Ask a Question