Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

split input data file and put into same output file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting split input data file and put into same output file
# 1  
Old 04-15-2011
split input data file and put into same output file
Hi All,
I have two input file and need to generate a CSV file. The existing report just "GREP" the records with the Header and Tailer records with the count of records.
Now i need to split the data into 25 records each in the same CSV file.

id_file (Input file )
Code:
227050994
232510151

report_data (Input file)
Code:
13,227050994,LALN3819959,2089851292,2085254977,20110224
.
.
.
13,227050994,LFLN3449126,2082113563,2082113396,20110224
283,232510151,LALC3914497,152469347,152466752,20110224
283,232510151,LFSD3449916,1329836600,1329836311,20110224
.
.
.
.
283,232510151,LFSL3455668,1142303778,1142301334,20110224
283,232510151,LFST3462358,1425672593,1425672226,20110224

Existing Report (output file)
Code:
Start of Report 20110224~ 
227050994 20110224 
13 227050994 LALN3819959 2089851292 2085254977 20110224
.
.
.
13 227050994 LFLN3449126 2082113563 2082113396 20110224
~End of Report 227050994 19
Start of Report 20110224~ 
232510151 20110224 
283 232510151 LALC3914497 152469347 152466752 20110224
283 232510151 LZNI0568201 2891873461 2891871770 20110224
.
.
.
.
283 232510151 LFSL3455668 1142303778 1142301334 20110224
283 232510151 LFST3462358 1425672593 1425672226 20110224
~End of Report 232510151 79

Script to process the files
Code:
OUT_FILE="report.csv"
for line in `cat id_file.dat`
do
  echo "Report,`date +%Y%m%d`~" >>$OUT_FILE
  echo "$line,`date +%Y%m%d`" >>$OUT_FILE
  grep ",$line," report_data.dat >>$OUT_FILE
  echo "~End of Report,$line,`grep -c ",$line," report_data.dat`" >>$OUT_FILE
done

Thank You,
Rasmith
Last edited by Scott; 04-15-2011 at 02:14 PM.. Reason: Removed formatting, added code tags, indentation
# 2  
Old 04-15-2011
So, you want N files of 25 and a file for residue, if any? Feed the grep to an inner loop, something like:
Code:
 
ln=0 fn=0
grep . . . | while read line2
do
 if (( ++ln == 1 ))
 then
  date "+header stuff" >$OUT_FILE-$(( ++fn ))
 fi
 echo $line2 >$OUT_FILE-$fn
 if (( ln == 25 ))
 then
  echo trailer stuff $ln >$OUT_FILE-$fn
  ln=0
 fi
done
if (( ln > 0 ))
then
 echo trailer stuff $ln >$OUT_FILE-$fn
fi

This User Gave Thanks to DGPickett For This Post:
rasmith
# 3  
Old 04-16-2011
Need to split the data in multiples of 25 with left over data of the ID in the same CSV file as given below and not N files of 25.

Code:
Start of Report    20110224~
227050994    20110224               
13    227050994    LALN3819959    2089851292    2085254977    20110224
.
.
13    227050994    LFLN3449126    2082113563    2082113396    20110224
~End of Report    227050994    19           
Start of Report    20110224~               
232510151    20110224               
283    232510151    LALC3914497    152469347    152466752    20110224
.
.
.
283    232510151    LBWM3183936    1905799142    1905773331    20110224
~End of Report    232510151    25           
Start of Report    20110224~               
232510151    20110224               
283    232510151    LBWW2842523    1752698299    1752690888    20110224
.
.
283    232510151    LDND3387092    1474815506    1474814211    20110224
~End of Report    232510151    25           
Start of Report    20110224~               
232510151    20110224               
283    232510151    LHSS3133014    1793433382    1793513907    20110224
.
.
283    232510151    LFLV3454969    1514942972    1514279499    20110224
~End of Report    232510151    25           
Start of Report    20110224~               
232510151    20110224               
283    232510151    LFMY3455276    1943464440    1943466154    20110224
.
.
283    232510151    LFST3462358    1425672593    1425672226    20110224
~End of Report    232510151    4           
Start of Report    20110224~               
232510151    20110224

Thanks
Rasmith

---------- Post updated 04-16-11 at 10:11 AM ---------- Previous update was 04-15-11 at 05:16 PM ----------

hi David,

Can you please explain your script.

Thanks,

Rasmith
Last edited by Franklin52; 04-16-2011 at 08:55 AM.. Reason: Please use code tags
# 4  
Old 04-19-2011
This bit of script can be put into a shell subroutine or subshell and will process lines as follows:
  1. Initialize two variables to zero, line number and file number,
  2. grep out the desired lines and pipe them to a while read loop.
  3. Increment the line number, and if it is 1, spit out a header using the date command + option (not echo and `date`) to a new file using the incremented file number in the entry name.
  4. Spit out the current line. (s/b >>)
  5. if this is line 25, spit out a trailer (s/b >>) and then zero the line counter.
  6. After the loop ends (EOF), if there is a partial file, add a trailer (using $ln).
It is good to start designing froun the outside in and with a high perspective, but then code in layers from the inside out. This is an inside bit to chop the files up as you asked.
# 5  
Old 04-21-2011
hope below perl can help you some
Code:
my %hash = ('227050994'=>1,'232510151'=>1);
while(<DATA>){
	chomp;
	my @arr = split(",",$_);
	next if not exists $hash{$arr[1]};
	$result{$arr[1]}->{$arr[$#arr]}->{'CNT'}++;
	$result{$arr[1]}->{$arr[$#arr]}->{'STR'}=$result{$arr[1]}->{$arr[$#arr]}->{'STR'}."\n".$_;
}
foreach my $key(keys %result){
	foreach my $k(sort {$a <=> $b} keys %{$result{$key}}){
		print "Start of Report ",$k,"~\n";
		print $key," ", $k;
		print $result{$key}->{$k}->{'STR'},"\n";
		print "~End of Report ", $key," ", $result{$key}->{$k}->{'CNT'},"\n\n";
	}
}
__DATA__
13,227050994,LALN3819959,2089851292,2085254977,20110224
13,227050994,LFLN3449126,2082113563,2082113396,20110224
283,232510151,LALC3914497,152469347,152466752,20110224
283,232510151,LFSD3449916,1329836600,1329836311,20110224
283,232510151,LFSL3455668,1142303778,1142301334,20110225
283,232510151,LFST3462358,1425672593,1425672226,20110226

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from a log file and put it in a file

Hi, I would like to seek your help for a script that will extract data from log file and put it in a file. Sample log file 2018-10-23 12:33:21 AI ERROR -- tpid: SAMPLE_TH account: 123456789 aiSessionNumber: 660640464 mapName: xxx to yyy errorDesc: Translation Error:ErrorNumber : 993 ... (2 Replies)
Discussion started by: neverwinter112
2 Replies

2. Shell Programming and Scripting

awk to reformat output if input file is empty, but not if file has data in it

The below awk improved bu @MadeInGermany, works great as long as the input file has data in it in the below format: input chrX 25031028 25031925 chrX:25031028-25031925 ARX 631 18 chrX 25031028 25031925 chrX:25031028-25031925 ARX 632 14... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

Split: File into multiple and keeping the same 3 lines from input into all output files

The following code will split the infile into multiple files. However, I need it to insert the same first 3 lines from the original input file into each splitted file. How do I modify my script below to do so: print -n "Enter file name to split? " ; read infile if then echo "Invalid file... (4 Replies)
Discussion started by: mrn6430
4 Replies

4. Shell Programming and Scripting

Get Data From CSV File and put into a txt file

Hi Guys, File A I have File A as CSV Format.... No R SS MK Par value S AL A1 PKL123 Lo12 1 S AL A2 PKl123 Lo34 22 S AL A3 PkLK234 Lo67 -34 S AL A4 PkLK235 Lo09 120 S AL A5 PkLK236 Lo76 19 S AL A6 PkLK237 Lo44 -17 S AL A7 PkLK238 Lo90 2 S AL A8 PkLK239 Lo34 -9 I want file B like... (4 Replies)
Discussion started by: asavaliya
4 Replies

5. Shell Programming and Scripting

adding data in input file if 2nd script output SUCCESS

Hi All, how can i edit my original data and add more data if my 2nd script outputs SUCESS? ex. input file: 1.txt nik,is,the 1script.sh if 2ndscript.sh output SUCCESS then i'm going to edit my input file and add data best,pogi.. sample outputdata. nik,is,the,best,pogi 2ndscript.sh... (3 Replies)
Discussion started by: nikki1200
3 Replies

6. Shell Programming and Scripting

How to add data from 2 input files and save it in 1 output file

Hi, i have 2 input files which are file1.txt and file2.txt. I need to extract data from file1.txt and file2.txt and save it in file3.txt like example below:- File1.txt ID scrap1 Name scrap1 start 1 end 10 ID scrap2 Name scrap2 start 11 end ... (4 Replies)
Discussion started by: redse171
4 Replies

7. Shell Programming and Scripting

Dynamic output file generation using a input text file with predefined output format

Hi, I have two files , one file with data file with attributes that need to be sent to another file to generate a predefined format. Example: File.txt AP|{SSHA}VEEg42CNCghUnGhCVg== APVG3|{SSHA}XK|"password" AP3|{SSHA}XK|"This is test" .... etc --------- test.sh has... (1 Reply)
Discussion started by: hudson03051nh
1 Replies

8. Shell Programming and Scripting

Need script to take input from file, match on it in file 2 and input data

All, I am trying to figure out a script to run in windows that will allow me to match on First column in file1 to 8th Column in File2 then Insert file1 column2 to file2 column4 then create a new file. File1: 12345 Sam 12346 Bob 12347 Bill File2:... (1 Reply)
Discussion started by: darkoth
1 Replies

9. UNIX for Dummies Questions & Answers

I want some selected data from first file and put into other file in specified format

I have a file with follwing content ---------------------------------- SCHEDULE XXXXXXXXX#JOBCOUNT ON EVERYDAY AT 0000 PRIORITY 50 SCHEDULE XXXXXXXXX#ABCDEFGH ON EVERYDAY AT 0001 PRIORITY 29 SCHEDULE... (5 Replies)
Discussion started by: shreyas
5 Replies

10. Shell Programming and Scripting

Replacing data of output file with input

Hi, I have a ksh which peocess and get me data from 3 days... ie if i process it on jan 28.. it gets data for 25, 26 and 27.... the process run every day and get previous 3 days data...all this data is appened to a file lets call time.out Now time.out cannot have deplicate data so what i want... (10 Replies)
Discussion started by: bhagya2340
10 Replies
Login or Register to Ask a Question