Split large file into 24 small files on one hour basis Post: 303036518

Sponsored Content

Top Forums UNIX for Beginners Questions & Answers Split large file into 24 small files on one hour basis Post 303036518 by Raghuram717 on Monday 1st of July 2019 02:25:44 AM

07-01-2019

Registered User

i Wrote some script in shell that was working for me
posting here for reference

timestamp=$1

Code:

z=1
 max=24
 for (( i=1; i <= $max; ++i ))
 do
 echo "starting date is $timestamp"
 get_starting_line_number=$(grep -nr "$timestamp" summaryReport_LoadTest.csv|head -1|cut -d: -f1)
 echo "starting_line_number is $get_starting_line_number"
 ntimestamp=$(date -d "$timestamp 1 hour" +"%Y-%m-%d %H:%M:%S")
 echo "ending date is $ntimestamp"
 get_ending_line_number=$(grep -nr "$ntimestamp" summaryReport_LoadTest.csv|tail -1|cut -d: -f1)
 echo "ending line number is $get_ending_line_number"
 awk 'NR >= '$get_starting_line_number' && NR <= '$get_ending_line_number'' summaryReport_LoadTest.csv > summaryReport_LoadTest_"$z".csv
 z=$(expr $z + 1)
 echo $z
 b=$(expr $get_ending_line_number - 1)
 sed -i ''$get_starting_line_number','$b'd'  summaryReport_LoadTest.csv
 timestamp=$ntimestamp
 echo "**********************************************************************************************"
done

Raghuram717

View Public Profile for Raghuram717

Find all posts by Raghuram717

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting large file into small files

Hi, I need to split a large file into small files based on a string. At different palces in the large I have the string ^Job. I need to split the file into different files starting from ^Job to the last character before the next ^Job. Also all the small files should be automatically named....

2. Shell Programming and Scripting

Split a file into 16 small files

Hi I want to split a file that has 'n' number of records into 16 small files. Can some one suggest me how to do this using Unix script? Thanks rrkk

3. Shell Programming and Scripting

Split large file and add header and footer to each small files

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file?

4. Shell Programming and Scripting

script to splite large file to number of small files

Dear All, Could you please help me to split a file contain around 240,000,000 line to 4 files all equally likely , note that we need to maintain that the end of each file should started by start flage (MSISDN) and ended by end flag (End), also the number of the line between the...

5. Shell Programming and Scripting

Split large zone file dump into multiple files

I have a large zone file dump that consists of ; DNS record for the adomain.com domain data1 data2 data3 data4 data5 CRLF CRLF CRLF ; DNS record for the anotherdomain.com domain data1 data2 data3 data4 data5 data6 CRLF

6. Shell Programming and Scripting

Split a large array into small chunks

Hi, I need to split a large array "@sharedArray" into 10 small arrays. The arrays should be like @sharedArray1,@sharedArray2,@sharedArray3...so on.. Can anyone help me with the logic to do so :(:confused:

7. Shell Programming and Scripting

How to split this txt file into small files?

Dear shell experts, I would like to spilt a txt file into small ones. However, I did not know how to program use shell. If someone could help, it is greatly appreciated! Specifically, I supposed there is file named A.txt. The content of the file likes this: Subject run condtion ACC time...

8. Shell Programming and Scripting

Breaking large file into small files

Dear all, I have huge txt file with the input files for some setup_code. However for running my setup_code, I require txt files with maximum of 1000 input files Please help me in suggesting way to break down this big txt file to small txt file of 1000 entries only. thanks and Greetings, Emily

9. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Dears, Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this...

10. Shell Programming and Scripting

Split large xml into mutiple files and with header and footer in file

Split large xml into mutiple files and with header and footer in file tried below it splits unevenly and also i need help in adding header and footer command : csplit -s -k -f my_XML_split.xml extrfile.xml "/<Document>/" {1} sample xml <?xml version="1.0" encoding="UTF-8"?><Recipient>...

LEARN ABOUT DEBIAN

ifpps

ifpps(8)							netsniff-ng-toolkit							  ifpps(8)

NAME

       ifpps - fetch and format kernel network statistics

SYNOPSIS

       ifpps	 -d|--dev <netdev> [-t|--interval <sec>][-p|--promisc][-c|--term]
	    [-C|--csv][-H|--csv-tablehead][-l|--loop][-v|--version][-h|--help]

DESCRIPTION

       A tiny tool to provide top-like reliable networking statistics.	ifpps reads out the 'real' kernel statistics, so it does not give
       erroneous statistics on high I/O load.

OPTIONS

       ifpps --dev eth0
	   Fetch eth0 interface statistics.

       ifpps --dev eth0 --interval 60 --csv
	   Output eth0 interface statistics every minute in CSV format.

OPTIONS

       -h|--help
	   Print help text and lists all options.

       -v|--version
	   Print version.

       -d|--dev <netdev>
	   Device to fetch statistics for i.e., eth0.

       -p|--promisc
	   Put the device in promiscuous mode

       -t|--interval <time>
	   Refresh time in sec (default 1 sec)

       -c|--term
	   Output to terminal

       -C|--csv
	   Output in CSV format.  E.g. post-processing with Gnuplot et al.

       -H|--csv-tablehead
	   Print CSV table head.

       -l|--loop
	   Loop terminal output.

AUTHOR

       Written by Daniel Borkmann <daniel@netsniff-ng.org>

DOCUMENTATION

       Documentation by Emmanuel Roullit <emmanuel@netsniff-ng.org>

BUGS

       Please report bugs to <bugs@netsniff-ng.org>

								    2012-06-29								  ifpps(8)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting large file into small files

Discussion started by: dncs

2. Shell Programming and Scripting

Split a file into 16 small files

Discussion started by: rrkks

3. Shell Programming and Scripting

Split large file and add header and footer to each small files

Discussion started by: ashish4422

4. Shell Programming and Scripting

script to splite large file to number of small files

Discussion started by: ahmed.gad