Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


Split large file into 24 small files on one hour basis


Login or Register to Reply

 
Thread Tools Search this Thread
# 8  
Actually, you do not even need to convert to a unix timestamp in PHP, you can use your formatted time string directly:

For example, I just tested your example data with your formatted time :

Code:
<?php
echo  date(h,"2019-05-20 09:55:39.945");

06

No need to convert to a unix timestamp.

Untested, but close:

Code:
<?php
$data = file_get_contents('mydata.txt');
foreach ($data as $entry) {
    $my_time_stuff = explode(' ', $entry);
    $formatted_time = $my_time_stuff[0] . ' ' . $my_time_stuff[1];
    $hour = date(h, $formatted_time);
    $file = 'myfilename' . $hour . '.txt';
    file_put_contents($file, $entry, FILE_APPEND | LOCK_EX);
}

Hope this PHP example helps.

If you want it to be "harder to read" and short:

Code:
<?php
$data = file_get_contents('mydata.txt');
foreach ($data as $entry) {
    $m= explode(' ', $entry);
    file_put_contents('myfilename' . date(h, $m[0] . ' ' . $m[1]) . '.txt', $entry, FILE_APPEND | LOCK_EX);
}

I prefer the long, easy to read, more-or-less self-documenting (first) version.
# 10  
Here is a shell script that simply splits on the hour (00...23):
Code:
#!/bin/sh
split_to(){
  prev=""
  read header
  while IFS=" :" read date hour rest
  do
    case $hour in
    ([0-9][0-9])
      if [ "$hour" != "$prev" ]
      then
        exec >"$hour.$1"
        prev=$hour
        echo "$header"
      fi
      echo "${date} ${hour}:${rest}"
    ;;
    esac
  done
}

if [ $# -eq 0 ]
then
  split_to out
fi
for arg
do
  split_to "$arg" < "$arg"
done

Run with /path/to/scriptname filename to get 00.filename ... 23.filename
or with /path/to/scriptname < filename to get 00.out ... 23.out
# 11  
i Wrote some script in shell that was working for me
posting here for reference


timestamp=$1

Code:
z=1
 max=24
 for (( i=1; i <= $max; ++i ))
 do
 echo "starting date is $timestamp"
 get_starting_line_number=$(grep -nr "$timestamp" summaryReport_LoadTest.csv|head -1|cut -d: -f1)
 echo "starting_line_number is $get_starting_line_number"
 ntimestamp=$(date -d "$timestamp 1 hour" +"%Y-%m-%d %H:%M:%S")
 echo "ending date is $ntimestamp"
 get_ending_line_number=$(grep -nr "$ntimestamp" summaryReport_LoadTest.csv|tail -1|cut -d: -f1)
 echo "ending line number is $get_ending_line_number"
 awk 'NR >= '$get_starting_line_number' && NR <= '$get_ending_line_number'' summaryReport_LoadTest.csv > summaryReport_LoadTest_"$z".csv
 z=$(expr $z + 1)
 echo $z
 b=$(expr $get_ending_line_number - 1)
 sed -i ''$get_starting_line_number','$b'd'  summaryReport_LoadTest.csv
 timestamp=$ntimestamp
 echo "**********************************************************************************************"
done

# 12  
Try also
Code:
paste <(date +%s -f<(cut -d, -f1 file) 2>&1) file | awk -F"\t" '
NR == 1         {sub ($1 FS, "")
                 HD = $0
                }
NR == 2         {BDT = $1 - $1%60
                }
NR >  1         {HCNT = int (($1 - BDT)/3600)
                 if (HCNT > 23) exit
                 FN = "summaryReport_LoadTest_" HCNT ".csv"
                 sub ($1 FS, "")
                 if (!X[FN]++) print HD  >  FN
                 print  >  FN
                }
'

It calculates the start from the first timestamp encountered in line 2. If you want to pass the start from outside, set BDT with awk's -v option. Then, to skip the leading lines outside the target window, additional logics need to be installed.
# 13  
getting error like this...
Code:
./script3.sh: line 1: syntax error near unexpected token `('
./script3.sh: line 1: `paste <(date +%s -f<(cut -d, -f1 file) 2>&1) file | awk -F"\t" ''


Last edited by rbatte1; 1 Week Ago at 07:24 AM.. Reason: Added CODE tags for clarity
# 14  
What is your shell?


NEVER forget to tell your OS and shell in a thread.
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Breaking large file into small files
emily
Dear all, I have huge txt file with the input files for some setup_code. However for running my setup_code, I require txt files with maximum of 1000 input files Please help me in suggesting way to break down this big txt file to small txt file of 1000 entries only. thanks and Greetings, Emily... Shell Programming and Scripting
12
Shell Programming and Scripting
Split a large array into small chunks
rkrish
Hi, I need to split a large array "@sharedArray" into 10 small arrays. The arrays should be like @sharedArray1,@sharedArray2,@sharedArray3...so on.. Can anyone help me with the logic to do so :(:confused:... Shell Programming and Scripting
6
Shell Programming and Scripting
Split large file and add header and footer to each small files
ashish4422
I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file?... Shell Programming and Scripting
7
Shell Programming and Scripting
Split a file into 16 small files
rrkks
Hi I want to split a file that has 'n' number of records into 16 small files. Can some one suggest me how to do this using Unix script? Thanks rrkk... Shell Programming and Scripting
10
Shell Programming and Scripting
Splitting large file into small files
dncs
Hi, I need to split a large file into small files based on a string. At different palces in the large I have the string ^Job. I need to split the file into different files starting from ^Job to the last character before the next ^Job. Also all the small files should be automatically named....... Shell Programming and Scripting
4
Shell Programming and Scripting

Featured Tech Videos