I am writing a script to split a log file - the log could contain multiple days worth of logs. The second line of the log contains the string "Version ". In my test log which comprises of two days worth of logs, this string appears twice - once each day.
Essentially I would like to split on the line before this line, unfortunately there is nothing unique in it. The script thus far gets the line number that "Version " is on and populates an array. My problem is....I am not sure how to subtract one from each array element. The main part of my code looks like this:
Code:
IFS=$'\n'
arr=($(grep -n "Version " $filename))
arr=("${arr[@]%%:*}")
unset IFS
for i in "${arr[@]}"
do
echo "${arr[a]}"
csplit -f $filename $filename $i &>/dev/null
rm -f $filename'00'
done
exit 1;
I delete the file ending in 00 currently as a hack because it currently splits on line 2, so that log actually only contains one line. And the result from the echo is:
Code:
2 1177299
So in this instance from my one log, I want to create two logs - the second log will begin on line 1177298
Thoughts on how to do this, or a better way to achieve it?
Note there are config options in the application to have the date and or time at the start of each line. In this example I would want two files created from this.....one containing lines 1-13 and the second containing line 14-end
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,288
Thanks Given: 430
Thanked 480 Times in 395 Posts
Hi.
Addressing the splitting of the file with standard utility csplit:
Code:
#!/usr/bin/env bash
# @(#) s1 Demonstrate splitting of file at /pattern/+-count, csplit.
# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C csplit
FILE=${1-data1}
# Remove debris from previous runs.
rm -f xx??
pl " Input data file $FILE:"
cat $FILE
pl " Current situation of resultant xx?? files (expecting none):"
ls xx??
pl " Results, counts of split file in characters:"
csplit -k -z $FILE '/Version/-1' '{*}'
pl " Current situation, and content of files:"
ls -lgG xx??
pe
head xx??
exit 0
producing:
Code:
$ ./s1
Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian GNU/Linux 5.0.8 (lenny)
bash GNU bash 3.2.39
csplit (GNU coreutils) 6.10
-----
Input data file data1:
Unrelated stuff
Version something
No discernible pattern
Version otherwise
-----
Current situation of resultant xx?? files (expecting none):
ls: cannot access xx??: No such file or directory
-----
Results, counts of split file in characters:
34
41
-----
Current situation, and content of files:
-rw-r--r-- 1 34 Sep 12 06:33 xx00
-rw-r--r-- 1 41 Sep 12 06:33 xx01
==> xx00 <==
Unrelated stuff
Version something
==> xx01 <==
No discernible pattern
Version otherwise
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,288
Thanks Given: 430
Thanked 480 Times in 395 Posts
Hi.
Here's one method with perl:
Code:
#!/usr/bin/env bash
# @(#) s2 Demonstrate splitting of file at /pattern/+-count, csplit.
# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C csplit perl
FILE=${1-data2}
# Remove debris from previous runs.
rm -f xx?? output*
pl " Input data file $FILE:"
cat $FILE
pl " Current situation of resultant xx?? files (expecting none):"
ls xx??
pl " Results, counts of split file in characters:"
csplit -k -z $FILE '/Version/-1' '{*}'
pl " Current situation, and content of files:"
ls -lgG xx??
pe
head xx??
pl " Renaming script, sample in perl:"
cat p1
# Rename split file based on content of file.
# Sample data line:
# Info: Started at: Tue Sep 10 10:30:04 2013
# New file name from fragments of data line:
# output.log_10Sep13_10:30:04
#
# extract line "Info: started at" from file
# split into array, create string from fields 758_6
# close, issue rename file to output.log_<string>
for file in xx*
do
./p1 $file
done
pl " Renamed files:"
ls -1 output*
exit 0
producing:
Code:
$ ./s2
Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian GNU/Linux 5.0.8 (lenny)
bash GNU bash 3.2.39
csplit (GNU coreutils) 6.10
perl 5.10.0
-----
Input data file data2:
Unrelated stuff
Info: App: Version 2.0.1 Date 2013/09/01
Info: ****************************************************************
Info: Started at: Tue Sep 10 10:30:04 2013
No discernible pattern
Info: App: Version 2.0.1 Date 2013/09/01
Info: ****************************************************************
Info: Started at: Tue Sep 11 10:30:03 2013
-----
Current situation of resultant xx?? files (expecting none):
ls: cannot access xx??: No such file or directory
-----
Results, counts of split file in characters:
181
188
-----
Current situation, and content of files:
-rw-r--r-- 1 181 Sep 12 10:14 xx00
-rw-r--r-- 1 188 Sep 12 10:14 xx01
==> xx00 <==
Unrelated stuff
Info: App: Version 2.0.1 Date 2013/09/01
Info: ****************************************************************
Info: Started at: Tue Sep 10 10:30:04 2013
==> xx01 <==
No discernible pattern
Info: App: Version 2.0.1 Date 2013/09/01
Info: ****************************************************************
Info: Started at: Tue Sep 11 10:30:03 2013
-----
Renaming script, sample in perl:
#!/usr/bin/env perl
# @(#) p1 Demonstrate manipulation to rename file.
# Info: Started at: Tue Sep 10 10:30:04 2013
# 0 1 2 3 4 5 6 7
# output.log_10Sep13_10:30:04
# 5 4 7 6
# extract line "Info: started at" from file
# split into array, create string from fields 758_6 (but i-1 # array)
# close, issue rename file to output.log_<string>
use strict;
use warnings;
my ( $file, $f, @pieces, $newname );
$file = shift || die " Expected filename, got nothing.\n";
open( $f, "<", $file ) || die " Cannot open file $file\.";
while (<$f>) {
chomp;
next if not /Info: Started at:/;
@pieces = split(/\s+/);
$newname = $pieces[5] . $pieces[4] . $pieces[7] . "_" . $pieces[6];
close $f || die " Oops, cannot close $file\n";
rename $file, "output_$newname" || die " Cannot rename $file\n";
last;
}
exit(0);
-----
Renamed files:
output_10Sep2013_10:30:04
output_11Sep2013_10:30:03
To avoid making the perl script too complex, the file loop is in the shell script. One optimization that could be done is to pull that loop into the perl script.
Hi ,
I am a total beginner so bear with me.
I have the below code which works . I need to extend it by iterating thru the array arr and executing a command in each loop. some thing on the lines of below.
I need to run this in a Jenkins script , so I would need below
bash script to run... (6 Replies)
hi all,
i have devised a script that starts in /restored/ and in there, there are a lot of sub folders called peoples names and in the sub folders are files/folders and it deletes the data in the sub folders BUT not the sub folder itself and it should then touch a file in all the sub folders... (3 Replies)
hello,
i need a bit of help on how to do this effectively in bash without a lot of extra looping or massive switch/case
i have a long array of M elements and a short array of N elements, so M > N always. M is not a multiple of N.
for case 1, I want to stretch N to fit M
arrayHuge
H = (... (2 Replies)
Trying to do some control flow parsing based on the index postion of an array member. Here is the pseudo code I am trying to write in (preferably in pure bash) where possible. I am thinking regex with do the trick, but need a little help.
pesudo code
if == ENDSINFIVEINTS ]]; then
do... (4 Replies)
Hi,
I have written a script that returns the line number of the pattern i want and i stored the line number in a variable.Now i want to delete all the lines in a file above this line number which is stored in a variable.
i am using sed '1,$getlinenumberd' > file1.txt which is not working(wrog... (5 Replies)
Hi,
I have written a script that returns the line number of the pattern i want and i stored the line number in a variable(getlinenumber).Now i want to delete all the lines in a file above this line number which is stored in a variable.
i am using sed '1,$getlinenumberd' > file1.txt which is... (2 Replies)
I have been trying this a lot of different ways and haven't found too much online. Here's what I've got so far:
j=0
declare -a first
zero=(`cat $tmpfile`)
for i in "${zero}"
do
command $i >> "${first}"
... (4 Replies)
1) I wrote a script and gave the desired permissions using "chmod 755 scriptname". Now if i edit the script file, why do i need to set the permission again? Didn't i set the permission attribute.. or if i edit the file, does the inode number of file changes?
2) I am running my unix on a server... (1 Reply)
Hello everybody!
I have a problem "Write script, which will display the actual number of bash command interpreter, working in the operating system." I used ps command to list all process running in the system but I don't know how to select only bash command interpreter. Pls give me some... (3 Replies)
