My attempt is FAR too slow


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting My attempt is FAR too slow
# 1  
Old 05-15-2012
My attempt is FAR too slow

I have numerous (hundreds) of data files with various values in each file. The values are 1 per line, and identified by the fieldname in the 1st field in the line, which is delimited from the actual field value by a colon. So an example from one of the files looks like this:

NAME: Bob Jones
ADDRESS: 123 Main Street
CITY: Omaha
STATE: Nebraska
1_TIMESTAMP: 1234567890
2_TIMESTAMP: 2012-05-15 10:15:20

So here's my dilemma - SOME of the timestamps are in epoch format (seconds since 1970) and SOME are already in human readable format (YYYY-MM-DD HH:MM:SS). This cannot be identified by the timestamp fieldname - the values are truely mixed. I need to convert ALL of the epoch timestamps into human readable ones, while NOT changing any other data in the files. I have this working in a shell script, but again, I have hundreds of files, and literally millions of lines to parse through, and my script has currently finished only 15 files in 50 minutes. How can I speed this up?

My script is below...

Code:
 
for DETAIL_FILE in `find ${MY_DATAFILE_LOCATION} -type f`
do
  grep -a TIMESTAMP ${DETAIL_FILE} |cut -d: -f2 |grep -v "-" |sed 's/^ *//' |sort -un > ${MY_DATAFILE_LOCATION}timestamp.txt
  while read UNIQUE_TIMESTAMP
  do
    HUMAN_READABLE_TIMESTAMP=`date +%F" "%T -d@${UNIQUE_TT_TIMESTAMP}`
    sed -i "s/TIMESTAMP: ${UNIQUE_TT_TIMESTAMP}/TIMESTAMP: ${HUMAN_READABLE_TIMESTAMP}/g" ${DETAIL_FILE}
  done<${MY_DATAFILE_LOCATION}timestamp.txt
done


Last edited by Finja; 05-15-2012 at 09:57 PM..
# 2  
Old 05-15-2012
This might work a bit quicker:

Code:
OLDIFS=$IFS
IFS=$': \t'
grep -d recurse "TIMESTAMP: [0-9]*$" ${MY_DATAFILE_LOCATION} | while read DETAIL_FILE ignore UNIQUE_TIMESTAMP
do
   HUMAN_READABLE_TIMESTAMP=`date +%F" "%T -d@${UNIQUE_TIMESTAMP}`
   sed -i "s/TIMESTAMP: ${UNIQUE_TIMESTAMP}/TIMESTAMP: ${HUMAN_READABLE_TIMESTAMP}/g" ${DETAIL_FILE}
done
IFS=$OLDIFS

You could keep the sort -un in the pipeline if you have a lot of instances of the same timestamp within each file

---------- Post updated at 11:05 AM ---------- Previous update was at 10:25 AM ----------

This should be quicker still if you have GNU awk:

Code:
grep -d recurse -l "TIMESTAMP: [0-9]*$" ${MY_DATAFILE_LOCATION} | xargs -r awk '
/TIMESTAMP: [0-9]*$/ { $0=$1 " " strftime("%F %T", $2) }
{ L[FILENAME]++; line[FILENAME,FNR]=$0 }
END { for(file in L) {
          for(i=1;i<=L[file];i++) print line[file,i] > file
          close(file) }}'


Last edited by Chubler_XL; 05-15-2012 at 09:30 PM..
This User Gave Thanks to Chubler_XL For This Post:
# 3  
Old 05-15-2012
The awk solution is awesome. Thanks!
Login or Register to Ask a Question

Previous Thread | Next Thread

4 More Discussions You Might Find Interesting

1. Programming

Failed SSL Connection Attempt

The below error message I started seeing using Ubuntu 14.04 and was wondering if the forum has seen it because I cant seem much on the net for this: perl -e 'use IO::Socket::SSL qw(debug3);IO::Socket::SSL->new(PeerAddr=>"10.0.0.100",PeerPort=> 443,Proto=>"TCP") or die $!' DEBUG:... (1 Reply)
Discussion started by: metallica1973
1 Replies

2. Solaris

ZFS Restore Attempt

We currently have T2000 servers attached to Sun 3320 Storage Arrays. We run a database on the 3320 devices with the storage created through ZFS. We use 3 RAIDZ2 pools and then create our partitions from those. Now onto the question. If our T2000 fails and we swap out machines is there a way... (1 Reply)
Discussion started by: buckhtr77
1 Replies

3. Linux

attempt to access beyond end of device

Hi, we have running 8 box sles 9 cluster and on an nfs filesystem we have the problem which is grepped from /var/log/messages. Jun 8 13:40:46 qnclpx02 kernel: attempt to access beyond end of device Jun 8 13:40:46 qnclpx02 kernel: sdat: rw=0, want=8894615912, limit=314572800 Is there... (1 Reply)
Discussion started by: ortsvorsteher
1 Replies

4. Linux

Error 24: Attempt to access block outside partition

Grub throwing Error:: Error 24: Attempt to access block outside partition. done the following root (hd0,0) After that when trying to load kernel it is throwing the error. Please guide. new to grub. Thx, Siva. (0 Replies)
Discussion started by: Sivaswami
0 Replies
Login or Register to Ask a Question