Order based on timestamp in a single file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Order based on timestamp in a single file
# 1  
Old 01-18-2013
Order based on timestamp in a single file

Hi All,

I have a large text file which is a combination of multiple files. This is what I used and it worked.

Code:
for i in /home/docs/text/*
 do
 cat $i >> Single_File
 done

Now wondering, if there is a way to sort that single large file based on timestamps in ascending order. Text file look something like this

Code:
[20100623 07:55:52.61|DEBUG  |#000062|test123 "GET /PowerBuilder/Compny3.htm HTTP/1.0" 200 5593  
CRequestWrapper: Destructor - object 123456789 cleaned up
[20100621 06:55:52.61|INFO |#000051|test123 "GET /PowerBuilder/Compny3.htm HTTP/1.0" 200 5593 
CRequestWrapper: Destructor - object 123456789 cleaned up
[20100622 08:55:52.61|WARNING|#000055|test123 "GET /PowerBuilder/Compny3.htm HTTP/1.0" 200 5593 
CRequestWrapper: Destructor - object 123456789 cleaned up

See the time stamps in bold. This is not in a sorted fashion. I want them to be aligned in ascending order.

Thanks for the help in advance.

Cheers!

Moderator's Comments:
Mod Comment Please use code tags next time for your code and data.
# 2  
Old 01-18-2013
Code:
awk -F\| '/^\[/ { 
  _1 = $1
  sub(/[^ ]+ */, x, _1)
  }
{ 
  print _1, NR, $0 
  }' OFS=_ infile  |
    sort -t_ -k1,1 -k2,2n |
      cut -d_ -f3-


Last edited by radoulov; 01-18-2013 at 09:34 AM..
This User Gave Thanks to radoulov For This Post:
# 3  
Old 01-18-2013
Hi Radoulov,

This worked for me. Thank you so much. But i need to understand awk a bit better to tweak any changes in the future.

Thanks.!
# 4  
Old 01-18-2013
-F\| - records are split at |
_1 = $1 - save $1 into a variable because we need to modify it
sub(/[^ ]+ */, x, _1) - strip the leading part of the string currently in _1 (the part before and including the white space characters)
print _1, NR, $0 - for every record: print the modified value of _1, the current record number and the actual record
OFS=_ - set the output field separator to a custom string for easy manipulation with cut

Now the records are easy to sort: first by timestamp and by record number.

Last edited by radoulov; 01-18-2013 at 10:53 AM..
This User Gave Thanks to radoulov For This Post:
# 5  
Old 01-18-2013
Can you confirm that the date/time stamp is in the format "YYYYMMDD" then Hour, minute, second, hundredth and that the lines are all headed by the date/time stamp and not split to a second line?

If so, you just need to insert a simple sort into your command, such as:-
Code:
for i in /home/docs/text/*
 do
 cat $i
 done | sort  > Single_File

........, but you could simplify your code a little. There is no need to build a loop.

Consider:-
Code:
sort /home/docs/text/* > Single_file

If you need to join the records up, then use radoulov's code and pipe it through sort before writing it to Single_File


I hope that this helps,
Robin
Liverpool/Blackburn
UK
This User Gave Thanks to rbatte1 For This Post:
# 6  
Old 01-18-2013
Thanks for the details Radoulov.

@ Rbatte1:

I could use sort if it is a single file bt I am trying to collate multiple files into 1 file and then applying the awk on that file to have the second file of my desired result..

Advise me if there is any better way!

Code I am using is below:

Code:
for i in /root/Desktop/test/*.*
 do
 cat $i >> Single1
 done
awk -F\| '/^\[/ { 
  _1 = $1
  sub(/[^ ]+ */, x, _1)
  }
{ 
  print _1, NR, $0 
  }' OFS=_ Single1  |
    sort -t_ -k1,1 -k2,2n |
      cut -d_ -f3- >> Single2


Last edited by radoulov; 01-18-2013 at 12:47 PM..
# 7  
Old 01-18-2013
If the total bytes of the filenames is below the ARG_MAX limit:

Code:
awk -F\| '/^\[/ { 
  _1 = $1
  sub(/[^ ]+ */, x, _1)
  }
{ 
  print _1, NR, $0 
  }' OFS=_  /root/Desktop/test/*.* |
    sort -t_ -k1,1 -k2,2n |       
      cut -d_ -f3- >  Single2


Last edited by radoulov; 01-18-2013 at 12:53 PM.. Reason: Corrected
This User Gave Thanks to radoulov For This Post:
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Filter records from a log file based on timestamp

Dear Experts, I have a log file that contains a timestamp, I would like to filter record from that file based on timestamp. For example refer below file - cat sample.txt Jan 19 20:51:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)... (6 Replies)
Discussion started by: mukulverma2408
6 Replies

2. Shell Programming and Scripting

Picking the latest file based on a timestamp for a Dynamic file name

Hi , I did the initial search but could not find what I was expecting for. 15606Always_9999999997_20160418.xml 15606Always_9999999998_20160418.xml 15606Always_9999999999_20160418.xml 9819Always_99999999900_20160418.xml 9819Always_99999999911_20160418.xmlAbove is the list of files I... (4 Replies)
Discussion started by: chillblue
4 Replies

3. Shell Programming and Scripting

Help with order lines from a file based on a pattern

Hi I need to order these lines from a txt file my file looks like this IMSI ........................ 1234567890 APN ......................... INTERNET.COM APN ......................... MMS.COM APN ......................... WAP.COM APN ......................... BA.COM IMSI... (4 Replies)
Discussion started by: alone77
4 Replies

4. UNIX for Dummies Questions & Answers

Display files based on particular file timestamp

Hi, I have requirement to list out files that are created after particular file. ex. I have below files in my directory. I want to display files created after /dirdat/CG1/cg004440 file. ./dirdat/CG1/cg004438 09/07/14 0:44:05 ./dirdat/CG1/cg004439 09/07/14 6:01:48 ... (3 Replies)
Discussion started by: tmalik79
3 Replies

5. Shell Programming and Scripting

File w/ many line pairs--how do I order based on 1st lines only?

I have a file in which the data is stored in pairs of lines. The first line (beginining with ">") is a header, the second line is a sequence. I would like to sort the file by species name. Desired output for the example file: I can use sort -t'_' -k2 to alphabetize headers in the... (1 Reply)
Discussion started by: pathunkathunk
1 Replies

6. UNIX for Dummies Questions & Answers

Sorting files based on timestamp and picking the latest file

Hi Friends, Newbie to shell scripting Currently i have used the below to sort data based on filenames and datestamp $ printf '%s\n' *.dat* | sort -t. -k3,4 filename_1.dat.20120430.Z filename_2.dat.20120430.Z filename_3.dat.20120430.Z filename_1.dat.20120501.Z filename_2.dat.20120501.Z... (12 Replies)
Discussion started by: robertbrown624
12 Replies

7. Shell Programming and Scripting

sort the files based on timestamp and execute sorted files in order

Hi I have a requirement like below I need to sort the files based on the timestamp in the file name and run them in sorted order and then archive all the files which are one day old to temp directory My files looks like this PGABOLTXML1D_201108121235.xml... (1 Reply)
Discussion started by: saidutta123
1 Replies

8. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

9. Shell Programming and Scripting

Purge files based on timestamp avl in file name

Dear All, I have the followoing requirement.. REQ-1: Suppose I have the following files XX_20070202000101.zip XX_20080223000101.zip XX_20080226000101.zip XX_20080227000101.zip XX_20080228000101.zip XX_20080229000101.zip Suppose sysdate = 29 Feb 2007 I need to delete all files... (3 Replies)
Discussion started by: sureshg_sampat
3 Replies

10. Shell Programming and Scripting

How to grep in order based on the input file

Here is an answer rather than a question that I thought I would share. In my first attempt, I was using grep to find a list from file.1 within another file, file.2 which I then needed to paste the output from file.3 to file.1 in the same order. However, the results weren't what I wanted. At... (2 Replies)
Discussion started by: Kelam_Magnus
2 Replies
Login or Register to Ask a Question