extract pattern from csv file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting extract pattern from csv file
# 1  
Old 09-17-2010
MySQL extract pattern from csv file

hi,

i have a situation where i have a csv file in the format

date,name

eg:

1284631889869,a
1284631889879,b
1284631889459,c
.
.
.
.
.
.
.

now i take a time and an interval from user.

eg:time:13:00:00 interval -1

what i must do is i must have all the data in the intervals like

13:00:00 a,b,g(i must have names in the interval 13:00:00-13:00:01)
13:00:02 c,d,h(i must have names in the interval 13:00:01-13:00:02)

.
.
.
.


how can i do it with perl

can some one help me.........
# 2  
Old 09-17-2010
Quote:
Originally Posted by niteesh_!7
...
date,name
eg:
1284631889869,a
1284631889879,b
1284631889459,c
.
.
.
.
...
now i take a time and an interval from user.
eg:time:13:00:00 interval -1

what i must do is i must have all the data in the intervals like

13:00:00 a,b,g(i must have names in the interval 13:00:00-13:00:01)
13:00:02 c,d,h(i must have names in the interval 13:00:01-13:00:02)
.
.
how can i do it with perl
...
Code:
$
$
$ cat input
1274706110,a
1274706254,b
1274706331,c
1274706287,d
1274706263,e
1274706358,f
1274706284,g
1274706366,h
1274706102,i
1274706314,j
1274706177,k
1274706280,l
1274706037,m
1274706342,n
1274706210,o
1274706263,p
1274706249,q
1274706255,r
1274706126,s
1274706002,t
1274706369,u
1274706093,v
1274706354,w
1274706110,x
1274706162,y
1274706280,z
$
$ cat -n bucket_divide.pl
     1  #perl -w
     2  use Time::Local;
     3  $start = $ARGV[0];
     4  $interval = $ARGV[1];
     5  ($mon,$day,$year,$hr,$min) = split/[\/ :]/,$start;
     6  $stime = timelocal(0,$min,$hr,$day,$mon-1,$year);
     7  $file="input";
     8  open(F, $file) or die "Can't open $file: $!";
     9  while (<F>) {
    10    chomp;
    11    ($sec,$item) = split/,/;
    12    $indx = int ($sec-$stime)/$interval;
    13    $bucket[$indx] .= ",$item";
    14  }
    15  close(F) or die "Can't close $file: $!";
    16  for($i=0; $i<=$#bucket; $i++){
    17    $from = scalar localtime($stime+($i*$interval));
    18    $to = scalar localtime($stime+(($i+1)*$interval)-1);
    19    printf("From: %-25s To: %-25s => %-s\n",$from,$to,substr($bucket[$i],1));
    20  }
$
$
$ perl bucket_divide.pl "5/24/2010 9:00" 30
From: Mon May 24 09:00:00 2010  To: Mon May 24 09:00:29 2010  => t
From: Mon May 24 09:00:30 2010  To: Mon May 24 09:00:59 2010  => m
From: Mon May 24 09:01:00 2010  To: Mon May 24 09:01:29 2010  =>
From: Mon May 24 09:01:30 2010  To: Mon May 24 09:01:59 2010  => a,i,v,x
From: Mon May 24 09:02:00 2010  To: Mon May 24 09:02:29 2010  => s
From: Mon May 24 09:02:30 2010  To: Mon May 24 09:02:59 2010  => k,y
From: Mon May 24 09:03:00 2010  To: Mon May 24 09:03:29 2010  =>
From: Mon May 24 09:03:30 2010  To: Mon May 24 09:03:59 2010  => o
From: Mon May 24 09:04:00 2010  To: Mon May 24 09:04:29 2010  => b,e,p,q,r
From: Mon May 24 09:04:30 2010  To: Mon May 24 09:04:59 2010  => d,g,l,z
From: Mon May 24 09:05:00 2010  To: Mon May 24 09:05:29 2010  => j
From: Mon May 24 09:05:30 2010  To: Mon May 24 09:05:59 2010  => c,f,n,w
From: Mon May 24 09:06:00 2010  To: Mon May 24 09:06:29 2010  => h,u
$
$
$ perl bucket_divide.pl "5/24/2010 9:00" 40
From: Mon May 24 09:00:00 2010  To: Mon May 24 09:00:39 2010  => m,t
From: Mon May 24 09:00:40 2010  To: Mon May 24 09:01:19 2010  =>
From: Mon May 24 09:01:20 2010  To: Mon May 24 09:01:59 2010  => a,i,v,x
From: Mon May 24 09:02:00 2010  To: Mon May 24 09:02:39 2010  => s
From: Mon May 24 09:02:40 2010  To: Mon May 24 09:03:19 2010  => k,y
From: Mon May 24 09:03:20 2010  To: Mon May 24 09:03:59 2010  => o
From: Mon May 24 09:04:00 2010  To: Mon May 24 09:04:39 2010  => b,e,p,q,r
From: Mon May 24 09:04:40 2010  To: Mon May 24 09:05:19 2010  => d,g,j,l,z
From: Mon May 24 09:05:20 2010  To: Mon May 24 09:05:59 2010  => c,f,n,w
From: Mon May 24 09:06:00 2010  To: Mon May 24 09:06:39 2010  => h,u
$
$
$

tyler_durden
# 3  
Old 09-23-2010
the script is working fine....but i have another problem....suppose the user does not give the start time we are supposed to take the least time in the file.can someone help me as to how to proceed.
# 4  
Old 09-23-2010
you can define a var with date

Code:
var1=$(date "+%m/%d/%Y %H:%M")

# 5  
Old 09-23-2010
i tried with that approach but now another prob

i may have 100's of such files to process....

the script looks good for one file....when i try to do it for all files i have a confusion as to how to take the date because the least time in the first file need not be the least in other files also....but one thing i can tell is the first record of the file has the least time.....so how better can i change the above code to implement this scenario.
# 6  
Old 09-23-2010
Quote:
Originally Posted by niteesh_!7
...
i may have 100's of such files to process....

the script looks good for one file....when i try to do it for all files i have a confusion as to how to take the date because the least time in the first file need not be the least in other files also....but one thing i can tell is the first record of the file has the least time.....so how better can i change the above code to implement this scenario.
If storage space is not an issue then combine all files, sort by the date, redirect to a file and try the logic on this file, where start time is on the first record.

If memory is not an issue then glob through your files pushing all records to an array, sort this array to get start time, loop through it and populate "bucket" array.

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

.csv.gz file extract errors using Winzip

HI All, Currently, i am working on bash shell. I have generated the csv file via shell script and because of high size , i have zip the csv file and send it to the outlook in windows. filename : viswa.csv.gz i can able to receive the mail successfully , But i am not able to view the... (4 Replies)
Discussion started by: venkatviswa
4 Replies

2. Shell Programming and Scripting

How to extract start/end times from log file to CSV file?

Hi, I have a log file (log.txt) that which contains lines of date/time. I need to create a script to extract a CSV file (out.csv) that gets all the sequential times (with only 1 minute difference) together by stating the start time and end time of this period. Sample log file (log.txt) ... (7 Replies)
Discussion started by: Mr.Zizo
7 Replies

3. Shell Programming and Scripting

Trying extract from text file and convert csv

I want to extract IP address, system ID and engine IDs of this file ( marked in red) and put in a csv. E.g. 1.1.1.1, SYSTEMID, 000012345678981123548912 I get these file by running an expect script from solaris. Here is the text file output of my expect script. working on 1.1.1.1 SNMP... (5 Replies)
Discussion started by: pbshillong
5 Replies

4. Shell Programming and Scripting

Extract data from XML file and write in CSV file

Hi friend i have input as following XML file <?xml version="1.0"?> <Document xmlns="urn:iso:std:iso:20022:tech:xsd:camt.054.001.02"> <BkToCstmrDbtCdtNtfctn> <GrpHdr><MsgId>LBP-RDJ-TE000000-130042430010001001</MsgId><CreDtTm>2013-01-04T03:21:30</CreDtTm></GrpHdr>... (3 Replies)
Discussion started by: mohan sharma
3 Replies

5. Shell Programming and Scripting

extract data in a csv file based on a certain field.

I have a csv file that I need to extract some data from depending on another field after reading info from another text file. The text file would say have 592560 in it. The csv file may have some data like so Field 1 Field2 Field3 Field4 Field5 Field6 20009756 1 ... (9 Replies)
Discussion started by: GroveTuckey
9 Replies

6. Shell Programming and Scripting

Extract last cell of csv file

How do I extract the last cell in a column of a csv file using linux shell scripting? Or alternatively, how do I get the number of cells of a csv file? (2 Replies)
Discussion started by: locoroco
2 Replies

7. UNIX for Dummies Questions & Answers

How to extract one column from csv file in perl?

Hi everyone, i am new to perl programming, i have a problem in extracting single column from csv file. the column is the 20th column, please help me.. at present i use this code #!C:/perl/bin use warnings; use strict; my $file1 = $ARGV; open FILE1, "<$file1" or die "Can't... (13 Replies)
Discussion started by: kvth
13 Replies

8. Shell Programming and Scripting

How to extract data from csv file

Hello everybody, Here is my problem, I don't know anything about shell programming and my boss is actually asking me to develop a shell script in order to get values in a csv file from a specific date. Here is a sample of the csv file : Date;Enchaînement;Titre;Libellé ;calendrier;Heure début;Heure... (11 Replies)
Discussion started by: freyr
11 Replies

9. Shell Programming and Scripting

Extract data from an XML file & write into a CSV file

Hi All, I am having an XML tag like: <detail sim_ser_no_1="898407109001000090" imsi_1="452070001000090"> <security>ADM1=????</security> <security>PIN1=????</security> <security>PIN2=????</security> ... (2 Replies)
Discussion started by: ss_ss
2 Replies

10. Shell Programming and Scripting

extract .csv file

Hi all I am new to unix . I need to write a script that extracts some data from oracle into a .csv file with heading of the columns in the file SO i created the following two scripts but they are not working ac.sql (this is the sql file that i will call inside the shell script when i run... (1 Reply)
Discussion started by: rajesh_tns
1 Replies
Login or Register to Ask a Question