I'm doing a project now and extracting tables from a webpage to MySQL table format.
I dumped the webpage with lynx and it is like this
Code:
id
Spec
524543
Developed especially for seniors
Spec
No
Java
Spec
Yes
Java MIDP
Spec
Available using application
Operating system
Spec
Android 2.3 (Gingerbread)
Qwerty keyboard (buttons)
Spec
No
Qwerty keyboard (screen)
Spec
Yes
SAR
Spec
0.34 W/kg
Stand by time
Spec
710 h
Talk time
Spec
18.2 h
Talk time (3G)
Spec
8.4 h
Stand By-Time (3G)
Spec
610 h
Audio features
Built-in speakers
Spec
Mono
Camera
Built-in camera
Spec
Yes
Auto focus
Spec
Yes
Built-in flash
Spec
Yes
Flash type
Spec
LED-flash
it is specification of a cellphone. And I got the line number of each 'Spec' word with grep cut and sed and it is like this:
Code:
2
5
8
11
14
17
20
23
26
29
32
35
39
43
46
49
52
now I want to get just the lines after each 'Spec' word and put in the MySQL format like this:
Code:
INSERT INTO `specifications` VALUES (524543,'No','Yes','Available using application','Android 2.3 (Gingerbread)',...);
I know that I need to get the lines with number +1 and put them between ',' in one line, but this part was very hard for me could you please help me about that?
You could use the DBI and DBD::mysql modules to add the values directly to the database.
WARNING CODE UNTESTED!!
Code:
#!/usr/bin/perl
use strict;
use warnings;
open (my $data, "<", $ARGV[0]); #Supply the spec file as the argument
my $after_spec=0;
my @values;
while(<$data>){
chomp;
if ($after_spec){
push @values $_;
$after_spec--;
}
elsif (/^Spec$/){
$after_spec++;
}
}
for (@values){ #add quotes to non numerical values and escape single quotes
s/'/\\'/g;
if (! /\d+(\.\d+)/){
$_="'".$_."'";
}
}
$values_string=join(', ',@values);
print "INSERT INTO `specifications` VALUES ($values);"
This User Gave Thanks to Skrynesaver For This Post:
$ data2.sh
/usr/bin/data2.sh: line 2: use: command not found
/usr/bin/data2.sh: line 2: $'\r': command not found
/usr/bin/data2.sh: line 3: use: command not found
/usr/bin/data2.sh: line 3: $'\r': command not found
/usr/bin/data2.sh: line 4: syntax error near unexpected token `my'
/usr/bin/data2.sh: line 4: `open (my $data, "<", $temp1.dat[0]); #Supply the spec
' file as the argument
$ data2.sh
/usr/bin/data2.sh: line 2: use: command not found
....
use is a Perl command , try renaming the file data2.pl and then runchmod +x ./data2.pl to make the script runnable.
Quote:
Originally Posted by Johanni
I input the file like this:
Code:
open (my $data, "<", $temp1.dat[0]);
is it correct?
Probably not, the original script is intended to be runnable as follows...
Code:
./datas2.pl PATH_TO_ORIGINAL_DATA_DUMP_FROM_LYNX
$ARGV[0] is the first argument provided to the script on the command line.
And here's the code after fixing up the bugs.
Code:
#!/usr/bin/perl
#These two pragmas catch typos for you.
use strict;
use warnings;
open (my $data, "<", $ARGV[0]); #Supply the spec file as the argument
my $after_spec=0; #flag which is set if the line is a spec line.
my @values; # an array to hold all the lines that match
while(<$data>){ # loop through the file one line at a time
chomp; #remove trailing newlines.
if ($after_spec){ # if we set the flag on the previous line.
push @values, $_; # add this string to the values array
$after_spec--; # unset the flag.
}
elsif (/^Spec$/){ # if the line matches the pattern
$after_spec++; # set the flag
}
}
for (@values){ #go through the values array cleaning up the data.
s/^\s*(.+)\s*$/$1/;#remove leading and trailing spaces
s/'/\\'/g; # escape single quotes
if (! /^\d+(\.\d+)?$/){ # if this is not a number
$_="'".$_."'"; # surround the value with quotes
}
}
my $values_string=join(', ',@values); # create a value string
print "INSERT INTO `specifications` VALUES ($values_string);\n" # print the result.
tnx man for the reply, this seems very complete code, but it seems that my perl is not going to work,
I'm recieving this error now:
Code:
$ data2.pl
bash: /usr/bin/data2.pl: /bin/per1^M: bad interpreter: Permission denied
I,ve googled and they say to put exec in the fstab file in etc folder (I'm using cygwin)
but it didn't worked and I'm recieiving the sam error. Do you have any idea about that?
---------- Post updated at 10:07 PM ---------- Previous update was at 08:59 PM ----------
okay, I fixed the perl
now I'm receiving this:
Code:
INSERT INTO `specifications` VALUES ();
with no value
I've checked the code, it is reading the file, but I don't know what is the problem?
I input the file like this:
Code:
open (my $data, "<", "temp1.dat");
is it correct? I didn't understand the
Code:
$ARGV[0]
Could you please tell me how should I input the file?
tnx
---------- Post updated at 10:30 PM ---------- Previous update was at 10:07 PM ----------
Hi ,
i am generating some data by firing sql query with connecting to the database by my solaris box.
The below one should be the header line of my excel ,here its coming in separate row.
TO_CHAR(C. CURR_EMP_NO
---------- ---------------
LST_NM... (6 Replies)
Hi All,
I have the file in this format
**** Results Data ****
Time or Step
1
2
20
0.000000000e+00 0s 0s 0s
1.024000000e+00 Us 0s 0s
1.100000000e+00 1s 0s 0s
1.100000001e+00 1s 0s 1s
2.024000000e+00 Us Us 1s
2.024000001e+00 ... (7 Replies)
Good afternoon!
I have been lurking in this forum for awhile now. I have just recently started posting. I think this is a really good site. With that being said, I don't like to just run and get an answer before I try my best first.
I have poured some blood, sweat and tears into... (4 Replies)
hello everyone. newbie here in unix. I am trying to extract the logs of a certain job and would like to output it in a readable format, see below the CAT part:
cat /var/opt/ctma/ctm/sysout/idwesct_sh30_eng_r6_cdcs_sh.LOG_05l0du_000* | egrep -i 'orderid:|file_name=' | sed... (1 Reply)
Hi,
I have one file, say file 1, that has data like below where 19900107 is the date,
19900107 12 144 129 0.7380047
19900108 12 168 129 0.3149017
19900109 12 192 129 3.2766666E-02
... (3 Replies)
Hi all,
Does anyone know how to format the output from mysql from within a shell script?
i.e.
RESULT=`mysql command`
echo ${RESULT}
the ${RESULT} only displays the output on one line instead of how mysql would display it as columns etc (3 Replies)
I have a list of dates in the following format: mm/dd/yyyy and want to change these to the MySQL standard format: yyyy-mm-dd.
The dates in the original file may or may not be zero padded, so April is sometimes "04" and other times simply "4".
This is what I use to change the format:
sed -i '' -e... (2 Replies)
I have a comma delimited log file which has the date as MM/DD/YY in the 2nd column, and HH:MM:SS in the 3rd column.
I need to change the date format to YYYY-MM-DD and merge it with the the time HH:MM:SS. How will I got about this?
Sample input
02/27/09,23:52:31
02/27/09,23:52:52... (3 Replies)
Hi,
Could anyone help me in changing a tabular format output to comma seperated file pls in K-sh. Its very urgent.
E.g : username empid
------------------------
sri 123
to
username,empid
sri,123
Thanks,
Hema:confused: (2 Replies)