|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Reformat MLS Data - Use AWK?
Post 302569800 by Chicago_Realtor on Tuesday 1st of November 2011 03:04:16 PM
11-01-2011
Reformat MLS Data - Use AWK?
I am helping my wife set up a real estate site and I am starting to integrate MLS listings. We are using a HostGator level 5 VPS running CentOS and have full root and SSH access to the VPS.
Thus far I have automated the daily FTP download of listings from our MLS server using a little sh script. It is a 90MB text file, pipe-delimited with about 150 fields.
Also, I have settled on using the IProperty component from TheThinkery which is a Joomla extension as the basis for the property search engine. IP uses a comma-delimited text file for input/output and has a different order of the fields.
Basically I just need to
1. convert pipe-delimited to comma-delimited and add double-quotes for all text fields.
2. re-order the fields.
Is AWK or Perl the way to go? I have confirmed AWK is installed. I used to have an O-Reilly AWK book fifteen years ago and actually read the whole thing believe it or not, but I haven't written a line of AWK in probably 10 years.
Would someone be willing to help me with this for a fee? I assume if you know what you're doing it would take less than an hour. I will do the grunt work re-ordering fields.
By the way, I came across this forum when searching and landed on this thread --
"Convert CSV file (with double quoted strings) to pipe delimited file"
That's sort of like what I want to do except the opposite. (Also I need to re-order fields).
Thanks!
Eric
Thus far I have automated the daily FTP download of listings from our MLS server using a little sh script. It is a 90MB text file, pipe-delimited with about 150 fields.
Also, I have settled on using the IProperty component from TheThinkery which is a Joomla extension as the basis for the property search engine. IP uses a comma-delimited text file for input/output and has a different order of the fields.
Basically I just need to
1. convert pipe-delimited to comma-delimited and add double-quotes for all text fields.
2. re-order the fields.
Is AWK or Perl the way to go? I have confirmed AWK is installed. I used to have an O-Reilly AWK book fifteen years ago and actually read the whole thing believe it or not, but I haven't written a line of AWK in probably 10 years.
Would someone be willing to help me with this for a fee? I assume if you know what you're doing it would take less than an hour. I will do the grunt work re-ordering fields.
By the way, I came across this forum when searching and landed on this thread --
"Convert CSV file (with double quoted strings) to pipe delimited file"
That's sort of like what I want to do except the opposite. (Also I need to re-order fields).
Thanks!
Eric
|
|
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I am trying to write an awk program to reformat a data table and convert the date to julian time. I have all the individual steps working, but I am having some issues joing them into one program. Can anyone help me out? Here is my code so far:
# This is an awk program to convert the dates from... (4 Replies)
Discussion started by: climbak
4 Replies
2. Shell Programming and Scripting
I am new to Perl. I need to reformat a data file as the last part of a script I am working on. I am stuck on this.
Here is the current format:
CUSTOMER Filename 09/04/07-08:49
CUSTOMER Filename 09/04/07-08:52
CUSTOMER Filename 09/04/07-08:52
CUSTOMER2 Filename 09/04/07-08:49
CUSTOMER2... (3 Replies)
Discussion started by: flood
3 Replies
3. Shell Programming and Scripting
Can anyone help me with a shell script that can do the following:
I have a data in fasta format (first line is the header, followed by a sequence of characters).
>ALLLY
GGCCCCTCGAGCCTCGAACCGGAACCTCCAAATCCGAGACGCTCTGCTTATGAGGACCTC
GAAATATGCCGGCCAGTGAAAAAATCTTGTGGCTTTGAGGGCTTTTGGTTGGCCAGGGGC... (5 Replies)
Discussion started by: manishabh
5 Replies
4. Shell Programming and Scripting
I have a file which have data like
A.txt
a
1Jan I am in a1.
1Jan I was born.
2Jan I am here.
3Jan I am in a3.
b
1Jan I am in b1.
c
2Jan I am in c2.
d
2Jan I am in d2.
5jan I am in d5.
date in the file might be vary evertime. (9 Replies)
Discussion started by: samkhu
9 Replies
5. Shell Programming and Scripting
input file:
hsa-miR-4726-5p
Score
hsa-miR-483-5p
Score
hsa-miR-125b-2*
Score
hsa-miR-4492
hsa-miR-4508
hsa-miR-4486
Score
Desired output file:
hsa-miR-4726-5p Score hsa-miR-483-5p Score hsa-miR-125b-2* Score hsa-miR-4492 hsa-miR-4508 hsa-miR-4486 Score ... (6 Replies)
Discussion started by: perl_beginner
6 Replies
6. Shell Programming and Scripting
Input file:
58227131
50087390
57339526
40578034
65348841
55614853
64363217
44178559
Desired output file:
58227131 50087390
57339526 40578034
65348841 55614853
64363217 44178559
Command that I try: (4 Replies)
Discussion started by: perl_beginner
4 Replies
7. Shell Programming and Scripting
Input file:
dependent general_process
dependent general_process
regulation general_process
- -
template component
food component
binding data_rearrangement
binding data_rearrangement
specific_activity data_rearrangement
- ... (7 Replies)
Discussion started by: cpp_beginner
7 Replies
8. Shell Programming and Scripting
Input file:
bv|111259484|pir||T49736_real_data
bv|159484|pir||T9736_data_figure
bv|113584|prf|T4736|truth
bv|113584|pir||T4736_truth
Desired output:
bv|111259484|pir|T49736|real_data
bv|159484|pir|T9736|data_figure
bv|113584|prf|T4736|truth
bv|113584|pir|T4736|truth
Once the... (8 Replies)
Discussion started by: perl_beginner
8 Replies
9. Shell Programming and Scripting
Input file
4CL1 O24145 CoA1
4CL1 P31684 CoA1
4CL1 Q54P77 CoA_1
73 O36421 Unknown
4CL3 Q9S777 coumarate
4CL3 Q54P79 coumarate
4CL3 QP7932 coumarate
Desired output result
4CL1 O24145#P31684 CoA1
4CL1 Q54P77 CoA_1
73 O36421 Unknown
4CL3 Q9S777#Q54P79#QP7932 coumarate
I... (5 Replies)
Discussion started by: perl_beginner
5 Replies
10. Shell Programming and Scripting
The below awk improved bu @MadeInGermany, works great as long as the input file has data in it in the below format:
input
chrX 25031028 25031925 chrX:25031028-25031925 ARX 631 18
chrX 25031028 25031925 chrX:25031028-25031925 ARX 632 14... (3 Replies)
Discussion started by: cmccabe
3 Replies
LEARN ABOUT DEBIAN
dtc_kill_vps_disk
dtc_kill_vps_disk(8) System Manager's Manual dtc_kill_vps_disk(8) NAME
dtc_kill_vps_disk - destroy LVM disks for freeing space SYNOPSIS
dtc_kill_vps_disk VPS_NUMBER [ lvm|loopback ] DESCRIPTION
dtc_kill_vps_disk This shell script is a part of the dtc-xen package that is to be used by the dtc panel to manage a Xen VPS server. This script is used to delete a partitions that was used by a VPS. It will delete the normal partition that was used as a root disk, and the swap partition. dtc_kill_vps_disk is to be used by the dtc-xen SOAP server, but can also be used as a standalone userland tool. OPTIONS
VPS_NUMBER has to be a number between 01 and 19. Let's say the number is 16, and that dtc-xen is configured to use the volume group called lvm1, then this script will create /dev/lvm1/xen16 and /dev/lvm1/xen16swap. IMAGE_TYPE This parameter can have 2 values: either lvm or loopback. If ommited, then lvm is used. If lvm is used, then this script will create a partition using lvmcreate, otherwise it will create an image file. Both will later be used for the VPS. EXAMPLE
dtc_setup_vps_disk 04 lvm This will delete the 2 LVM partitions that were used by the VPS named xen04, the first one being the root partition, and the second one being the swap partition. VERSION
This documentation describes dtc_kill_vps_disk See http://www.gplhost.com/software-dtc-xen.html for updates. SEE ALSO
dtc_reinstall_os(8), dtc_setup_vps_disk(8) dtc_kill_vps_disk(8)