09-10-2014
Here is my input file "feed.txt" (first row is a header with column names. I have pasted dummy column values for 3 rows)
url;image;id;showcase_id;showcase_name;showcase_zip_code;vehicle_id;carmodel_id;make_id;external_mpn ;carmodel_name;make_name;colour;colour_alias;colour_secondary_name;colour_secondary_alias;trim;catal og_price;customer_bonus;reprise_bonus;legal_notice;rebate_legal_notice;eco_bonus;price;customer_bene fit_percent;options;code;name;matriculation_on;status;mileage;vehicle_acceleration_0_100kph;vehicle_ air_conditioning;vehicle_body_type;vehicle_co2_emission_level;vehicle_combined_fuel_economy;vehicle_ cylinders_count;vehicle_doors_count;vehicle_driven_wheels;vehicle_emission_standard;vehicle_engine_c apacity;vehicle_extra_urban_fuel_economy;vehicle_fiscal_horse_power;vehicle_height;vehicle_kerbweigh t;vehicle_length;vehicle_max_load_capacity;vehicle_name;vehicle_primary_fuel_tank_capacity;vehicle_p rimary_fuel_type;vehicle_published_hp_metric;vehicle_seats_count;vehicle_secondary_fuel_type;vehicle _type;vehicle_luggage_capacity;vehicle_tvs_tax;vehicle_transmission_shiftable;vehicle_trim_name;vehi cle_trim_code;vehicle_urban_fuel_economy;vehicle_vehicle_warranty_kms;vehicle_vehicle_warranty_month s;vehicle_width_excluding_mirrors;vehicle_transmission_type;vehicle_engine_name;vehicle_engine_code; vehicle_rsi_id;image_1;image_2;image_3;image_4;image_5
url_1;image_1;id_1;showcase_id_1;showcase_name_1;showcase_zip_code_1;vehicle_id_1;carmodel_id_1;make _id_1;external_mpn_1;carmodel_name_1;make_name_1;colour_1;colour_alias_1;colour_secondary_name_1;col our_secondary_alias_1;trim_1;catalog_price_1;customer_bonus_1;reprise_bonus_1;legal_notice_1;rebate_ legal_notice_1;eco_bonus_1;price_1;customer_benefit_percent_1;options_1;code_1;name_1;matriculation_ on_1;status_1;updated_at_1;created_at_1;mileage_1;vehicle_acceleration_0_100kph_1;vehicle_air_condit ioning_1;vehicle_body_type_1;vehicle_co2_emission_level_1;vehicle_combined_fuel_economy_1;vehicle_cy linders_count_1;vehicle_doors_count_1;vehicle_driven_wheels_1;vehicle_emission_standard_1;vehicle_en gine_capacity_1;vehicle_extra_urban_fuel_economy_1;vehicle_fiscal_horse_power_1;vehicle_height_1;veh icle_kerbweight_1;vehicle_length_1;vehicle_max_load_capacity_1;vehicle_name_1;vehicle_primary_fuel_t ank_capacity_1;vehicle_primary_fuel_type_1;vehicle_published_hp_metric_1;vehicle_seats_count_1;vehic le_secondary_fuel_type_1;vehicle_type_1;vehicle_luggage_capacity_1;vehicle_tvs_tax_1;vehicle_transmi ssion_shiftable_1;vehicle_trim_name_1;vehicle_trim_code_1;vehicle_urban_fuel_economy_1;vehicle_vehic le_warranty_kms_1;vehicle_vehicle_warranty_months_1;vehicle_width_excluding_mirrors_1;vehicle_transm ission_type_1;vehicle_engine_name_1;vehicle_engine_code_1;vehicle_rsi_id_1;image_1_1;image_2_1;image _3_1;image_4_1;image_5_1
url_2;image_2;id_2;showcase_id_2;showcase_name_2;showcase_zip_code_2;vehicle_id_2;carmodel_id_2;make _id_2;external_mpn_2;carmodel_name_2;make_name_2;colour_2;colour_alias_2;colour_secondary_name_2;col our_secondary_alias_2;trim_2;catalog_price_2;customer_bonus_2;reprise_bonus_2;legal_notice_2;rebate_ legal_notice_2;eco_bonus_2;price_2;customer_benefit_percent_2;options_2;code_2;name_2;matriculation_ on_2;status_2;updated_at_2;created_at_2;mileage_2;vehicle_acceleration_0_100kph_2;vehicle_air_condit ioning_2;vehicle_body_type_2;vehicle_co2_emission_level_2;vehicle_combined_fuel_economy_2;vehicle_cy linders_count_2;vehicle_doors_count_2;vehicle_driven_wheels_2;vehicle_emission_standard_2;vehicle_en gine_capacity_2;vehicle_extra_urban_fuel_economy_2;vehicle_fiscal_horse_power_2;vehicle_height_2;veh icle_kerbweight_2;vehicle_length_2;vehicle_max_load_capacity_2;vehicle_name_2;vehicle_primary_fuel_t ank_capacity_2;vehicle_primary_fuel_type_2;vehicle_published_hp_metric_2;vehicle_seats_count_2;vehic le_secondary_fuel_type_2;vehicle_type_2;vehicle_luggage_capacity_2;vehicle_tvs_tax_2;vehicle_transmi ssion_shiftable_2;vehicle_trim_name_2;vehicle_trim_code_2;vehicle_urban_fuel_economy_2;vehicle_vehic le_warranty_kms_2;vehicle_vehicle_warranty_months_2;vehicle_width_excluding_mirrors_2;vehicle_transm ission_type_2;vehicle_engine_name_2;vehicle_engine_code_2;vehicle_rsi_id_2;image_1_2;image_2_2;image _3_2;image_4_2;image_5_2
url_3;image_3;id_3;showcase_id_3;showcase_name_3;showcase_zip_code_3;vehicle_id_3;carmodel_id_3;make _id_3;external_mpn_3;carmodel_name_3;make_name_3;colour_3;colour_alias_3;colour_secondary_name_3;col our_secondary_alias_3;trim_3;catalog_price_3;customer_bonus_3;reprise_bonus_3;legal_notice_3;rebate_ legal_notice_3;eco_bonus_3;price_3;customer_benefit_percent_3;options_3;code_3;name_3;matriculation_ on_3;status_3;updated_at_3;created_at_3;mileage_3;vehicle_acceleration_0_100kph_3;vehicle_air_condit ioning_3;vehicle_body_type_3;vehicle_co2_emission_level_3;vehicle_combined_fuel_economy_3;vehicle_cy linders_count_3;vehicle_doors_count_3;vehicle_driven_wheels_3;vehicle_emission_standard_3;vehicle_en gine_capacity_3;vehicle_extra_urban_fuel_economy_3;vehicle_fiscal_horse_power_3;vehicle_height_3;veh icle_kerbweight_3;vehicle_length_3;vehicle_max_load_capacity_3;vehicle_name_3;vehicle_primary_fuel_t ank_capacity_3;vehicle_primary_fuel_type_3;vehicle_published_hp_metric_3;vehicle_seats_count_3;vehic le_secondary_fuel_type_3;vehicle_type_3;vehicle_luggage_capacity_3;vehicle_tvs_tax_3;vehicle_transmi ssion_shiftable_3;vehicle_trim_name_3;vehicle_trim_code_3;vehicle_urban_fuel_economy_3;vehicle_vehic le_warranty_kms_3;vehicle_vehicle_warranty_months_3;vehicle_width_excluding_mirrors_3;vehicle_transm ission_type_3;vehicle_engine_name_3;vehicle_engine_code_3;vehicle_rsi_id_3;image_1_3;image_2_3;image _3_3;image_4_3;image_5_3
Here is my declaration file containing the predefined header (columns)
url;image;id;showcase_id;showcase_name;showcase_zip_code;vehicle_id;carmodel_id;make_id;external_mpn ;carmodel_name;make_name;colour;colour_alias;colour_secondary_name;colour_secondary_alias;trim;catal og_price;customer_bonus;reprise_bonus;legal_notice;rebate_legal_notice;eco_bonus;price;customer_bene fit_percent;options;code;name;matriculation_on;status;mileage;vehicle_acceleration_0_100kph;vehicle_ air_conditioning;vehicle_body_type;vehicle_co2_emission_level;vehicle_combined_fuel_economy;vehicle_ cylinders_count;vehicle_doors_count;vehicle_driven_wheels;vehicle_emission_standard;vehicle_engine_c apacity;vehicle_extra_urban_fuel_economy;vehicle_fiscal_horse_power;vehicle_height;vehicle_kerbweigh t;vehicle_length;vehicle_max_load_capacity;vehicle_name;vehicle_primary_fuel_tank_capacity;vehicle_p rimary_fuel_type;vehicle_published_hp_metric;vehicle_seats_count;vehicle_secondary_fuel_type;vehicle _type;vehicle_luggage_capacity;vehicle_tvs_tax;vehicle_transmission_shiftable;vehicle_trim_name;vehi cle_trim_code;vehicle_urban_fuel_economy;vehicle_vehicle_warranty_kms;vehicle_vehicle_warranty_month s;vehicle_width_excluding_mirrors;vehicle_transmission_type;vehicle_engine_name;vehicle_engine_code; vehicle_rsi_id;image_1;image_2;image_3;image_4;image_5
between two files the difference was, "two new columns "updated_at" and "created_at" (updated_at;created_at) were added in between by the data provider, without giving any notice", due to which the data loading got messed up one day. Similar problem is happening frequently.
So i have planned to alert when the new columns get added and at the same time load the data without any issues by removing the newly added columns and it's relevant values in each row. So that we can add the new columns in the database and also redefine in the declaration file for considering the newly added columns in a planned manner.
Thanks & regards
Praveen
Last edited by Praveen Pandit; 09-10-2014 at 02:27 PM..
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi all,
I want to create column names in a flat file and then load the data through some other application. For example, I have a file with emp.txt and I need column names as eno,ename,sal in the first line. The delimiter here is comma and record delimiter is end of line or unix new line. Could... (1 Reply)
Discussion started by: srivsn
1 Replies
2. Shell Programming and Scripting
Hi,
I want to write a script which will compare the 1st column of both the files and will give the difference.
e.g:-
my 1st file contains:
89 /usr
52 /usr/local
36 /tmp
92 /opt
96 /home
27 /etc/opt/EMCom
1 ... (3 Replies)
Discussion started by: adityam
3 Replies
3. Shell Programming and Scripting
Hi All,
i have a file that is tab delimited. i need help to find the rows which are having same price based on the site code but some times, there are difference so i need to find only the records which are different in all site code.
Dept Sec Barcode 10001 10002 10003 10004... (1 Reply)
Discussion started by: malcomex999
1 Replies
4. Shell Programming and Scripting
Hi All,
I have a file example.csv which looks like this
GrpID,TargetID,Signal,Avg_Num
CSCH74_1_1,2007,61,256
CSCH74_1_1,212007,647,679
CSCH74_1_1,12007,3,32
CSCH74_1_1,207,299,777
I want the output as
GrpID,TragetID,Signal-CSCH74_1_1,Avg_Num
CSCH74_1_1,2007,61,256... (4 Replies)
Discussion started by: Vavad
4 Replies
5. UNIX for Dummies Questions & Answers
Hi All,
I have a file example.csv which looks like this
GrpID,TargetID,Signal,Avg_Num
CSCH74_1_1,2007,61,256
CSCH74_1_1,212007,647,679
CSCH74_1_1,12007,3,32
CSCH74_1_1,207,299,777
I want the output as
GrpID,TragetID,Signal-CSCH74_1_1,Avg_Num
CSCH74_1_1,2007,61,256... (1 Reply)
Discussion started by: Vavad
1 Replies
6. UNIX for Dummies Questions & Answers
I have a table, say this:
name1 num1 num2 num3 num4
name2 num5 num6 num7 num8
name3 num1 num3 num4 num9
name2 num8 num9 num1 num2
name2 num4 num5 num6 num4
name4 num4 num5 num7 num8
name5 num1 num3 num9 num7
name5 num6 num8 num3 num4
I want a code that will sort my data according... (4 Replies)
Discussion started by: FelipeAd
4 Replies
7. Shell Programming and Scripting
Hi , i am newbie to shell scripting and am trying to do the below job,
A shell script to be run with a command like
sh Compare.ksh file1.txt file2.txt 1 2 > file3.txt
1 2-are the key columns
Consider the delimiter would be Tab or comma
File 1:
SK TEST NAME MATHS PHYSICS
21 1... (1 Reply)
Discussion started by: shakthi666
1 Replies
8. Homework & Coursework Questions
Hi , i am newbie to shell scripting and am trying to do the below job,
A shell script to be run with a command like
sh Compare.ksh file1.txt file2.txt 1 2 > file3.txt
1 2-are the key columns
Consider the delimiter would be Tab or comma
File 1:
SK TEST NAME MATHS PHYSICS
21 1 AAA... (1 Reply)
Discussion started by: shakthi666
1 Replies
9. Shell Programming and Scripting
File 1:
20130416,235800,10.78.25.104,BR2-loc,60.0,1624,50.0,0,50.0,0
20130416,235800,10.78.25.104,BR1-LOC,70.0,10,50.0,0,70.0,0
20130416,235800,10.78.25.104,Hub_None,60.0,15,60.0,0,50.0,0
File 2:
20130417,000200,10.78.25.104,BR2-loc,60.0,1626,50.0,0,50.0,0... (3 Replies)
Discussion started by: Lakshmikumari
3 Replies
10. Shell Programming and Scripting
In a bash shell, I have to prefix a variable to two .CSV files File1.CSV and File2.CSV. One of the files has a header and the other one is with no header in the below format:
"value11","value12","value13","value14","value15","value16"
"value21","value22","value23","value24","value25","value26"... (7 Replies)
Discussion started by: dhruuv369
7 Replies