Sponsored Content
Top Forums Shell Programming and Scripting Need to optimize the loop to load parallelly Post 302997895 by Mannu2525 on Monday 22nd of May 2017 07:07:25 AM
Old 05-22-2017
Need to optimize the loop to load parallelly

Hi,

I'm trying to load the huge amount of records in the database. I did the successful load but it took more time to load as numbers are huge.

Here what I have -

1. create a database table (t) with 2 columns- Not an issue

2. create a script to load huge amount of data - Here I would like to the parallel execution for faster load.

Code:
perl -e 'for ($i=0; $i<=4294968230; $i++) { printf "%d,%d,\n", $i,$i*10; }' | vsql -p $PORT -c "COPY t FROM STDIN  DELIMITER ',' direct";

The main command that does the load for me is
Code:
perl -e 'for ($i=0; $i<=4294968230; $i++) { printf "%d,%d,\n", $i,$i*10; }'

I would like my load to be faster, either by running this parallelly or using some other technique.

Please suggest something and let me know the best possible way to optimize it.

Thanks,
Mannu

Last edited by Mannu2525; 05-22-2017 at 09:01 AM.. Reason: To rectify the number of columns used
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Running three scripts parallelly

Hi All, We have three shell script batch, which extract data from three different systems(oracle, db2, db2/400). By running each shell script batch, the data is extracted from respective systems. while the batch is running, job date, system_name, start_date and end_date will be inserted into... (1 Reply)
Discussion started by: anwarsait
1 Replies

2. Shell Programming and Scripting

Need help in wrting Load Script for a Load-Resume type of load.

hi all need your help. I am wrting a script that will load data into the table. then on another load will append the data into the existing table. Regards Ankit (1 Reply)
Discussion started by: ankitgupta
1 Replies

3. Shell Programming and Scripting

pl help me to Optimize the given code

Pl help to me to write the below code in a simple way ... i suupose to use this code 3 to 4 places in my makefile(gnu) .. **************************************** @for i in $(LIST_A); do \ for j in $(LIST_B); do\ if ;then\ echo "Need to sign"\ echo "List A = $$i , List B =$$j"\ ... (2 Replies)
Discussion started by: pk_arun
2 Replies

4. UNIX for Dummies Questions & Answers

VPS has load 200, httpd load no activity, netstat nothing

Hello, on my hostserver i see one VPS of mine got load of 200.00 and netstat nothing (not a single blank line on netstat command) after some time, netstat started showing connections, but i see no excessive IP connections. tail -f /var/log/httpd/access_log shows no activity /var/log/messages ;... (1 Reply)
Discussion started by: postcd
1 Replies

5. Shell Programming and Scripting

Search patterns in multiple logs parallelly.

Hi All, I am starting a service which will redirect its out put into 2 logs say A and B. Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly. Now my requirement is to find the patterns in the increasing logs A and B... (19 Replies)
Discussion started by: Girish19
19 Replies

6. Shell Programming and Scripting

Executes scripts parallelly based on their success

Hi Team , I have one Master.sh file which call X,Y,Z scripts , but here X may call again some sub scripts X_sub1.sh , X_sub2.sh Y calls Y_sub1.sh,Y_sub2.sh and similarly Z script also . Now requirement is Both X and Y should execute parallel bcz X and Y are independent... (9 Replies)
Discussion started by: chandini
9 Replies

7. Shell Programming and Scripting

To execute scripts parallelly

Hi I have set two set of scripts sets in a file which perform similar operations but with different script names for e.g.: 1st set of script1.txt: 1.sh 2.sh 3.sh 4.sh 2nd set of script2.txt: 1_1.sh 2_1.sh 3_3.sh 4_4.sh I want to execute these set of scripts parallelly in such... (16 Replies)
Discussion started by: rohit_shinez
16 Replies

8. Shell Programming and Scripting

Optimize my mv script

Hello, I'm wondering if there is a quicker way of doing this. Here is my mv script. d=/conversion/program/out cd $d ls $d > /home/tempuser/$$tmp while read line ; do a=`echo $line|cut -c1-5|sed "s/_//g"` b=`echo $line|cut -c16-21` if ;then mkdir... (13 Replies)
Discussion started by: whegra
13 Replies

9. Shell Programming and Scripting

Looking to optimize code

Hi guys, I feel a bit comfortable now doing bash scripting but I am worried that the way I do it is not optimized and I can do much better as to how I code. e.g. I have a whole line in a file from which I want to extract some values. Right now what I am doing is : STATE=`cat... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies
lh_stats(3)							      OpenSSL							       lh_stats(3)

NAME
lh_stats, lh_node_stats, lh_node_usage_stats, lh_stats_bio, lh_node_stats_bio, lh_node_usage_stats_bio - LHASH statistics SYNOPSIS
#include <openssl/lhash.h> void lh_stats(LHASH *table, FILE *out); void lh_node_stats(LHASH *table, FILE *out); void lh_node_usage_stats(LHASH *table, FILE *out); void lh_stats_bio(LHASH *table, BIO *out); void lh_node_stats_bio(LHASH *table, BIO *out); void lh_node_usage_stats_bio(LHASH *table, BIO *out); DESCRIPTION
The LHASH structure records statistics about most aspects of accessing the hash table. This is mostly a legacy of Eric Young writing this library for the reasons of implementing what looked like a nice algorithm rather than for a particular software product. lh_stats() prints out statistics on the size of the hash table, how many entries are in it, and the number and result of calls to the routines in this library. lh_node_stats() prints the number of entries for each 'bucket' in the hash table. lh_node_usage_stats() prints out a short summary of the state of the hash table. It prints the 'load' and the 'actual load'. The load is the average number of data items per 'bucket' in the hash table. The 'actual load' is the average number of items per 'bucket', but only for buckets which contain entries. So the 'actual load' is the average number of searches that will need to find an item in the hash table, while the 'load' is the average number that will be done to record a miss. lh_stats_bio(), lh_node_stats_bio() and lh_node_usage_stats_bio() are the same as the above, except that the output goes to a BIO. RETURN VALUES
These functions do not return values. SEE ALSO
bio(3), lhash(3) HISTORY
These functions are available in all versions of SSLeay and OpenSSL. This manpage is derived from the SSLeay documentation. 1.0.1e 2013-02-11 lh_stats(3)
All times are GMT -4. The time now is 03:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy