Shell Programming and Scripting

I have a main ksh script which takes several huge tables
and translate into individual csv files for processing
1. A run an sqlplus script that translates each oracle table into a csv file
2 next, python script verify/split the large csv file into smaller files for processing
2.Then. another shell scripts find split files and submits each line of the split file for processing by another script. I'm parallel processing with use of xargs -n -p (I need to find a way where some lines are not skipped)
3. this shell script processes each line as input to a restful service
4. The output json response is processed in a python script which translates it into an sql insert statements
5. i group the sql insert statements by id and then consolidate all ids under one main id in the main script
6. perform a bulk insert of the main id in sqlplus
i'm trying to process as many files as possible

its currently taking about 10 minutes to process 10k with submission of 100 split files with 75 lines for each files submitted to the service. I crashed the server with this amount of processes and some lines were skipped.

I trying to do it in as little time as possible.
gnu parallel is not available for us.
Any assistance would be greatly appreciated.
I have not coded any years and this is my first project
I have time constraints i need some serious help ASAP

Thanks for reading

Welcome to the forum.

Confused.
You're processing an incredible sequence of different programs to cast a set of DB tables into another (set of) DB tables?
Don't you think that could be done sort of directly DB to DB at tremendously increased speed? BTW, don't expect too much from "parallel processing" as parallel processes will compete for DB and system resources...
Please post more details of the problem, e.g. data structures, algorithms, scripts to get a (hopefully different) persective of the task.

ASAP, btw, is a term highly deprecated in these fora. We're all volunteers who try to help alongside a regular occupation.

So, can you tell us the database versions at each end?

As RudiC suggests, it might be far better to get the database(s) to do the work for you. Can you consider a few options:-
You could help us by explaining the database versions for the source and the target and the Operating systems you're dealing with. We might be able to untangle the multi-step process you have. The fewer steps, then (usually) the better the performance. The best processing will likely be done on the source database to extract the data you want in a way that can be loaded directly, but we won't know unless you tell us what is needed.



Sorry about the questions, but I'm sure we can help if we have a better idea of what's going on.


Kind regards,
Robin