Load mutliple files into Netezza table


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Load mutliple files into Netezza table
# 1  
Old 02-18-2015
Load mutliple files into Netezza table

Hi All,
I get ~1000 files everyday and the current unix script process one file at a time... like it unzips the file, loads into table . This process will repeat how many ever files were present in the directory. I was wondering can we process multiple files (like unzipping and loading X amount of files.. for instance load 5 files parallel) at a time.

If needed I can post the script that I have.

Thanks,
Raja
# 2  
Old 02-18-2015
I dont know that program, but if you can load multiple instances of files at the same time, this might help:

As mentioned in [BASH] Script to manage background scripts (running, finished, exit code) I'd so much would like to say, install TUI then run:

Note, that the only reason i choose to use diffrent sub folders, is the keep the visual output of the tui-psm command, and i do think that 40 lines/scripts + statistics fill a full screen output, thus the limit to 40.

Within the folder those 10k files are:
Code:
BASE=/tmp/psm/
LINES=40 	# This many scripts will be generated per folder, NOT required, but for visual purpose
COUNT=0

for thisFile in *
do
	DIR=$(( $COUNT / $LINES ))
	target="$BASE/dir_$(( $COUNT / $LINES ))/script-$thisFile"
	[ -d "$(dirname $target)" ] || mkdir -p "$(dirname $target)"
	
	touch "$target"
	cat > "$target" << EOF
echo "tasks you do for each of the file $thisFile"
echo "Replace these two echo commands with your code..."
EOF
	chmod +x "$target"
done

cd "$BASE"
for thisDir in *
do
	cd "$thisDir"
	tui-psm -c *
	echo "Successfully ran: $? / $(ls|wc -l) scripts" && sleep 3
	cd ..
done

If you dont care about visual output/info, change LINES=40 to LINES=0, and change tui-psm -c * to tui-psm -qc * for quiet mode.
Keep the -c, so echo $? will still tell you how many scripts exited with 0 (success).

Other than this self advertisement, here are some search result to handle the generated script files other than with tui-psm.

SEARCH: run scripts simultaneouslySEARCH: run scripts paralellSEARCH: run scripts in background
  • Do yourself, there are more than 5 pages of results...

If you want more custom help, provide more custom intel. Smilie

Hope this helps

Last edited by sea; 02-18-2015 at 10:13 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Load multiple files into a table

Hi, I need to load data from two files to a single table. My requirement is that I get two files in which a few column data are manadatory. These files are identified based on the file name. For example, I have two files ABCFile and BCDFile. ABCFile has mandatory data in column 3 and 4... (0 Replies)
Discussion started by: reshma15193
0 Replies

2. UNIX for Dummies Questions & Answers

How To Load Files in (.log) into a table

Hi, I was just wondering how can i load a files from a log file in unix into a table. e.i. a.log contains the ff: text.dmp text2.dmp Table contains the ff: col 1 filename col 2 date created Thank You! (1 Reply)
Discussion started by: lienyca
1 Replies

3. Shell Programming and Scripting

Load data to flat file from table.

Hi all, I need to know how to copy data from a table say ABC to a flat file say XYZ.dat in unix, Please leave ur comments and the fastest way to do so, I need to load the table records into flat file. Regards Ann (4 Replies)
Discussion started by: Haque123
4 Replies

4. Shell Programming and Scripting

How do I load records from table to a flat file!

Hi All, I need to load records from oracle table XYZ to a flat file say ABC.dat. could any one tell me how do i do this in UNXI, Regards Ann (1 Reply)
Discussion started by: Haque123
1 Replies

5. Shell Programming and Scripting

Need help on Mulitple files mutliple actions

Hi all, I have mistkanely gzipped twice an entire folder and sub folders, and also renamed the files during that process. I am trying to undo this, and I need help to create the batch to work on it. All folders are under my images directory, I have a output.txt file that holds all the... (1 Reply)
Discussion started by: saariko
1 Replies

6. Shell Programming and Scripting

To load data from variable to oracle table ....???

Hi all, How to load variable value into Oracle table? I have created a file validation shell script. After the validation completes, i need to update a Oracle table with the variable value... Can someone help me how to do it? I have searched, but not able to get answer. i have 4 variables... (2 Replies)
Discussion started by: Amit.Sagpariya
2 Replies

7. Shell Programming and Scripting

To load data from variable to oracle table

Hi all, How to load variable value into Oracle table? I have created a file validation shell script. After the validation completes, i need to update a Oracle table with the variable value... Can someone help me how to do it? I have searched, but not able to get answer. i have 4... (1 Reply)
Discussion started by: Amit.Sagpariya
1 Replies

8. Shell Programming and Scripting

mutliple files in the same directory

I have over 900 files that have the same name except for a unique numeric assignment. For all files I would like to cut the 2nd column and paste all into one new file. All in bash. sample input format for each file: 1 2 3 1 2 3 1 2 3 sample command for what I want to do: cut -d' ' -f2... (8 Replies)
Discussion started by: epi8
8 Replies

9. UNIX for Dummies Questions & Answers

process mutliple files in the same directory

I have over 900 files that have the same name except for a unique numeric assignment. For all files I would like to cut the 2nd column and paste all into one new file. All in bash. sample input format for each file: 1 2 3 1 2 3 1 2 3 sample command for what I want to do: cut -d' ' -f2... (1 Reply)
Discussion started by: epi8
1 Replies

10. UNIX for Dummies Questions & Answers

Please Help:Need to Split the file into mutliple files depends on the KEY field value

Hi Gurus, I am new to UNIX(HP). my requirmnet is File needs to needs to be split into multiple files dependa on one key value. example 1 abc 333 us 2 bbc 444 ch 5 nnn 333 kk 7 hhh 555 ll 3 hhh 333 hh now the requirment is line with 333 should be copied into test1.txt and... (14 Replies)
Discussion started by: arund_01
14 Replies
Login or Register to Ask a Question