Python Script to take file count and insert into DB2 table


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Python Script to take file count and insert into DB2 table
# 1  
Old 12-26-2017
Python Script to take file count and insert into DB2 table

Team

I have files in different directories . How can i take the count of latest file and insert into Db2 table .
I am using
Code:
awk 'END{print NR+1-ARGC}' (File name)

to get the counts.
How can i take
1.The count of latest file
2.Insert into Db2 table( File Name and Counts) .

cd /var/tmp/ARCH_20171201
ARCH_20171201.csv
ARCH_20171202.csv

cd /var/tmp/random
UNIQUE_20171201.csv
UNIQUE_20171202.csv

How can i pick latest file counts ( ARCH_20171202.csv & UNIQUE_20171202.csv) and insert in DB2 temporary table( File Name and Counts)?
Please suggest if Python or perl scripts can be used.
Any help appreciated
Thanks
# 2  
Old 12-26-2017
Please be way more precise!
What is a "latest file"? By file system time stamp? Time string in file name? Running sequence number?
What is "the count of" latest file?
What do you need to "insert into DB2 table"? The algorithm? The data? The SQL- code?
And, which would be your preferred tool? awk (as shown)? perl? python? any other?
# 3  
Old 12-26-2017
Building on what RudiC has already asked... Are we correct in assuming:
  1. you want the number of lines in each selected file - 1,
  2. all of the files you want to process can be selected by the filename matching pattern /var/tmp/*/*.csv,
  3. all filenames end with an eight digit date string in the format YYYYMMDD followed by the string .csv, and
  4. within each directory under /var/tmp, the string before the date string is a constant?
Do you want the line count from each selected file in a directory or do you want the sum of the line counts from the selected files in all of the directories containing *.csv files?

What operating system and shell are you using?

Since you're already using awk, is it OK to just use awk without adding php or python code?

Does the awk you are using on your system have the nextfile command?
# 4  
Old 12-27-2017
Thanks RudiC and Don for reply .

Just trying to make it more precise for better understanding ( Dropping the insert idea into DB2 tables )
1.you want the number of lines in each selected file - 1,
I would like to take just the count of latest file based on timestamp and previous timestamp file count .
a) Compare both counts ( if latest file >= old File --> just assign FLAG=SUCCESS
b) Compare both counts (if latest file < 10% of old File --> just assign FLAG=SUCCESS
c) Compare both counts (if latest file < 0 --> just assign FLAG=FAIL
2.all of the files you want to process can be selected by the filename matching pattern /var/tmp/*/*.csv,
yes , file pattern will be filename_timestamp ( exam ARCH_20171202.csv)
Lets use only one path /var/tmp/
3.all filenames end with an eight digit date string in the format YYYYMMDD followed by the string .csv, and
yes
4.within each directory under /var/tmp, the string before the date string is a constant?
yes

5.Do you want the line count from each selected file in a directory or do you want the sum of the line counts from the selected files in all of the directories containing *.csv files?
Lets have only file counts . use only one path /var/tmp/
I would like to use python instead of unix/perl . Using windows 7.
let me know for any questions
# 5  
Old 12-27-2017
Quote:
Originally Posted by Perlbaby
Thanks RudiC and Don for reply .

Just trying to make it more precise for better understanding ( Dropping the insert idea into DB2 tables )
1.you want the number of lines in each selected file - 1,
I would like to take just the count of latest file based on timestamp and previous timestamp file count .
a) Compare both counts ( if latest file >= old File --> just assign FLAG=SUCCESS
b) Compare both counts (if latest file < 10% of old File --> just assign FLAG=SUCCESS
c) Compare both counts (if latest file < 0 --> just assign FLAG=FAIL
2.all of the files you want to process can be selected by the filename matching pattern /var/tmp/*/*.csv,
yes , file pattern will be filename_timestamp ( exam ARCH_20171202.csv)
Lets use only one path /var/tmp/
3.all filenames end with an eight digit date string in the format YYYYMMDD followed by the string .csv, and
yes
4.within each directory under /var/tmp, the string before the date string is a constant?
yes

5.Do you want the line count from each selected file in a directory or do you want the sum of the line counts from the selected files in all of the directories containing *.csv files?
Lets have only file counts . use only one path /var/tmp/
I would like to use python instead of unix/perl . Using windows 7.
let me know for any questions
You have now completely changed your original requirements and introduced some new questions.

Before you wanted a single file from each subdirectory under /var/tmp, now we have to collect data from two files in each of those subdirectories. We now also have to compare results from those two files and create a flag. Furthermore, the flag is set to SUCCESS if there are more lines in the newest file than in the previous file or if the newest file contains less than 10% of the number of lines in the previous file (not counting the header line in either file), and the flag is set to FAIL if the newest file is an empty file. There is no indication of how the flag should be set if the number of lines in the newest file has a number of lines that is greater than or equal to 10% of the number of lines in the previous file but less or equal to the number of lines in the previous file. And, there is no indication of whether or not there is another special case to be handled if the previous file is empty.

And, there is no indication of what is supposed to be done with that flag once it has been set. You need to show us the output you hope to produce with your new specifications (in CODE tags).

In your first post in this thread you showed us an awk script that had been given to you before that calculated the total number of lines (not counting header lines and assuming that every file contained at least a header line) in a group of one of more files. The awk code you showed us won't work with empty files, so if you were willing to accept an awk solution to your problem, the code you provided can't be used as a template. But, you now say that a shell script or an awk script is not allowed and that anyone wanting to help you must instead write it in python. I'm not proficient in python so I won't be able to help you, but I think you will still need to answer the questions I've posed above for anyone else to be able to help you.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

How to track table status delete/update/insert status in DB2 V10 z/os?

Dear Team I am using DB2 v10 z/os database . Need expert guidance to figure out best way to track table activities ( Ex Delete, Insert,Update ) Scenario We have a table which is critical and many developer/testing team access on daily basis . We had instance where some deleted... (1 Reply)
Discussion started by: Perlbaby
1 Replies

2. Shell Programming and Scripting

Intelligent Script to Insert Records in Database Table

Hello All Members, I am new to this forum and to the shell scripting. I want to write a shell script to do the following: Scenario: I have a pipe delimited .txt file with multiple fields in it. The very first row of the file contains the name of the column which resembles the name of the... (18 Replies)
Discussion started by: chetanojha
18 Replies

3. Shell Programming and Scripting

Insert script result into Oracle Table

Hi All, I want to insert STAT and ENDTIME values for each job in joblist into TBL_DAILY_STATUS table. Eg: insert into tbl_daily_status values(STAT,ENDTIME); Please help me on this. #!/bin/ksh joblist="com_abc_job com_abc_dot_job com_abc_seq com_abc_det" for i in $joblist do... (8 Replies)
Discussion started by: vichuelaa
8 Replies

4. Shell Programming and Scripting

Read parameter file in a shell script to unload a DB2 Table???

Hi , I Have following requirement: DB2 Sql query to pass from a parameter file for example, I would create a parameter file with (SELECT column 1, column 2 FROM Table name) then job would read it and create a file with the contents named table.txt How to write/modify below ksh script to... (10 Replies)
Discussion started by: developer.dwh9
10 Replies

5. UNIX for Dummies Questions & Answers

Bash script to insert data into an html table

hi, I need to create a bash shell script which picks up data from a text file and in the output file puts it into an html made table. I have to use sed and awk utilties to do this the input text file will contain data in the format: job name para1 para2 para3 para4 para4 1 ... (1 Reply)
Discussion started by: intern123
1 Replies

6. Shell Programming and Scripting

select values from db1 table and insert into table of DB2

Hi I am having three oracle databases running in three different machine. their ip address is different. from one of the DB am able to access both the databases.(means am able to select values and insert values in to tables individually.) I need to fetch some data from DB1 table(say DB1 ip is... (2 Replies)
Discussion started by: aemunathan
2 Replies

7. Shell Programming and Scripting

Export data from DB2 table to .txt file(space delimited)

Hi I need help on this. Its very urgent for me.. please try to help me out.. I have data in tables in DB2 database. I would like to export the data from DB2 tables into a text file, which has to be space delimited. so that I can carry out awk, grep operations on that file. I tried to export... (2 Replies)
Discussion started by: ss3944
2 Replies

8. UNIX for Dummies Questions & Answers

Advice on extracting special characters from a DB2 table to a file in the UNIX ENV

need some advice on the following situation. I have a DB2 table which has a varchar Column. This varchar column can have special characters like ©, ®, ™ . When I extract from this table to a sequential file for this varchar column I am only able to get © and ® . To Get the ™... (1 Reply)
Discussion started by: cosec
1 Replies

9. Shell Programming and Scripting

How to connect DB2 table using shell script

Hi All, I want to connect two tables in DB2 using shell script and then compare the contents of two tables field by field. Could any one please help me in connecting database tables using Unix and retriving data from the same. Thanks, Durwas (0 Replies)
Discussion started by: dtidke
0 Replies

10. UNIX for Advanced & Expert Users

unix script for update or insert records from a file to a oracle table

Hi, I have delimited file(|). Sample data: 1|name|50009|DS24|0|12 2|name|30009|DS24|0|13 3|name|20409|DS24|0|14 4|name|20009|DS24|0|15 5|name|10009|DS24|0|16 I want to load this data into a oracle table (update and insert) Please help me the commands and also... (1 Reply)
Discussion started by: unihp1
1 Replies
Login or Register to Ask a Question