awk use to generate report


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers awk use to generate report
# 1  
Old 10-15-2019
awk use to generate report

Hi ,
In a directory list of ddl files are stored in the given format above. Above is the sample ddl file. The ddl file name is same as that of table name ie email_notifications.ddl
I want to generate below report using awk utility reading all the ddl files stored in /ddl path
Code:
Desired output:
table name, last_modified_time, totalsize
email_notifications,1559883505,8765717674
email_bounced, 1559883505,999999

Also, would like to know if the modified time in unix can be converted to actual date time instead of unix time

Note: It may happen that in other DDL files TBLPROPERTIES information may or may not available.


here is one of the file sample format:
Code:
CREATE TABLE `test.email_notifications`(
  `email_address_id` decimal(16,0), 
  `individual_id` decimal(16,0), 
  `email_address` string, 
  `last_bounce_date` string 
)
ROW FORMAT DELIMITED 
  FIELDS TERMINATED BY '|' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://apps/hive/db/test.db/email_notifications'
TBLPROPERTIES (
  'last_modified_by'='ops', 
  'last_modified_time'='1559883505', 
  'numFiles'='1', 
  'numRows'='0', 
  'rawDataSize'='0', 
  'skip.header.line.count'='1', 
  'totalSize'='8765717674', 
  'transient_lastDdlTime'='1559883505')

# 2  
Old 10-15-2019
You can process text with any "modern" programming language you choose.

You should pick a programming language, assuming you work in an organization, based on the skill sets and tech support policies of the organization. If your shop is mostly python, then you should write text processing scripts in python; if you are in web shop using PHP, you should process test in PHP. The more different languages your organization must support, the more difficult (and expensive) it is. If you work in a shop with hard-core unix people who
have all their code base in shell scripts, then use these same scripts, based on the organizational culture, business model and skill sets.

So, the question you should ask yourself is "what is the programming language to process text in my organization based on the policy of the organization and the current and future skill sets of the staff?"

Most every programming language can process text. You should work within your organizations skill sets and match the programming needs with the skills of the staff, especially the younger staff, who will be around much younger than the retiring staff.

Having said that,

If you want to use awk and your organization supports this, you should be programming in awk and posting your code here for review and help.

Often the question "can I do this?" with this programming language is a veiled question to ask others to code for them.

Please show your own coding work.

Thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To get older than last 7days records using awk scripting to generate report and send email

Hello All, I have need as below: 1--> I need to get all users(who submit jobs) and their details by using below command: qstat -u \* output of the above command looks line below: job-ID prior name user-id state "submit/start at" queue jclass slots ja-task-ID... (5 Replies)
Discussion started by: VasuKukkapalli
5 Replies

2. Shell Programming and Scripting

Generate .csv/ xls file report

There can be thousand of .ksh in a specific directory where sql files are called from ksh. Requirement is to loop through all the files content and generate a report like below: Jobname Type type sqlname gemd1970 sql daily tran01 gemw1971 sql weekly ... (6 Replies)
Discussion started by: vedanta
6 Replies

3. Shell Programming and Scripting

Generate a Summary report

Hi All, Script to meet my requirement might be simpler for UINIX experts.. :) I need to generate an summary report in .txt file using shell script I have Reject directory in Unix server which contains all reject files for three diff categories- Presentation, Chapter and Scene Following... (3 Replies)
Discussion started by: Sakthikalluri
3 Replies

4. Shell Programming and Scripting

Need a script that generate a report from a combined sudoers

Hello Gurus Is anyone aware of a script/utility that can be used to generate a report from a combined "sudoers" file for more than 100 servers in one go please? Basic query: Which particular user or an user group can execute root equivalent commands or has the ability for e.g. to do a... (1 Reply)
Discussion started by: kapil514
1 Replies

5. Shell Programming and Scripting

generate report based on data in files.

Hi All, I need to develop a shell script which does sanity check of a data file, as below. 1. For DATE columns, it should check if date is given in proper format or not? For example, if format of date is expected as DD-MON-YYYY HH24:MI:SS and we received the date in formation like DDMMYYYY HH24,... (1 Reply)
Discussion started by: ace_friends22
1 Replies

6. Shell Programming and Scripting

generate a report

I am trying to generate a report for a file called phone_book awk -f {phone_book} why does this not work? Nothing happens at all. (2 Replies)
Discussion started by: gustave
2 Replies

7. AIX

Re-Generate error report

Hi Is there a way to re-generate AIX error report after clearing it? I mean if I did errclear 0, Does the error log get erased? Is there a command that will get entries back in errpt? I hope that made sense. (5 Replies)
Discussion started by: Dardeer
5 Replies

8. Shell Programming and Scripting

generate a report

Hi Please help me to resolve the below query. My shell script has generated a file output.file like below ******************************** DROP TABLE GPS_CONTACT_DETAILS DB20000I The SQL command completed successfully. CREATE TABLE GPS_CONTACT_DETAILS ( CONTACT_ID ... (8 Replies)
Discussion started by: sailaja_80
8 Replies

9. Shell Programming and Scripting

Generate report in HTML file from Oracle DB

Hi Team, I need a suggestion/comments for my below requirement. I have a procedure which performs some DDL operations & loads data into a Oracle table. This status contains Audit data. What i wanted to do is, once the procedure is completed (daily), shell script should retrive the data from the... (4 Replies)
Discussion started by: Amit.Sagpariya
4 Replies

10. Shell Programming and Scripting

Help generate report from log files

Hi Expert, I have some confusing to generate report from the log file as shown below: filename :test1.log start_time date end_time number code P 000029.621 20070823 000029 12134567890 111111111111 00 0 000 003... (4 Replies)
Discussion started by: bucci
4 Replies
Login or Register to Ask a Question