Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: FASTEN count line of dat file and compare with the CTRL file
Top Forums Shell Programming and Scripting FASTEN count line of dat file and compare with the CTRL file Post 302873709 by ckwan on Tuesday 12th of November 2013 10:21:23 PM
Old 11-12-2013
FASTEN count line of dat file and compare with the CTRL file
Hi All,
I thinking on how to accelerate the speed on calculate the dat file against the number of records CTRL file.

There are about 300 to 400 folder directories that contains both DAT and CTL files.
DAT contain all the flat files records
CTL is the reference check file for the DAT for verification purpose.
It contain the DAT total number of records information, process date and so on.

for your information, some of the DAT files are pretty large may contain up to > 10Mil records (due to full file and part of the application requirement).

Sometimes it took more than 30 min to perform wc -l for all the DAT file (The script that I created which run serially).

One of the way to accelerate the process of perform aggregation is to spawn multiple processes to perform the tasks since the CPU is idle just for this verification purposes.

Anyone can share the idea ?


Thanks.

---------- Post updated at 10:21 PM ---------- Previous update was at 10:10 PM ----------

Code:
  cat LNS_DSLNT01/DSLNT01.CTL
  DSLNT01.DAT§2013-10-21§13636§......

  ::This 3rd Filed in the CTRL indicate the number of records of DAT file will be.

  datdss@root:/dat3/data/UAT_TEST/wc -l ./LNS_DSLNT01/DSLNT01.DAT
  13636

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Adding a character in the beginning of every line in a .dat file

How can i add a character(#) in the beginning of every line in a .dat file (2 Replies)
Discussion started by: Cool Coder
2 Replies

2. Shell Programming and Scripting

compare two .dat files and if there is any difference pulled into a separate file

Hi, compare two .dat files and difference will be moved into separate file.if anybody having code for this please send asap. using diff command, i don't know how to write shell programming. and my first file is like this including Header and trailer 10Ç20060323Ç01(Header) 01ÇIÇbabuÇ3000 01ÇIÇbaluÇ4000... (1 Reply)
Discussion started by: kirankumar
1 Replies

3. Shell Programming and Scripting

Read dat file line by line

Hello, I am a bit stuck on something I am sure is easy to most. I have a dat file that has a list of server names. sentra blue red willy clawcrab I need to take each server name from the dat file in a shell script and attempt to ssh to it to run a script on that server. So I guess I... (1 Reply)
Discussion started by: LRoberts
1 Replies

4. Shell Programming and Scripting

Shell script to count number of ~ from each line and compare with next line

Hi, I have created one shell script in which it will count number of "~" tilda charactors from each line of the file.But the problem is that i need to count each line count individually, that means. if line one contains 14 "~"s and line two contains 15 "~"s then it should give an error msg.each... (3 Replies)
Discussion started by: Ganesh Khandare
3 Replies

5. UNIX for Dummies Questions & Answers

Count rows in .dat file

Could you please tell e how to find the total number of rows in a .dat file. edit by bakunin: this is exactly why in "advanced and expert" section?? I transfer this thread to "Unix for Dummies Questions and Answers" (2 Replies)
Discussion started by: Deeptanshu
2 Replies

6. Shell Programming and Scripting

Compare two string in two separate file and delete some line of file

Hi all i want to write program with shell script that able compare two file content and if one of lines of file have # at the first of string or nothing find same string in one of two file . remove the line in second file that have not the string in first file. for example: file... (2 Replies)
Discussion started by: saleh67
2 Replies

7. Shell Programming and Scripting

Compare file1 header count with file2 line count

What I'm trying to accomplish. I receive a Header and Detail file for daily processing. The detail file comes first which holds data, the header is a receipt of the detail file and has the detail files record count. Before processing the detail file I would like to put a wrapper around another... (4 Replies)
Discussion started by: pone2332
4 Replies

8. Shell Programming and Scripting

How to read file line by line and compare subset of 1st line with 2nd?

Hi all, I have a log file say Test.log that gets updated continuously and it has data in pipe separated format. A sample log file would look like: <date1>|<data1>|<url1>|<result1> <date2>|<data2>|<url2>|<result2> <date3>|<data3>|<url3>|<result3> <date4>|<data4>|<url4>|<result4> What I... (3 Replies)
Discussion started by: pat_pramod
3 Replies

9. Shell Programming and Scripting

Help with Getting distinct record count from a .dat file using UNIX command

Hi, I have a .dat file with contents like the below: Input file ============SEQ NO-1: COLUMN1========== 9835619 7152815 ============SEQ NO-2: COLUMN2 ========== 7615348 7015548 9373086 ============SEQ NO-3: COLUMN3=========== 9373086 Expected Output: (I just... (1 Reply)
Discussion started by: MS06
1 Replies

10. UNIX for Beginners Questions & Answers

Compare one file and get the count of multiple file

Hi All, I need to compare a record in one file and find the matching across two file. This is the master file. File name : CUST.dat CUST_ID 9998 10000 10004 10005 DATAFILE1 9998;80000091;4;687582837443;;;;;;;;; 9998;80000091;4;687582841003;;;;;;;;;... (4 Replies)
Discussion started by: arunkumar_mca
4 Replies

LEARN ABOUT SUNOS

dat_get_handle_type

dat_get_handle_type(3DAT)			     Direct Access Transport Library Functions				 dat_get_handle_type(3DAT)

NAME

       dat_get_handle_type - get handle type

SYNOPSIS

       cc [ flag... ] file... -ldat [ library... ]
       #include <dat/udat.h>

       DAT_RETURN
	   dat_get_handle_typet (
	   IN	 DAT_HANDLE	   dat_handle,
	   OUT	 DAT_HANDLE_TYPE   *handle_type
	   )

PARAMETERS

       dat_handle      Handle for a DAT Object.

       handle_type     Type of the handle of dat_handle.

DESCRIPTION

       The dat_get_handle_type() function allows the Consumer to discover the type of a DAT Object using its handle.

       The  dat_handle	can  be  one  of  the following handle types: DAT_IA_HANDLE, DAT_EP_HANDLE, DAT_EVD_HANDLE, DAT_CR_HANDLE, DAT_RSP_HANDLE,
       DAT_PSP_HANDLE, DAT_PZ_HANDLE, DAT_LMR_HANDLE, or DAT_RMR_HANDLE.

       The handle_type is one of the following handle  types:  DAT_HANDLE_TYPE_IA,  DAT_HANDLE_TYPE_EP,  DAT_HANDLE_TYPE_EVD,  DAT_HANDLE_TYPE_CR,
       DAT_HANDLE_TYPE_PSP, DAT_HANDLE_TYPE_RSP, DAT_HANDLE_TYPE_PZ, DAT_HANDLE_TYPE_LMR, DAT_HANDLE_TYPE_RMR, or DAT_HANDLE_TYPE_CNO.

RETURN VALUES

       DAT_SUCCESS	       The operation was successful.

       DAT_INVALID_HANDLE      The dat_handle parameter is invalid.

USAGE

       Consumers  can  use  this  operation to determine the type of Object being returned. This is needed for calling an appropriate query or any
       other operation on the Object handle.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Standard: uDAPL, 1.1, 1.2    |
       +-----------------------------+-----------------------------+
       |MT-Level		     |Unsafe			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       libdat(3LIB), attributes(5)

SunOS 5.10							    16 Jul 2004 					 dat_get_handle_type(3DAT)
All times are GMT -4. The time now is 07:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy