Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Performance issue in UNIX while generating .dat file from large text file
# 8  
Old 04-21-2009

.txt and .DAT tell me nothing about the format of the files.

What does a line from the .txt file look like?

What has to be done to it to prepare it for the .DAT file?
# 9  
Old 04-21-2009
Hi John,
Please find the answer as below:


What does a line from the .txt file look like?

A record from the input text file looks like below:

0 1509999999900000002A200811AA 0 108012121315LREPO 150
*/

The name of the input file looks like this : glbd92_1000112008_0402110932

What has to be done to it to prepare it for the .DAT file?

I am just redirecting the output to .DAT extension. no Conversion is there related to data from .txt to .DAt . its simply creating new .DAT file or APPENDING the .DTA file based of the loop conditions.

If you need more details of script then please do let me know.
# 10  
Old 04-21-2009
Quote:
Originally Posted by KRAMA
Hi John,
Please find the answer as below:


What does a line from the .txt file look like?

A record from the input text file looks like below:

0 1509999999900000002A200811AA 0 108012121315LREPO 150
*/

In your script you use cut to get characters at columns 166 to 168 of some line. That line is not that long. What were you trying to do?
Quote:

The name of the input file looks like this : glbd92_1000112008_0402110932

What has to be done to it to prepare it for the .DAT file?

I am just redirecting the output to .DAT extension. no Conversion is there related to data from .txt to .DAt . its simply creating new .DAT file or APPENDING the .DTA file based of the loop conditions.

What are those "loop conditions"?

If all you are doing is copying lines from one file to another, why do you need a complicated script?
# 11  
Old 04-22-2009
Hi John,

This is an actual record from the input .TXT file.

002000012008AA01000 10000405 7010000
150609 Y G5 PRD 000000000000000.00
000000 000000000000000.00000000 000000000000000.00000000 000000000000000.00000000 000000000000000.00000000 000000000000000.00000000 000000000000000.00000000
000000000000000.00000000 000000000000000.00000000-000000000000100.00000000 000000000000000.00000000 000000002104734.00000000 000000000000000.00000000
LREPO ZGIFBM_GL_108012121315

The script is not written by me , its written by someone else i have to enhance the script to make the performance better. The looping conditions are important because there is one Branch/site '400' which they want to cut the characters from the input record and need to keep it as branch.

Can you please let me know wheteher it is possible to avoid the m_arr_ctr and to generate/append the output in a batch instead line by line. Or how to apply awk over ther.
# 12  
Old 04-30-2009
Can any one help me on this...
# 13  
Old 04-30-2009
Try using perl. It was designed for fast text processing.

tyler_durden
# 14  
Old 05-01-2009
It will be easier for us to suggest solutions if you could lay down the input file structer( which you already did)..tell us the logic of what you wanted to achieve and then an o/p sample for the said inputs..


cheers,
Devaraj Takhellambam
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Generating xml file from UNIX

i have a unix script which generates the csv file. the data in csv file is dynamic. how can i convert/move the data from csv file to xml. please suggest (1 Reply)
Discussion started by: archana25
1 Replies

2. Answers to Frequently Asked Questions

How to split a dat file based on another file ni UNIX?

i have two files , one is var.txt and another res.dat file var.txt contains informaton like below date,request,sales,item 20171015,1,123456,216 20171015,1,123456,217 20171015,2,345678,214 20171015,3,456789,218 and res.dat contains is a one huge file contains information like... (1 Reply)
Discussion started by: pogo
1 Replies

3. UNIX for Dummies Questions & Answers

Generating a CSV file from a text file

Hi Guys, I have a simple request. I have a file in w3c format. Each file has 2 header lines. Rest of the lines are 16 columns each. They are separated by Tab. I need to discard the first 2 lines and then write each column of the txt file into a seperate column of CSV. I tried the command below... (1 Reply)
Discussion started by: tinkugadu
1 Replies

4. Shell Programming and Scripting

Performance issue in Grepping large files

I have around 300 files(*.rdf,*.fmb,*.pll,*.ctl,*.sh,*.sql,*.prog) which are of large size. Around 8000 keywords(which will be in the file $keywordfile) needed to be searched inside those files. If a keyword is found in a file..I have to insert the filename,extension,catagoery,keyword,occurrence... (8 Replies)
Discussion started by: millan
8 Replies

5. Shell Programming and Scripting

Remove <CR><LF> from the dat file in unix

Hi, The source system has created the file in the dat format and put into the linux directory as mentioned below. I want to do foloowing things. a) Delete the Line started with <CR><LF> in the record b)Also line ...........................................................<CR><LF> ... (1 Reply)
Discussion started by: mr_harish80
1 Replies

6. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

7. UNIX for Dummies Questions & Answers

How do I delete a data string from a .dat file in unix

I have a .dat file in unix and it keeps failing file validation on line x. How do I delete a data string from a .dat file in UNIX? I tried the following: sed -e 'data string' -e file name and it telling me unrecognized command (4 Replies)
Discussion started by: supergirl3954
4 Replies

8. Shell Programming and Scripting

How to read from a .dat file in Unix

Hi All, I have a .dat file named test.dat where I have stored some process IDs. Now I need to pick a process ID, one by one and then fire kill -9 for each of those. The logic should be: 1. open file <filename.dat> 2. read until last line of file 3. if process ID is found fire kill -9... (5 Replies)
Discussion started by: Sibasish
5 Replies

9. Shell Programming and Scripting

How to attach an excel file/ dat file thru unix mails

Hi. I want to attach a .xls or .dat file while sending mail thru unix. I have come across diff attachments sending options, but allthose embeds the content in the mail. I want the attachement to be send as such. Please help me out. regards Diwakar (1 Reply)
Discussion started by: diwakar82
1 Replies

10. UNIX for Dummies Questions & Answers

Unix File System performance with large directories

Hi, how does the Unix File System perform with large directories (containing ~30.000 files)? What kind of structure is used for the organization of a directory's content, linear lists, (binary) trees? I hope the description 'Unix File System' is exact enough, I don't know more about the file... (3 Replies)
Discussion started by: dive
3 Replies
Login or Register to Ask a Question