Please help !!!!Problem with data file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Please help !!!!Problem with data file
# 15  
Old 04-07-2007
Quote:
Originally Posted by dsravan
vgresh,

Each record has 11 pipes in it for sure. And there are 12 data values in each record.
to join 'broken' lines....
nawk -f ds.awk myBigFile.txt > myBigFile_new.txt

ds.awk
Code:
BEGIN {
  FS=OFS="|"

  FLD_max=12
}
# NF - number of fields in the current record
# if the number of fields in mthe current record is less than maximum number 
# of fields - execute the action: if "rec" is NOT empty - concatenate it with
# the current record [rec OFS $0$0]; if "rec" IS empty - assign the current
# record/line to it [$0]. Increment the "running" number of fields [fld] by the
# number of the fields in the current record [NF]. Proceed to the next record
# [next].
NF < FLD_max {rec=(rec != "") ? rec OFS $0 : $0; fld+=NF;next }

# if the running number of fields [fld] is greater tan the MAX: output the
# "rec" and initialize "rec" to be empty and the running number of fields to be "0"

fld >= FLD_max { print rec; rec=""; fld=0 }

# if NONE of the above actions could be executed - "print" the current 
# record/line: "1" is a shortname for "print $0"
1


Last edited by vgersh99; 04-07-2007 at 01:39 PM..
# 16  
Old 04-07-2007
Vgresh,

thank you very much.

can you please explain this.

Quote:
NF < FLD_max {rec=(rec != "") ? rec OFS $0 : $0; fld+=NF;next }
fld >= FLD_max { print rec; rec=""; fld=0 }
1
Thanks,
Arun
# 17  
Old 04-07-2007
vgresh,

I ran the script but still there are still records starting with "|" in the beginning which means it is spannig to 2 records.

Please help
# 18  
Old 04-07-2007
Quote:
Originally Posted by dsravan
Vgresh,

thank you very much.

can you please explain this.



Thanks,
Arun
explanation in the code.
# 19  
Old 04-07-2007
Quote:
Originally Posted by dsravan
vgresh,

I ran the script but still there are still records starting with "|" in the beginning which means it is spannig to 2 records.

Please help
post the offending records. I ran your first sample and it works fine.
# 20  
Old 04-07-2007
vgresh,

The code is not working as desired becuase i am still having bad records spaning to 2 lines. Pleae help.
# 21  
Old 04-07-2007
run the code below and post the snippet from the myBigFile.log log file

nawk -f ds.awk myBigFile.txt > myBigFile_new.txt 2> myBigFile.log

Code:
BEGIN {
  FS=OFS="|"

  FLD_max=12

  stderr="cat 2>&1"
}
NF < FLD_max {printf("Bad record: [%d] :: [%s]\n", FNR, $0) | stderr; rec=(rec != "") ? rec OFS $0 : $0; fld+=NF;next }
fld >= FLD_max { print rec; rec=""; fld=0 }
1

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replace data of a file with data from another file using shell scripting.

Dears, I'm new to shell scripting and i was wondering if you can help me with following matter. I have a file containing 400,000 records. The file contains two columns like: 00611291,0270404000005453 25262597,1580401000016155 25779812,1700403000001786 00388934,1200408000000880... (1 Reply)
Discussion started by: paniklas
1 Replies

2. Shell Programming and Scripting

Extract header data from one file and combine it with data from another file

Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people, Here I have attached... (17 Replies)
Discussion started by: nex_asp
17 Replies

3. UNIX for Dummies Questions & Answers

Mapping a data in a file and delete line in source file if data does not exist.

Hi Guys, Please help me with my problem here: I have a source file: 1212 23232 343434 ASAS1 4 3212 23232 343434 ASAS2 4 3234 23232 343434 QWQW1 4 1134 23232 343434 QWQW2 4 3212 23232 343434 QWQW3 4 and a mapping... (4 Replies)
Discussion started by: kokoro
4 Replies

4. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

5. Shell Programming and Scripting

Scan and change file data content problem

Input file >Read_1 XXXXXXXXXXSDFXXXXXDS (condition 1: After the last "X" per line, if the distance is less than or equal to 3 letter, replace those not "X" letter with "X") TREXXXXXXXSDFXXXXXDS (condition 2: Before the first "X" per line, if the distance is less than or equal to 3 letter,... (12 Replies)
Discussion started by: patrick87
12 Replies

6. Shell Programming and Scripting

Compare and print out data only appear in file 1 problem

Below is the data content of file_1 and file_2: file_1 >sample_1 FKGJGPOPOPOQA ASDADWEEWERE ASDAWEWQWRW ASDASDASDASDD file_2 >sample_1 DRTOWPFPOPOQA ASDADWEEASDF ASDADRTYWRW ASDASDASDASDD I got try the following perl script. Unfortunately, it can't give my desired output result... (7 Replies)
Discussion started by: patrick87
7 Replies

7. Shell Programming and Scripting

Find and replace data in text file with data in same file

OK I will do my best to explain what I need help with. I am trying to format an ldif file so I can import it into Oracle oid. I need the file to look like this example. Keep in mind there are 3000 of these in the file. changetype: modify replace: userpassword dn:... (0 Replies)
Discussion started by: timothyha22
0 Replies

8. Shell Programming and Scripting

C Shell problem: using a key from one file to find data in another

I've never written scripts (just switched from Ada to C++). I have a book that's over my head and a few examples, other then that I'm floundering. Everything here at work is being done in C Shell. None of the C++ programmers are experienced in shell scripting. I have a data file with the... (2 Replies)
Discussion started by: bassmaster
2 Replies

9. Shell Programming and Scripting

Problem in writing the data to a file in one row

Hi All I am reading data from the database and writing to temporary file in the below format. 1=XP|external_component|com.adp.meetingalertemail.processing.MeetingAlertEmail|EMAILALERTPUSH|32|4#XP |classpath|/usr/home/dfusr/lib/xalan.jar: /usr/home/dfusr/lib/xerces.jar: ... (2 Replies)
Discussion started by: rajeshorpu
2 Replies

10. Shell Programming and Scripting

Problem getting data to a report file.

Hi all, I'm trying in vain to workout how I can generate a report from a months worth of files that get created every day. There is one file per day and each daily file contain the output from a df -v command. With the following section of code ... for xdffile in $1$2/df?? do ... (4 Replies)
Discussion started by: Cameron
4 Replies
Login or Register to Ask a Question