Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Implement in one line sed or awk having no delimiter and file size is huge
# 1  
Old 01-19-2011
Implement in one line sed or awk having no delimiter and file size is huge
I have file which contains around 5000 lines.
The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters.

I want to delete the lines
a> if it starts with 1 and if 576th postion is a digit i,e 0-9
or
b> if it starts with 0 or 9(i,e header and footer)

I want to redirect all the lines to another file which does not meet above criteria.

Following is the file named A.DAT where i have placed only one line otherwise it would be huge
Code:
018012011---->header
120110116 000001000000000001200000.00USD0000000001.2000135058492000094000000.00PL 000.150000B 000000000000.00 0105.00000.000
0002011011820110121MODE0 D4028 27385 JP31368000044028 6466866 FJT140280 COMNIVBIHARA SANGYO Y50
BPY ABC000000000000.00Eabcdend VICVBERTTN0093.000
053.000000.00 20110118 05:57:31 000.00000010185450 TFLOW NBD NNF CHASNEAB MODFAN CHABD CDBV as agdnt (NE180 ESCROW)
UTCHBS
ABC.BADKS NMRSJPJT AD-SD5213
NIMIPA
ABC.BCDSS MHCBJPJ2 0000380
MIZUHO 000.150000000001200000.0000000100
0000 VICKRRTT NNNN NNN000000000000.00N0000N 00 000000000000
.00 0000000000.
0000000000000001000000 000.000000000.000000000.000000000000000000.00000000000000.00000000000000.00 000000000000.00 000000000
0.000000000000000000.00000000000000.000000000000.000000000000000000.00000000000000.00000000000000.00000000000000.00000000000000.0020110121000000000000.00Y0000
01136432.33 NY10185450 N0900080.00
0000000001.20000000000000000.0000000T000000000000.000000010000
00.00000001000000.00000001000000.00ABC 0100.00 0000000000
94567--->footer

I am using the following but it is giving error "sed: command garbled"
Code:
sed -e 's#^1\(.\{575\}\)[0-9]\(.*\)$##' -e's/^[9]//' A.DAT >> B.DAT

But if am running this command for a less no position like 57 instead of 575 ,it is working fine.

Can you pls someone tell me what is the issue here.And what would be the appropriate command using sed or awk.
Last edited by Franklin52; 01-20-2011 at 04:16 AM.. Reason: Please use code tags
# 2  
Old 01-19-2011
Script looks fine to me - Perhaps it's a limitation of your sed?

How about using awk:

Code:
awk '!/^[90]/ && substr($0,576,1) !~ "[0-9]"' A.DAT >> B.DAT

Last edited by Chubler_XL; 01-19-2011 at 11:48 PM.. Reason: Fix typos
# 3  
Old 01-19-2011
In awk...
Code:
awk '!/^[09]/&&!(/^1/&&substr($0,576,1)~/[0-9]/)' A.DAT > B.DAT

---------- Post updated at 11:54 AM ---------- Previous update was at 11:49 AM ----------

In perl...
Code:
perl -n -e 'print $_ if !/^[09]/ && !(/^1/ && substr($_, 576, 1) =~ /[0-9]/)' A.DAT > B.DAT

These 2 Users Gave Thanks to Ygor For This Post:
Chubler_XL millan
# 4  
Old 01-20-2011
Opps, I forgot the Starts with '1' criteria, Ygor's awk solution trumps mine.
# 5  
Old 01-20-2011
Thanks a lot guys...

Code:
awk '!/^[90]/ && substr($0,576,1) !~ "[0-9]"' A.DAT >> B.DAT

it worked.

but whn i tried with
Code:
awk '!/^[09]/&&!(/^1/&&substr($0,576,1)~/[0-9]/)' A.DAT > B.DAT

it was giving error like the too long line..

Anyway,i got the solution..Thanks...........

Cheers!!!

---------- Post updated at 08:38 PM ---------- Previous update was at 08:16 PM ----------

oh!!! yes...how do i implement the criteria of starts with 1?
Ygor's command is giving error "too long line".

Ygor/Chubler,

Any other way?????

---------- Post updated at 08:48 PM ---------- Previous update was at 08:38 PM ----------

Hey Guys,

This worked for me.

Code:
nawk '!/^[90]/ && /^1/ && substr($0,576,1) !~ "[0-9]"' A.DAT > B.DAT

I think it also do the required operation on my file.

whats say?
Last edited by Franklin52; 01-20-2011 at 04:16 AM.. Reason: Please use code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Join the line on delimiter using sed/awk in UNIX

I've input as , abcd| ef 123456| 78| 90 Desired output as, abcdef 1234567890 Anyone please give the solution. (5 Replies)
Discussion started by: jinixvimal
5 Replies

2. Shell Programming and Scripting

Need next line as a space delimiter in awk

Hi,Below is the output for p3fi_dev services 1/app/oracle> . ./oraprofile_p3fi_dev p3fi_dev_01 (P):/devoragridcn_01/app/oracle> srvctl config service -d p3fi_dev p3fi_p3fi_dev.world PREF: p3fi_dev_01 AVAIL: p3fi_dev_02 pplnet_p3fidev PREF: p3fi_dev_01 AVAIL: p3fi_dev_02 nexus_p3fidev PREF:... (3 Replies)
Discussion started by: Vishal_dba
3 Replies

3. HP-UX

Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: while read line do emp_name=`echo $line` grep -e... (7 Replies)
Discussion started by: arb_1984
7 Replies

4. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

5. Shell Programming and Scripting

FTP a huge Size file

Dear All, Good Evening!! I have a requirement to ftp a 220GB backup file to a remote backup server. I wrote a script for this purpose. But it takes more than 8 hours to transfer this file. Is there any other method to do it in less time??? Thanks in Advance!!! ---------- Post updated... (5 Replies)
Discussion started by: Naga06
5 Replies

6. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

7. Shell Programming and Scripting

how to implement in one-line awk in a fixed file having no delimiter

Hi, I have a file a.txt having no delimiter. I want to exclude the line which contains 435th character as 1 or 2 and redirect the rest of the lines to another file b. Can you pls suggest how to do this in one liner awk. Following is just one line of the input file a:- 120110116 ... (10 Replies)
Discussion started by: millan
10 Replies

8. Shell Programming and Scripting

awk,nawk,sed, delimiter |~|

RECORD=NEW|~|VENDORN=LUCENT|~|VENDORM=CBX500_REAR|~|NETWORK=ATM|~|SUBNETWORK=N/A|~|SITE=CIL|~|REGION=KN|~|COUNTRY=PS|~|SWITCH=SWITCH1|~|E THERNET=N/A|~|LOOPBACK=N/A|~|SHELF=N/A|~|SLOT=14|~|SUBSLOT=N/A|~|STSCHAN=N/A|~|PORT=S14|~|DS1SLOT=N/A|~|LINE=N/A|~|LPORTID=N/A|~|CARDDESC=N/A|~|CARDTYPE=BAC2RT0... (7 Replies)
Discussion started by: knijjar
7 Replies

9. Shell Programming and Scripting

Split a huge line into multiple 120 characters lines with sed?

Hello , I'm trying to split a file which contains a single very long line. My aim is to split this single line each 120 characters. I tried with the sed command : `cat ${MYPATH}/${FILE}|sed -e :a -e 's/^.\{1,120\}$/&\n/;ta' >{MYPATH}/${DEST}` but when I wc -l the destination file it is... (2 Replies)
Discussion started by: jerome_1664
2 Replies

10. UNIX for Advanced & Expert Users

Insert a line as the first line into a very huge file

Hello, I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it? Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies
Login or Register to Ask a Question