Large file data handling issue | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

Large file data handling issue

UNIX for Dummies Questions & Answers


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 11-14-2012
Gurkamal83 Gurkamal83 is offline
Registered User
 
Join Date: Nov 2012
Last Activity: 11 August 2013, 9:56 AM EDT
Posts: 19
Thanks: 1
Thanked 0 Times in 0 Posts
Large file data handling issue

I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long"

I need to actually remove the last | symbol from this file.

Code:
sed -e 's/\|*$//' filename

is working fine for small files. But not working on this big file. The file size is "614077"

Last edited by Scrutinizer; 11-14-2012 at 03:40 PM.. Reason: code tags
Sponsored Links
    #2  
Old 11-14-2012
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 29 August 2014, 9:07 PM EDT
Location: Saskatchewan
Posts: 19,271
Thanks: 774
Thanked 3,237 Times in 3,035 Posts
614077 what? 600K isn't very large at all, sed should be able to handle it fine.

Even if it's a large file, sed will still work with it, it will just take longer.

What might be a problem are really really long lines...
Sponsored Links
    #3  
Old 11-14-2012
Gurkamal83 Gurkamal83 is offline
Registered User
 
Join Date: Nov 2012
Last Activity: 11 August 2013, 9:56 AM EDT
Posts: 19
Thanks: 1
Thanked 0 Times in 0 Posts
hmm, yes this is a 600K file and as already said it is a single record. How can i handle the file manupulation?
    #4  
Old 11-14-2012
spacebar's Avatar
spacebar spacebar is offline
Registered User
 
Join Date: Oct 2009
Last Activity: 2 June 2014, 3:23 PM EDT
Location: spaceBAR Central
Posts: 440
Thanks: 0
Thanked 70 Times in 70 Posts
Quote:
Originally Posted by Gurkamal83 View Post
I need to actually remove the last | symbol from this file.

Code:
sed -e 's/\|*$//' filename

is working fine for small files. But not working on this big file. The file size is "614077"
What exactly do you mean by "sed is not working on this big file", What messages do you get when you press enter?
Paste the first and last 100 bytes of the file.
Also try it this way:


Code:
sed 's/|$//' filename

Sponsored Links
    #5  
Old 11-14-2012
Gurkamal83 Gurkamal83 is offline
Registered User
 
Join Date: Nov 2012
Last Activity: 11 August 2013, 9:56 AM EDT
Posts: 19
Thanks: 1
Thanked 0 Times in 0 Posts
Thanks for the reply.. but your command also didnt work.
With the small files i can see record values and the record getting trimmed off with the trailling '|'.
When I use the same command with my actual file there is nothing that is coming on screen even if I echo it into a newfile nothing with get populated in the new file.

Starting bytes
001;04.5.1;2012-10-25 08:47:41;ABCDE||3;1351169231;1351169261;;;1351169256;1351169256;1351169261;;;;;;;;;;;00:00:05;

Ending bytes.
255;8;1;incoming;;;;;2;0;;;0;;10;0;0;0;0;;abc.com;x1-6-00-1c-fb-2f-8e-14.XXXX.xc.xr.com;;;;;0;000000;;Singh Gurkamal;1;;;this is a very big file;;;||1000;2012-10-25 08:49:03|
Sponsored Links
    #6  
Old 11-14-2012
methyl methyl is offline Forum Advisor  
Advisor
 
Join Date: Mar 2008
Last Activity: 18 April 2014, 5:13 AM EDT
Posts: 6,396
Thanks: 287
Thanked 672 Times in 642 Posts
Is this "file" actually a "stream" with no record terminators?
Sponsored Links
    #7  
Old 11-14-2012
Gurkamal83 Gurkamal83 is offline
Registered User
 
Join Date: Nov 2012
Last Activity: 11 August 2013, 9:56 AM EDT
Posts: 19
Thanks: 1
Thanked 0 Times in 0 Posts
yes.. I am getting from where u r coming.. do u mean the line feed??
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
UNIX file handling issue Gurkamal83 Shell Programming and Scripting 3 11-14-2012 03:28 AM
Advice regarding filesystems handling large number of files shoaibjameel123 Red Hat 2 09-20-2011 06:02 AM
Severe performance issue while 'grep'ing on large volume of data Souvik Shell Programming and Scripting 6 04-04-2011 04:47 AM
UNIX File handling -Issue in reading a file KuldeepSinghTCS Shell Programming and Scripting 1 08-16-2010 11:47 AM
Performance issue in UNIX while generating .dat file from large text file KRAMA Shell Programming and Scripting 19 05-20-2009 10:55 AM



All times are GMT -4. The time now is 06:15 PM.