Visit Our UNIX and Linux User Community


Large volume file formatting


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Large volume file formatting
# 1  
Old 08-01-2009
Large volume file formatting

Hi,
I have a file which is around 193 gb in size. This file has tonnes of spaces and I need to sanitize it.
I tried to use awk script to split this file but it gave me an error like line to long...
As of now I am using a sed command to search replace the spaces; however its too slow for such a big file.

Is there a better way to remove spaces from a file?

Another constraint that I have is I should not be touching the first two columns in the file even if they have spaces or not spaces.

Sed Command that I have used to get the job fast and also to ensure my first two colums are not touched :

sed 's/ *//g' bigfile > outfile

First Line from big file:----
; 00000000000000000000000000000000000000000000000000 ;/; ;111111111111338; ; ; ;/;/;/;/;/;/;/;/
;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/;/; ; ; ; ; ;
;/;00; ;I; ; ;2
0012;91; ; ;ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ; ;ONE ;
; ; ; ; ; ; ;2430 ;
; ; ; ; ; ; ;
; ; ; ; ; ; ;28 XXXXXXXXXXXXXXXXXXXXXXXXXXXXX ST ; ;
; ; ; ; ; ;
; ; ; ; ; ;US;ENGLISH ;N
SW ; ; ; ; ; ;
; ; ; ; ;
; ;I; ; ;99991231;BAD ;11111111;X;G ; ;11111111; ;
;/;/; ;
;/;/;/;/; ;/; ;/;
;/;/;/;/;/; ;/; ; ; ; ; ; ;/;
;/;/; ; ; ; ; ;/;/; ; ; ; ; ; ;
; ;/;/; ; ; ; ;/;/;/;/;/;/; ; ;786643219987879875756; ; ; ;0009000789y969y7;200
10821; ; ;87147146 ;35042346 ; ; ; ; ; ; ; ;
; ; ; ; ; ; ; ; ;/;I; ;US;02 65578109 ; ; ;
; ;20090731 ;99991231 ; ; ; ; ; ; ;
; ; ; ; ; ; ;
; ;
; ; ; ; ; ; ; ; ; ;
; ; ; ; ; ; ; ;XXDEFAULT ;20010821;19990716; ;
; ;/; ; ; ; ; ; ; ; ; ;/;/; ; ;/; ;/;/;/;/;/
;/;/;/;/;/;/;/;/; ; ; ;/; ; ; ; ;
# 2  
Old 08-02-2009
The fastest way would be to write a small C utility.
# 3  
Old 08-02-2009
You've got the fastest way already, my friend. You're not going to be able to write anything yourself that's faster than sed.

But why wait till the file is already generated to do it? Why not pass it through sed as it's being created?

Previous Thread | Next Thread
Test Your Knowledge in Computers #141
Difficulty: Easy
The original Unix operating system was not written in assembly language.
True or False?

10 More Discussions You Might Find Interesting

1. Red Hat

No space in volume group. How to create a file system using existing logical volume

Hello Guys, I want to create a file system dedicated for an application installation. But there is no space in volume group to create a new logical volume. There is enough space in other logical volume which is being mounted on /var. I know we can use that logical volume and create a virtual... (2 Replies)
Discussion started by: vamshigvk475
2 Replies

2. Shell Programming and Scripting

Output large volume of data to CSV file

I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message when I run the program. The program is not outputting to the csv file. Error: the file access permissions do not allow the specified action cannot... (2 Replies)
Discussion started by: dellanicholson
2 Replies

3. UNIX for Dummies Questions & Answers

How to create a volume group, logical volume group and file system?

hi, I want to create a volume group of 200 GB and then create different file systems on that. please help me out. Its becomes confusing when the PP calculating PP. I don't understand this concept. (2 Replies)
Discussion started by: kamaldev
2 Replies

4. UNIX for Dummies Questions & Answers

Confusion Regarding Physical Volume,Volume Group,Logical Volume,Physical partition

Hi, I am new to unix. I am working on Red Hat Linux and side by side on AIX also. After reading the concepts of Storage, I am now really confused regarding the terminologies 1)Physical Volume 2)Volume Group 3)Logical Volume 4)Physical Partition Please help me to understand these concepts. (6 Replies)
Discussion started by: kashifsd17
6 Replies

5. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

6. Filesystems, Disks and Memory

Configure large volume on Sun StorageTek 2540 array

Hi, We have 12x1TB SATA disks in our array and I need to create 10TB volume. I defined new storage profile on array and when I tried to add volume, I faced with ~2TB limit for new volumes. I didn't find how to set another limit on my storage profile. Is there is a way to configure one large... (3 Replies)
Discussion started by: Sapfeer
3 Replies

7. AIX

Basic Filesystem / Physical Volume / Logical Volume Check

Hi! Can anyone help me on how I can do a basic check on the Unix filesystems / physical volumes and logical volumes? What items should I check, like where do I look at in smit? Or are there commands that I should execute? I need to do this as I was informed by IBM that there seems to be... (1 Reply)
Discussion started by: chipahoys
1 Replies

8. UNIX for Advanced & Expert Users

Formatting large Ascii file

Hi, I've got a very large ASCII file (1.5 GB), which doesn't have line terminators, making it impossible to read it right, i need to format it like: add line terminators. delete some lines on the file. delete some fields on all the remainings lines (can be identified by a string). After... (4 Replies)
Discussion started by: fahoo
4 Replies

9. Solaris

How to resize mirror volume in veritas volume manager 3.5 on Solaris 9 OE

Hi all, I have a problem with vxvm volume which is mirror with two disks. when i am try to increase file system, it is throwing an ERROR: can not allocate 5083938 blocks, ERROR: can not able to run vxassist on this volume. Please find a sutable solutions. Thanks and Regards B. Nageswar... (0 Replies)
Discussion started by: nageswarb
0 Replies

10. UNIX for Advanced & Expert Users

LVM - Extending Logical Volume within Volume Group

Hello, I have logical volume group of 50GB, in which I have 2 logical volumes, LogVol01 and LogVol02, both are of 10GB. If I extend LogVol01 further by 10GB, then it keeps the extended copy after logical volume 2. I want to know where it keeps this information Regards Himanshu (3 Replies)
Discussion started by: ghimanshu
3 Replies

Featured Tech Videos