Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Removing White spaces from a huge file
Top Forums Shell Programming and Scripting Removing White spaces from a huge file Post 302990015 by Corona688 on Friday 20th of January 2017 02:29:48 PM
Old 01-20-2017
If you tell awk what your "lines" are, it won't have to read 2GB of data at once. RS and ORS variables control this. They usually default to newline, but they can as easily be <EOFD>.

Code:
 $ cat data
457 <EOFD> Mar  1 2007 12:00:00:000AM   <EOFD> Mar 31 2007 12:00:00:000AM   <EOFD>  system  <EORD> 458 <EOFD>    Mar  1 2007 12:00:00:000AM<EOFD>agf <EOFD> Apr 20 2007  9:10:56:036PM    <EOFD>

$  awk '{ sub(/ +$/, ""); sub(/^ +/, ""); } 1' RS="<EOFD>" ORS="<EOFD>" datafile ; echo

457<EOFD>Mar  1 2007 12:00:00:000AM<EOFD>Mar 31 2007 12:00:00:000AM<EOFD>system  <EORD> 458<EOFD>Mar  1 2007 12:00:00:000AM<EOFD>agf<EOFD>Apr 20 2007  9:10:56:036PM<EOFD>

$

..the "echo" afterwards is just to move the cursor to the next line, since it wouldn't print a newline otherwise.
Last edited by Corona688; 01-20-2017 at 03:35 PM..
This User Gave Thanks to Corona688 For This Post:
amvip
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

deleting white spaces in a file

Hello Guys, I am a newbie to unix. I am having a requirement. Please help me for finding a solution for this, I am having a file as mentioned below: $ cat shank ackca acackac akcajc akcjkcja akcj ckcklc I want to delete all the white spaces in this file, I tried... (2 Replies)
Discussion started by: mraghunandanan
2 Replies

2. Shell Programming and Scripting

ksh: removing all white spaces

'String' file contains the following contents, D11, D31, D92, D29, D24, using ksh, I want to remove all white spaces between characters no matter how long the string is. Would you please give me some help? (1 Reply)
Discussion started by: yoonius
1 Replies

3. Solaris

removing special characters, white spaces from a field in a file

what my code is doing, it is executing a sql file and the resullset of the query is getting stored in the text file in a fixed format. for that fixed format i have used the following code:: Code: awk -F":"... (2 Replies)
Discussion started by: priyanka3006
2 Replies

4. Shell Programming and Scripting

Removing blank spaces, tab spaces from file

Hello All, I am trying to remove all tabspaces and all blankspaces from my file using sed & awk, but not getting proper code. Please help me out. My file is like this (<b> means one blank space, <t> means one tab space)- $ cat file NARESH<b><b><b>KUMAR<t><t>PRADHAN... (3 Replies)
Discussion started by: NARESH1302
3 Replies

5. UNIX for Dummies Questions & Answers

[Solved] Help with using tr - Removing white spaces

Hi, I have a file that contains whitespaces with spaces and spaces and tabs on each line and am wanting to remove the whitespaces. My version of sed is one that does not recognize \t etc. The sed and awk one-liners below that I found via Google both does not work. So my next best... (3 Replies)
Discussion started by: newbie_01
3 Replies

6. Post Here to Contact Site Administrators and Moderators

Want a tcl script to compare a string in a file ignoring white spaces

Hi , I want a tcl script to search a string ignoring whitespaces in a .log file . It should correctly match . The string are as follows "Output-Maps 1 1 0 0 0" 1 and Active Intermediate-Maps 0 0 0 ... (1 Reply)
Discussion started by: kulua
1 Replies

7. UNIX for Advanced & Expert Users

Performance problem with removing duplicates in a huge file (50+ GB)

I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file. I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far.. Any suggestions please ? Thanks !! (9 Replies)
Discussion started by: Kannan K
9 Replies

8. Shell Programming and Scripting

Remove white spaces from flat file generated from Oracle table...

I have to export data from table into flat file with | delimited. In the ksh file, I am adding below to do this activity. $DBSTRING contains the sqlplus command and $SQL_STRING contains the SQL query. File is created properly with the data as per SQL command. I am getting white spaces in the... (1 Reply)
Discussion started by: mgpatil31
1 Replies

9. Shell Programming and Scripting

Removing blank/white spaces and special characters

Hello All , 1. I am trying to do a task where I need to remove Blank spaces from my file , I am usingawk '{$1=$1}{print}' file>file1Input :- ;05/12/1990 ;31/03/2014 ; Output:- ;05/12/1990 ;31/03/2014 ;This command is not removing all spaces from... (6 Replies)
Discussion started by: himanshu sood
6 Replies

10. Shell Programming and Scripting

List and Delete Files which are older than 7 days, but have white spaces in file name

I need to list and delete all files in current older which are olderthan 7 days. But my file names have white spaces. Before deleting I want to list all the files, so that I can verify.find . -type f -mtime +7 | xargs ls -l {} But the ls command is the working on the files which have white... (16 Replies)
Discussion started by: karumudi7
16 Replies

LEARN ABOUT SUSE

pm-powersave

PM-POWERSAVE(8) 					       pm-utils User Manual						   PM-POWERSAVE(8)

NAME

       pm-powersave - Put your computer into low power mode

SYNOPSIS

       pm-powersave [{true | false}]

DESCRIPTION

       This manual page documents briefly the pm-powersave command.

       pm-powersave can be used to set your system into low power mode.

OPTIONS

       true
	   Set the computer into low power mode to reduce energy consumption and extend your laptop's battery life.

       false
	   Turn off low power mode.

FILES

       /etc/pm/power.d/, /usr/lib/pm-utils/power.d/
	   When you run pm-powersave it combines the scripts in these two directories and executes them in sorted order. If both directories
	   contain a script with the same name, the one in /etc/pm/power.d/ has a higher precedence and only this one will be executed.

	   Therefore to disable a script from /usr/lib/pm-utils/power.d/ simply create an empty file in /etc/pm/power.d/ with the same name and
	   without the execute bit set.

SEE ALSO

       pm-suspend(8), pm-hibernate(8)

AUTHOR

       Michael Biebl <biebl@debian.org>
	   Manpage author.

COPYRIGHT

       Copyright (C) 2007 Michael Biebl

       This manual page was originally written for the Debian(TM) system, and has been adopted by the pm-utils project.

       Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 2 or (at
       your option) any later version published by the Free Software Foundation.

pm-powersave							    Mar 5, 2007 						   PM-POWERSAVE(8)
All times are GMT -4. The time now is 05:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy