Sponsored Content
Top Forums Shell Programming and Scripting Removing White spaces from a huge file Post 302990015 by Corona688 on Friday 20th of January 2017 02:29:48 PM
Old 01-20-2017
If you tell awk what your "lines" are, it won't have to read 2GB of data at once. RS and ORS variables control this. They usually default to newline, but they can as easily be <EOFD>.

Code:
 $ cat data
457 <EOFD> Mar  1 2007 12:00:00:000AM   <EOFD> Mar 31 2007 12:00:00:000AM   <EOFD>  system  <EORD> 458 <EOFD>    Mar  1 2007 12:00:00:000AM<EOFD>agf <EOFD> Apr 20 2007  9:10:56:036PM    <EOFD>

$  awk '{ sub(/ +$/, ""); sub(/^ +/, ""); } 1' RS="<EOFD>" ORS="<EOFD>" datafile ; echo

457<EOFD>Mar  1 2007 12:00:00:000AM<EOFD>Mar 31 2007 12:00:00:000AM<EOFD>system  <EORD> 458<EOFD>Mar  1 2007 12:00:00:000AM<EOFD>agf<EOFD>Apr 20 2007  9:10:56:036PM<EOFD>

$

..the "echo" afterwards is just to move the cursor to the next line, since it wouldn't print a newline otherwise.

Last edited by Corona688; 01-20-2017 at 03:35 PM..
This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

deleting white spaces in a file

Hello Guys, I am a newbie to unix. I am having a requirement. Please help me for finding a solution for this, I am having a file as mentioned below: $ cat shank ackca acackac akcajc akcjkcja akcj ckcklc I want to delete all the white spaces in this file, I tried... (2 Replies)
Discussion started by: mraghunandanan
2 Replies

2. Shell Programming and Scripting

ksh: removing all white spaces

'String' file contains the following contents, D11, D31, D92, D29, D24, using ksh, I want to remove all white spaces between characters no matter how long the string is. Would you please give me some help? (1 Reply)
Discussion started by: yoonius
1 Replies

3. Solaris

removing special characters, white spaces from a field in a file

what my code is doing, it is executing a sql file and the resullset of the query is getting stored in the text file in a fixed format. for that fixed format i have used the following code:: Code: awk -F":"... (2 Replies)
Discussion started by: priyanka3006
2 Replies

4. Shell Programming and Scripting

Removing blank spaces, tab spaces from file

Hello All, I am trying to remove all tabspaces and all blankspaces from my file using sed & awk, but not getting proper code. Please help me out. My file is like this (<b> means one blank space, <t> means one tab space)- $ cat file NARESH<b><b><b>KUMAR<t><t>PRADHAN... (3 Replies)
Discussion started by: NARESH1302
3 Replies

5. UNIX for Dummies Questions & Answers

[Solved] Help with using tr - Removing white spaces

Hi, I have a file that contains whitespaces with spaces and spaces and tabs on each line and am wanting to remove the whitespaces. My version of sed is one that does not recognize \t etc. The sed and awk one-liners below that I found via Google both does not work. So my next best... (3 Replies)
Discussion started by: newbie_01
3 Replies

6. Post Here to Contact Site Administrators and Moderators

Want a tcl script to compare a string in a file ignoring white spaces

Hi , I want a tcl script to search a string ignoring whitespaces in a .log file . It should correctly match . The string are as follows "Output-Maps 1 1 0 0 0" 1 and Active Intermediate-Maps 0 0 0 ... (1 Reply)
Discussion started by: kulua
1 Replies

7. UNIX for Advanced & Expert Users

Performance problem with removing duplicates in a huge file (50+ GB)

I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file. I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far.. Any suggestions please ? Thanks !! (9 Replies)
Discussion started by: Kannan K
9 Replies

8. Shell Programming and Scripting

Remove white spaces from flat file generated from Oracle table...

I have to export data from table into flat file with | delimited. In the ksh file, I am adding below to do this activity. $DBSTRING contains the sqlplus command and $SQL_STRING contains the SQL query. File is created properly with the data as per SQL command. I am getting white spaces in the... (1 Reply)
Discussion started by: mgpatil31
1 Replies

9. Shell Programming and Scripting

Removing blank/white spaces and special characters

Hello All , 1. I am trying to do a task where I need to remove Blank spaces from my file , I am usingawk '{$1=$1}{print}' file>file1Input :- ;05/12/1990 ;31/03/2014 ; Output:- ;05/12/1990 ;31/03/2014 ;This command is not removing all spaces from... (6 Replies)
Discussion started by: himanshu sood
6 Replies

10. Shell Programming and Scripting

List and Delete Files which are older than 7 days, but have white spaces in file name

I need to list and delete all files in current older which are olderthan 7 days. But my file names have white spaces. Before deleting I want to list all the files, so that I can verify.find . -type f -mtime +7 | xargs ls -l {} But the ls command is the working on the files which have white... (16 Replies)
Discussion started by: karumudi7
16 Replies
DATEFUDGE(1)							      Debian							      DATEFUDGE(1)

NAME
datefudge - pretend the system time is different SYNOPSIS
datefudge [-s|--static] at_date program [arguments ...] DESCRIPTION
datefudge is a small utility that pretends that the system time is different by pre-loading a small library which modifies the time(2), gettimeofday(2) and clock_gettime(2) system calls. OPTIONS
--static, -s set date as a `static' one. The above mentioned system calls will always return the date passed as a parameter of the program regardless of time passing. See EXAMPLES below. --help, -h print short usage information and exit. --version, -v print version information and exit. EXAMPLES
Basic example: $ datefudge "2007-04-01 10:23" date -R Sun, 01 Apr 2007 10:23:00 +0200 Non-static vs. static example: $ datefudge "2007-04-01 10:23" sh -c "sleep 3; date -R" Sun, 01 Apr 2007 10:23:03 +0200 $ datefudge --static "2007-04-01 10:23" sh -c "sleep 3; date -R" Sun, 01 Apr 2007 10:23:00 +0200 AUTHOR
Written by Matthias Urlichs <smurf@noris.de>. Modified by Robert Luberda <robert@debian.org>. BUGS
There is no attempt to make this change undetectable by the program. In particular, file modification times are not modified. COPYRIGHT
Copyright (C) 2003 by Matthias Urlichs. Copyright (C) 2008-2011 by Robert Luberda. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. You may redistribute copies of datefudge under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. SEE ALSO
ld.so(1), time(2), gettimeofday(2), clock_gettime(2) datefudge 1.17 June 23th, 2011 DATEFUDGE(1)
All times are GMT -4. The time now is 12:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy