Removing blank/white spaces and special characters


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing blank/white spaces and special characters
# 1  
Old 08-13-2014
Removing blank/white spaces and special characters

Hello All ,

1. I am trying to do a task where I need to remove Blank spaces from my file , I am using
Code:
awk '{$1=$1}{print}' file>file1

Code:
Input :-
;05/12/1990                      ;31/03/2014                         ;
Output:-
;05/12/1990 ;31/03/2014 ;

This command is not removing all spaces from file . Still i can see a single space before each semicolon.

Could you help me on this .

2. My file also contains some special characters specially French characters.
Code:
derniÉres annÅes

When I try replacing this character using sed/awk , I am unable to replace them that is I am unable to print them on command prompt . When I copy paste these two words in my command special characters replace them with some other characters.

So until I print them in my command, I cannot replace them.

Any way to do this I need to replace that A stuff with small a and E stuff with small e.

Kindly help on this.

Thanks,
Himanshu Sood

Last edited by rbatte1; 08-14-2014 at 06:48 AM..
# 2  
Old 08-13-2014
Code:
 
tr -d ' ' < file > file1

or
Code:
sed 's/ //g' file > file1

awk approach
Code:
awk '{gsub(/ /, x)}1' file > file1

# 3  
Old 08-13-2014
Quote:
Originally Posted by SriniShoo
Code:
awk '{gsub(/ /, x)}1' file > file1

Could you explain me what is this doing ?

Also why my awk was unable to remove all spaces?

Thanks,
Himanshu Sood
# 4  
Old 08-13-2014
Quote:
Originally Posted by himanshu sood
Could you explain me what is this doing ?

Also why my awk was unable to remove all spaces?

Thanks,
Himanshu Sood
Heregsub(/ /, x), we are replacing all the spaces with nothing
In your code, you are resetting the field separator from multiple spaces to 1, it will supresses the continues spaces to one
# 5  
Old 08-13-2014
Quote:
Originally Posted by himanshu sood
Also why my awk was unable to remove all spaces?
$1=$1 removes all empty fields but maintains the default fieldseparators (spaces) between existing fields.

You could remove the spaces if you set an empty value to the built-in variable OFS (Output Field Separator):
Code:
awk '$1=$1' OFS= file

# 6  
Old 08-13-2014
Quote:
Originally Posted by Franklin52
$1=$1 removes all empty fields but maintains the default fieldseparators (spaces) between existing fields.

You could remove the spaces if you set an empty value to the built-in variable OFS (Output Field Separator):
Code:
awk '$1=$1' OFS= file

@Franklin, if $1==0, it will not print the output.
Code:
awk '{$1=$1}1' OFS= file

# 7  
Old 08-13-2014
Hi.

For the second task, one could use iconv, like so:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate conversion of UTF-8 to ASCII, iconv.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C iconv

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

pl " Results:"
LANG=en_US.UTF-8 LC_ALL="" iconv -c -f utf8 -t ascii//TRANSLIT $FILE |
tee data2
pe
file $FILE data2

exit 0

producing:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian 5.0.8 (lenny, workstation) 
bash GNU bash 3.2.39
iconv (GNU libc) 2.7

-----
 Input data file data1:
derniÉres annÅes

-----
 Results:
derniEres annAes

data1: UTF-8 Unicode text
data2: ASCII text

I usually use "C" for the locale, but you may need to set it to something different as noted on the line executing the iconv command. See man pages for details.

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing White spaces from a huge file

I am trying to remove whitespaces from a file containing sample data as: 457 <EOFD> Mar 1 2007 12:00:00:000AM <EOFD> Mar 31 2007 12:00:00:000AM <EOFD> system <EORD> 458 <EOFD> Mar 1 2007 12:00:00:000AM<EOFD>agf <EOFD> Apr 20 2007 9:10:56:036PM <EOFD> prodiws<EORD> . Basically these... (11 Replies)
Discussion started by: amvip
11 Replies

2. Solaris

Removing blank spaces

Hi , I want to go out of vi editor temporarily and execute a command in command prompt and again going back to the editor . Is it possible . Any help on this is really helpful. 1. Need to open vi 2. Temporarily come out and execute a command and go back to vi editor (6 Replies)
Discussion started by: rogerben
6 Replies

3. AIX

Replace all TAB characters with white spaces

Dear Gurus Can you please advise me on how to Replace all TAB characters with white spaces in a text file in AIX? Either using vi or any utilities (2 Replies)
Discussion started by: tenderfoot
2 Replies

4. UNIX for Dummies Questions & Answers

[Solved] Help with using tr - Removing white spaces

Hi, I have a file that contains whitespaces with spaces and spaces and tabs on each line and am wanting to remove the whitespaces. My version of sed is one that does not recognize \t etc. The sed and awk one-liners below that I found via Google both does not work. So my next best... (3 Replies)
Discussion started by: newbie_01
3 Replies

5. Shell Programming and Scripting

Removing blank spaces, tab spaces from file

Hello All, I am trying to remove all tabspaces and all blankspaces from my file using sed & awk, but not getting proper code. Please help me out. My file is like this (<b> means one blank space, <t> means one tab space)- $ cat file NARESH<b><b><b>KUMAR<t><t>PRADHAN... (3 Replies)
Discussion started by: NARESH1302
3 Replies

6. Solaris

removing special characters, white spaces from a field in a file

what my code is doing, it is executing a sql file and the resullset of the query is getting stored in the text file in a fixed format. for that fixed format i have used the following code:: Code: awk -F":"... (2 Replies)
Discussion started by: priyanka3006
2 Replies

7. Shell Programming and Scripting

(g)awk how to preseve white spaces (FS characters) or read a right subpart of $0?

Hi, I am using gawk (--posix) for extracting some information from something like the following lines (in a text file): sms_snath_hp_C/CORE BUILD PREREQUISITE: total 1556 drwxrwxrwx 2 sn sn 4096 2008-06-27 08:31 ./ drwxrwxrwx 13 sn sn 4096 2009-07-22 14:48 ../ -rwxrwxrwx 1 sn sn ... (14 Replies)
Discussion started by: shri_nath
14 Replies

8. Shell Programming and Scripting

ksh: removing all white spaces

'String' file contains the following contents, D11, D31, D92, D29, D24, using ksh, I want to remove all white spaces between characters no matter how long the string is. Would you please give me some help? (1 Reply)
Discussion started by: yoonius
1 Replies

9. UNIX for Dummies Questions & Answers

Replace only first found white spaces with some other characters

Anybody can help me How can I replace only four first white spaces with , or any other characters aaaa 08/31/2004 08/31/2009 permanent Logical Processors in System: 64 bedad 08/16/2001 08/15/2011 permanent Logical Processors in System: 64 badnv14 05/31/2008 05/30/2013 permanent Logical... (5 Replies)
Discussion started by: pareshan
5 Replies

10. Shell Programming and Scripting

Removing blank spaces from a file?

Guys, I need some help... how can I remove the blank spaces between the lines below? (between the date and the hour fields) 21/05/07 00:05:00 99 21/05/07 00:10:01 99 21/05/07 00:15:00 99 21/05/07 00:20:00 99 21/05/07 00:25:00 99 I want to make the file... (4 Replies)
Discussion started by: dfs
4 Replies
Login or Register to Ask a Question