Need help to delete special characters exists only at the end of the each record in UNIX file? Post: 303029667

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Need help to delete special characters exists only at the end of the each record in UNIX file? Post 303029667 by Don Cragun on Wednesday 30th of January 2019 09:09:29 AM

01-30-2019

Registered User

Quote:

Originally Posted by rakeshp

Hi Don,

Except space,all other non numeric characters are treated as special characters. ^M represents two characters.

Thanks
Rakesh

--- Post updated at 07:10 AM ---

Actual data contains 16fields only, but I have shown here only the sample data where I cannot show the actual data here.

--- Post updated at 07:12 AM ---

I understand that I should show the actual sample data, But i cannot do that here

I never asked you to show us actual data; I only asked you to show us representative sample data. You could easily have taken a couple of examples of real data and replaced every occurrence of a numeric character with a "9" and replaced every occurrence of a lowercase alphabetic character with an "x" and every occurrence of an uppercase alphabetic character with an "X" (except for things like <circumflex><capital-latin-M> where you should have left the "M" as it appears in your real data.

Why you would need to tell us that there are 20 fields and that you want to remove special characters from field 15 when there are only 15 fields in your actual data makes absolutely no sense to me. Why you would explicitly want us to remove the characters "^" and "M" from the 15th field (which is also the last field in your actual data) and then remove any characters that are not numeric or <space> characters from the same field makes no sense to me. Since "^" and "M" are both non-<space> and non-<digit> characters, removing what you are calling "special" characters from the 15th field will remove "^" and "M" from the end of the record without adding any special code to just remove those two characters.

If the above accurately describes your real data and what you are trying to do to it (i.e., remove all characters that are not <space> and are not numeric from the last field of each record where each record contains 15 fields), then all you need is something like:

Code:

awk 'BEGIN {FS="|"; OFS="|"} {gsub("[^ [:digit:]]", "", $15)} 1'

If you want to remove all characters that are not <space> and are not numeric from the 15th field in each record where each record contain 15 or more fields, and you also want to remove the two character sequence ^M from the end of each input record, then you need something more like:

Code:

awk 'BEGIN {FS="|"; OFS="|"} {sub("[^]M$", ""); gsub("[^ [:digit:]]", "", $15)} 1' file

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Unix file does not display special characters

We have a unix file that contains special characters (ie. �, �, �, � , � , � ). When I try to read this file I get a codepage error and the characters are replaced by the # symbol. How do I keep the special characters from being read? Thanks. Ryan

2. UNIX for Dummies Questions & Answers

How to delete a file with special characters

I don't now exactly how I did it, but I created a file named " -C " cexdi:/home1 $ls -lt total 1801336 -rw------- 1 cexdi ced-group 922275840 23 mars 10:03 -C How do I delete this file ? cexdi:/home1 $rm -C rm: invalid option -- C Syntax : rm filename ... Doesn't work......

3. UNIX for Dummies Questions & Answers

Advice on extracting special characters from a DB2 table to a file in the UNIX ENV

need some advice on the following situation. I have a DB2 table which has a varchar Column. This varchar column can have special characters like �, �, � . When I extract from this table to a sequential file for this varchar column I am only able to get � and � . To Get the �...

4. Shell Programming and Scripting

sed delete pattern with special characters

Hi all, I have the following lines A gtwrhwrthwr text hghthwrhtwrtw ; text text (text) B gtwrhwrthwr text hghthwrhtwrtw ; text text (text) J gtwrhwrthwr text hghthwrhtwrtw ; text text (text) and I would like to...

5. Shell Programming and Scripting

how to delete special characters from the file content

Hello Team, Any one suggest how to delte the below special character from a file which is having one column 10 rows of same below content. ---------------------------------------- Kosten|bersicht gemd_ ' =Welche Kosten kvnnen...

6. Shell Programming and Scripting

Windows to UNIX FTP Special characters!

I have a file that has the name in one of the lines as MAR�A MENDOZA in Windows. When this gets FTPed over to UNIX it appears as MAR�A MENDOZA. Is there anyway to overcome this? Its causing a issue because the file is Postional and fields are getting pushed by 2 digits.. Any help would be...

7. Shell Programming and Scripting

Need unix commands to delete records from one file if the same record present in another file...

Need unix commands to delete records from one file if the same record present in another file... just like join ... if the record present in both files.. delete from first file or delete the particular record and write the unmatched records to new file.. tried with grep and while...

8. Shell Programming and Scripting

How to add trailer record at the end of the flat file in the unix ksh shell scripting?

Hi, How to add trailer record at the end of the flat file in the unix ksh shell scripting can you please let me know the procedure Regards Srikanth

9. Shell Programming and Scripting

UNIX Special Characters

Any time I do : ls *.txt > mytext.txt I get something like this in the output file: ^ Tue Jan 22 16:19:19 EST 2013 x86_64 x86_64 x86_64 GNU/Linux t1Fam_BrOv :~>alias | grep ls alias l.='ls -d .* --color=tty' alias lR='ls -R' alias la='ls -Al' alias lc='ls -ltcr' alias ldd='ls -ltr |...

10. Shell Programming and Scripting

Delete special characters

My sed is not working on deleting the entire special characters and leaving what is necessary.grep connections_per a|sed -e 's/\<\!\-\-//g' INPUT:  <method>HALF</method> <!--...

LEARN ABOUT DEBIAN

srec_tektronix_extended

srec_tektronix_extended(5)					File Formats Manual					srec_tektronix_extended(5)

NAME

       srec_tektronix_extended - Tektronix Extended hexadecimal file format

DESCRIPTION

       This  format allows binary files to be uploaded and downloaded between two computer systems, typically between a computer system (such as a
       PC, Macintosh, or workstation) and an emulator or evaluation board for microcontrollers and microprocessors.

   The Lines
       Lines always start with a percent (%) character.  Each line consists of 5 fields.  These are the length field, the type field,  the  check-
       sum, the address field (including address length), and the data field.

   The Fields
						     +--+--------+------+----------+---------+------+
						     |% | Length | Type | Checksum | Address | Data |
						     +--+--------+------+----------+---------+------+
       Record Length
	       The  record length field is a 2 character (1 byte) field that specifies the number of characters (not bytes) in the record, exclud-
	       ing the percent, the length field, the type field and the checksum.

       Type    The type field is a 1 character field that specifies whether the record is data (6) or termination (8).

       Checksum
	       The checksum is an 2 character (1 byte) field that represents the sum of all the nibbles on the line, excluding the checksum.

       Address This is a 9 character field.  The first character is the address size; it is always 8.  The remaining 8 chgaracters are the  4-byte
	       address that specifies where the data is to be loaded into memory.

       Data    The data field contains the executable code, memory-loadable data or descriptive information to be transferred.

   Record Types
       6       A record containing data.  The data is placed at the address specified.

       8       A  termination  record.	The address field may optionally contain the address of the instruction to which control is passed.  There
	       is no data field.

   Size Multiplier
       In general, binary data will expand in sized by approximately 2.5 times when represented with this format.

EXAMPLE

       Here is an example Tektronix extended file.  It contains the data "Hello, World[rq] to be loaded at address 0x006B.
	      %256D980000006B48656C6C6F2C20576F726C64210A
	      %09819800000000

COPYRIGHT

       srec_cat version 1.58
       Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Peter Miller

       The srec_cat program comes with ABSOLUTELY NO WARRANTY; for details use the 'srec_cat -VERSion License' command.  This is free software and
       you are welcome to redistribute it under certain conditions; for details use the 'srec_cat -VERSion License' command.

AUTHOR

       Peter Miller   E-Mail:	pmiller@opensource.org.au
       //*		 WWW:	http://miller.emu.id.au/pmiller/

Reference Manual						      SRecord						srec_tektronix_extended(5)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Unix file does not display special characters

Discussion started by: Ryan2786

2. UNIX for Dummies Questions & Answers

How to delete a file with special characters

Discussion started by: yveslagace

3. UNIX for Dummies Questions & Answers

Advice on extracting special characters from a DB2 table to a file in the UNIX ENV

Discussion started by: cosec

4. Shell Programming and Scripting

sed delete pattern with special characters

Discussion started by: stinkefisch