Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Removing special chars from file and maintain field separator
Top Forums UNIX for Advanced & Expert Users Removing special chars from file and maintain field separator Post 302941795 by iffy290 on Tuesday 21st of April 2015 08:33:06 AM
Old 04-21-2015
Removing special chars from file and maintain field separator
Running SunOs 5.6. Solaris.

I've been able to remove all special characters from a fixed length file which appear in the first column but as a result all subsequent columns have shifted to the left by the amount of characters deleted.
It is a space separated file. Line 1 in input file is corrupt. Line 2 is what it should look like. The string 000022000362700 starts at position 49 on both lines. The problem I'm having is that after removing the 3 special chars the field moves to position 46.

Code:
Line 1
        GAVISCON LIQUID PEPPERMINT �OT        000022000362700   159588000007979400  50001584182        0006S020000

Line 2
        GAVISCON LIQUID PEPPERMINT OT        000022000362700   159588000007979400  50001584182        0006S020000


The command I'm using is as follows :

Code:
cat file.txt | grep '[^ -~]' | sed 's/[^ -~]//g'

This produces following output:

Code:
        GAVISCON LIQUID PEPPERMINT OT        000022000362700   159588000007979400  50001584182        0006S020000

By removing the special characters every field to the right of the changed field has moved to the left changing the field start positions.

It appears that the line of text is not showing the correct character spacing but all fields from the first numeric field onwards should have the same starting position.

I've been searching for a while now and cannot find any solution for this issue.

How should I proceed?
Last edited by iffy290; 04-21-2015 at 10:12 AM..
 

10 More Discussions You Might Find Interesting

1. Solaris

removing special characters, white spaces from a field in a file

what my code is doing, it is executing a sql file and the resullset of the query is getting stored in the text file in a fixed format. for that fixed format i have used the following code:: Code: awk -F":"... (2 Replies)
Discussion started by: priyanka3006
2 Replies

2. Shell Programming and Scripting

Help in formating a file with field separator

hi thanks a lot But pls help me if my input file is the below format: Name priya 2010-09-21 10:43:49 TEXT ID 1 hi TEXT ID 2 how TEXT ID 3 r TEXT ID 4 u Name2 priya2 2010-09-21 10:43:49 TEXT ID 1 hi1 TEXT ID 2 how1 TEXT ID 3 r1 (4 Replies)
Discussion started by: bha148
4 Replies

3. Shell Programming and Scripting

awk, comma as field separator and text inside double quotes as a field.

Hi, all I need to get fields in a line that are separated by commas, some of the fields are enclosed with double quotes, and they are supposed to be treated as a single field even if there are commas inside the quotes. sample input: for this line, 5 fields are supposed to be extracted, they... (8 Replies)
Discussion started by: kevintse
8 Replies

4. Shell Programming and Scripting

Removing unknow chars from file names.

I'm trying to move a large folder to an external drive but some files have these weird chars that the external drive won't accept. Does anyone know any command of any bash script that will look through a given folder and remove any weird chars? (4 Replies)
Discussion started by: Joktaa
4 Replies

5. Shell Programming and Scripting

Removing rows and chars from text file

Dear community, maybe I'm asking the moon :rolleyes:, but I'm scratching my head to find a solution for it. :wall: I have a file called query.out (coming from Oracle query), the file is like this: ADDR TOTAL -------------------- ---------- TGROUPAGGR... (16 Replies)
Discussion started by: Lord Spectre
16 Replies

6. Shell Programming and Scripting

How to reverse all columns of a file having some field separator?

Hello, I have a file: xandyandz x & y & z x*y*z*a I require output as: zandyandx z & y & x a*z*y*x here all lines have different field seperator (and & * )based on that i want to reverse the column of a file. Pl. help. (8 Replies)
Discussion started by: nehashine
8 Replies

7. Shell Programming and Scripting

Matching two files with special field separator

Hello, I have a file with such structure: >ENSGALG00000000011|ENSGALT00000000012|57|1123|1125 AACTGTGTGTTTTTT >ENSGALG00000000012|ENSGALT00000000013|57|1145|1155 AAAAAAGGTCCTGTGTGC >ENSGALG00000000015|ENSGALT00000000014|57|1144|1155 AAAATGTGTGTGTGTGTGTGTG I want to use another file... (8 Replies)
Discussion started by: Homa
8 Replies

8. Shell Programming and Scripting

shell scripting to determine special chars in file

Hi, I need all your help to achieve the below functionality. I have a big 2 GB file and inside the file we need to identify, whether having a comma(,) or pipe(|) or tab or fixed position or semicolon(;) delimiter. If any of those delimiter found need to replace the file with pipe(|)... (1 Reply)
Discussion started by: lkeswar
1 Replies

9. UNIX for Beginners Questions & Answers

Change encoding, no removing special chars. inconv

Hi all, I'm using iconv command to change files encoding to UTF-8 If my input file has chars as those are removed creating the file without those special chars. I tried using iconv -c, but there is still the removal. Is there a way to keep those special chars changing just the... (6 Replies)
Discussion started by: mrreds
6 Replies

10. UNIX for Beginners Questions & Answers

Shell script to split data with a delimiter having chars and special chars

Hi Team, I have a file a1.txt with data as follows. dfjakjf...asdfkasj</EnableQuotedIDs><SQL><SelectStatement modified='1' type='string'><! The delimiter string: <SelectStatement modified='1' type='string'><! dlm="<SelectStatement modified='1' type='string'><! The above command is... (7 Replies)
Discussion started by: kmanivan82
7 Replies

LEARN ABOUT ULTRIX

mvwdelch

delch(3cur)															       delch(3cur)

Name
       delch, mvdelch, mvwdelch, wdelch - remove character from window

Syntax
       #include <cursesX.h>

       int delch()

       int wdelch(win)
       WINDOW *win;

       int mvdelch(y, x)
       int y, x;

       int mvwdelch(win, y, x)
       WINDOW *win;
       int y, x;

Description
       The  routine  deletes  the character under the cursor in the default window.  All characters to the right on the same line are moved to the
       left one position and the last character on the line is filled with a blank.  The cursor position does not change.

       The routine deletes the character under the cursor in the specified window.  All characters to the right on the same line are moved to  the
       left one position and the last character on the line is filled with a blank.  The cursor position does not change.

       The  routine moves the cursor to the specified position in the default window.  The character found at this location is deleted.  All char-
       acters to the right on the same line are moved to the left one position and the last character on the line is filled  with  a  blank.   The
       cursor position does not change.

       The  routine  moves  the  cursor  to the specified position in the specified window.  The character found at this location is deleted.  All
       characters to the right on the same line are moved to the left one position and the last character on the line is filled with a blank.  The
       cursor position does not change.

       The routines and are macros.

Return Values
       The and functions return OK on success and ERR on error.

																       delch(3cur)
All times are GMT -4. The time now is 03:11 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy