Removing "Hidden Characters" on a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing "Hidden Characters" on a file
# 8  
Old 06-24-2011
The problem with removing ALL non-keyboard characters is that the needed hidden characters will be deleted too.

The file is generated from mysql, then dumped to a file. The file will be sed to replace the comma with |. The | is the colseparator on the mysql.

I don't know much about chomp. I'll be researching on that.

---------- Post updated at 05:50 PM ---------- Previous update was at 05:33 PM ----------

Quote:
perl -i -pe 's/[^\x20-\x7f]//g' filename.csv
This one worked. But it removed all the Hidden Characters.

I just want to remove the hidden character on 'InterUnit$' string and its result from sql query.

InterUnit column will give a Y or N result from the database.

---------- Post updated 06-24-11 at 02:40 PM ---------- Previous update was 06-23-11 at 05:50 PM ----------

Hi all, I am now facing a more complicated problem with hidden characters '$'.

I noticed that the '$' characters are printed in a particular pattern resulting for my CSV file to produce incorrect format upon the execution of the ksh on Linux.

Do you guys know how to remove unnecessary hidden characters on Linux in ksh?

There are unnecessary '$' characters on my output file, but not all '$'. As we know, '$' is like \n on hidden character when we used "set list" in vi.

If my question is unclear, please let me know how could I elaborate it better.
# 9  
Old 06-24-2011
how you are taking dump (using which command) from mysql DB ?
what is your email command ? (which sends the file as attachment ) ?
# 10  
Old 06-24-2011
I have a function to call the run_sql.ksh script using sqlplus.
Whenever I call the function, I dump the result on a $LOGFILE and also spool it on ${file}.csv
Now, the CSV file will be "sed" using the following commands:
Code:
#Remove commas seen on all results
 sed 's/,/ /g' ${file}.csv > $CSV_RESULT_PIPE
#Change pipes delimiter to comma
 sed 's/|/,/g' $CSV_RESULT_PIPE > $CSV_RESULT_UNZIPPED

The $CSV_RESULT_UNZIPPED file will be compressed to ${CSV_RESULT}
Then the following command:
Code:
uuencode ${CSV_RESULT} ${CSV_FILE_ZIPPED} > $EMAIL_ATTACHMENT

Then send the email using:

Code:
cat $EMAIL_HEADER | /usr/sbin/sendmail -t $RECIPIENTS

Note that the $EMAIL_HEADER includes the $EMAIL_ATTACHMENT, and I can receive the email successfully but the data in it are misplaced.
# 11  
Old 06-24-2011
ok, in the inital csv file itself the $ value appears ?

${file}.csv # is the file having $value ?
Code:
 
#Remove commas seen on all results
sed 's/,/ /g' ${file}.csv > $CSV_RESULT_PIPE

# 12  
Old 06-24-2011
Yes, it has. The value is the real file name to be generated. Please consider file=CSV_FILE:

Code:
 
export CSV_FILE=$PATH/MI_RESULT_OF_QUERY_DATETIME

The file is successfully generated and received but the content is not aligned properly.
# 13  
Old 06-24-2011
Quote:
The file is successfully generated and received but the content is not aligned properly.
Please describe what you see, mentioning what program you are using to view the file. If it is the "stairs" effect it will almost certainly be due to the line terminator.

This happens when you send a unix format text file to a MSDOS platform.
In unix the line terminator is just a linefeed character. This is conventionally represented as "$" or "\n" in unix.
In MSDOS the line terminator is two characters: carriage-return then linefeed.
You will probably need to convert the file to MSDOS format before using "uuencode".

We need to know what Operating System and version you have.
Where supplied, the program is variously called "ux2dos" or "unix2dos". Otherwise the conversion can be achieved with "awk". In your case you could presumably handle the whole conversion (including the CSV bit) in SQL.
# 14  
Old 06-26-2011
deleted

wrong data

Last edited by Jin_; 06-27-2011 at 05:24 AM.. Reason: *
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. Shell Programming and Scripting

Removing duplicates on a single "column" (delimited file)

Hello ! I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts... Could you guys help me removing the duplicate lines of a file, based only on a single "column"? For example: M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies

3. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

4. Shell Programming and Scripting

finding the strings beween 2 characters "/" & "/" in .txt file

Hi all. I have a .txt file that I need to sort it My file is like: 1- 88 chain0 MASTER (FF-TE) FFFF 1962510 /TCK T FD2TQHVTT1 /jtagc/jtag_instreg/updateinstr_reg_1 dff1 (TI,SO) 2- ... (10 Replies)
Discussion started by: Behrouzx77
10 Replies

5. Shell Programming and Scripting

Removing "^M" from the end of a String (i.e. "Ctrl+M")?

Hello All, I have an Expect script that ssh's to a remote server and runs some commands before exiting. One of the commands I run is the "hostname" Command. After I run this command I save the output using this line in the code below... Basically it executes the hostname command, then I... (2 Replies)
Discussion started by: mrm5102
2 Replies

6. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

7. Shell Programming and Scripting

Removing special characeter "~V" in a unix file

I have the Unix XML file as below: <?xml version="1.0" encoding="UTF-8"?> <ReportData version="1.0"><DisplayName>Non-Agency CMO Daily Trade Recap - Hybrids</DisplayName><ReportType>MgmtTradingReport</ReportType><Description>Management Trading... (7 Replies)
Discussion started by: mohsin.quazi
7 Replies

8. Solaris

removing "/" file system from solaris volume

Hi all, I have created a volume for the root device as d0 and the sub mirror for same is d10. the output from metastat d0 is as below I want to clear these volume , as i cant unmount the "/ " file system , please suggest as how can i clear this. Also the required entries are there... (2 Replies)
Discussion started by: kumarmani
2 Replies

9. Shell Programming and Scripting

How to remove "New line characters" and "spaces" at a time

Dear friends, following is the output of a script from which I want to remove spaces and new-line characters. Example:- Line1 abcdefghijklmnopqrstuvwxyz Line2 mnopqrstuvwxyzabcdefghijkl Line3 opqrstuvwxyzabcdefdefg Here in above example, at every starting line there is a “tab” &... (4 Replies)
Discussion started by: anushree.a
4 Replies

10. UNIX for Dummies Questions & Answers

File names that contain "01" act hidden

Haveing an issue. Anytime a file is created with "01" (zero one) in the name - it cannot be viewed by LS or any other file listing command. Although the file is there, it cannot be seen. I can edit it, run it, anything, except see it..... What happened? Any ideas? (8 Replies)
Discussion started by: n9ninchd
8 Replies
Login or Register to Ask a Question