Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Replace char between chars - help needed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replace char between chars - help needed
# 8  
Old 07-07-2012
A quick link while searching through forum

remove newline characters between two delimiters

hope this helps you!! Smilie
This User Gave Thanks to PikK45 For This Post:
go0ogl3
# 9  
Old 07-07-2012
Lightbulb
Thank you Pikk45! One problem solved!

Using
Code:
awk -F'\^\|' '{while(NF<4 && getline p)$0=$0 p}1'

on the file I can remove max 4 new lines on the 4th field.
The delimiter used is ^| which is at the end of the 4th field.

Now I must replace the "|" between the "^"...

---------- Post updated at 12:15 PM ---------- Previous update was at 09:47 AM ----------

I have come to the:

Code:
awk ' {for (i=1;i<=NF;i++) if (i!=NF) {printf (5<i && i<(NF-1))?$i"-":$i"|"} print $NF} ' FS='|' OFS='^'

but this replaces all "|" till the end... how can I tell awk to stop replacing at first "^|"?

---------- Post updated at 04:19 PM ---------- Previous update was at 12:15 PM ----------

Hello,

I finally found the answer. It's not so pretty, but it works and is extremely fast (a few seconds) on a almost 2Mb file:

Code:
cat csv_in_file | awk -F'\^\|' '{while(NF<4 && getline p)$0=$0 p}1' \
| sed 's/%/_procent_/g'\
| awk '{for (i=1;i<=NF;i++) if (i!=NF) {printf (3<i && i<(NF-1))?$i"!":$i"|"} print $NF}' FS='|' OFS='^' \
| perl -npe 's/^(.*)!(.*)!(.*)!(.*)!(.*)!(.*)!(.*)!(.*)!(.*)!(.*)!(.*)$/$1|$2|$3|$4|$5|$6|$7|$8|$9|$10|$11/' \
| sed 's/_procent_/%/g' > csv_out_file

I'll try to explain what every command does:
- first awk sets the field delimiter to "^|" and deletes at most 3 new lines (I've discovered I have at most 4 lines of data in the 4th record = 3 new lines)
- the sed replaces the "%" with "_precent_" because I found that it breaks somehow the next awk command...
- the second awk command replaces all the "|" symbols beginning with the 4th field with "!" (I'm sure there will never be "!" in my last fields)
- the perl command replaces the last "!" with "|"
- the last sed replaces back the "%"

The result is a csv file with pipe as a delimiter wich I can use...

Maybe there is a more elegant solution but this works and is damn fast compared to the windows excel.
Hope this will help others too.
# 10  
Old 07-07-2012
Can you post few lines of your csv file so that it we can analyze and try this in a single line?

This looks like a long long command, but yes it is indeed faster! Smilie
# 11  
Old 07-07-2012
I've selected 3 lines from the file:

Code:
^TV Tuner^|LogiLink|VG0011|^Stick USB pt. captura audio-video, Logilink ""VG0011""^|18.50|EUR|s|24|0|Link||scutit|||
^Tablete Internet^|GoClever|^TAB I71^|^Tableta GoClever TAB I71, InfoTMIC iMAPx220 1GHz (ARM11, GPU GC600), Android 2.3, 256MB DDR2, 4GB MLC NAND flash, 7" LCD TFT 800x480 Backlight - LED. 16:9 16bit culori.
Resistiv single-touch, Wi-Fi (802.11 b/g), AVI, 3GP, MP4, RM, RMVB, FLV, MKV, MOV, VOB, DAT, WMV, AVI,
 3GP, MP4, RM, RMVB, FLV, MKV, MOV, VOB, DAT, WMV, MP3|WAV|OGG|FLAC|APE|AAC, Conectori: 1x mini jack 3.5 mm,
 2 porturi USB 2.0, 4000 mAh lithium-ion polymer 3.7V^|89.00|USD|w|24|0|Link||^5 ron^|||
^Notebook - accesorii^|DeepCool|N17|^Stand notebook DeepCool 14"" - metal, fan, USB, dimensiuni 330X250X25mm, dimensiuni Fan 140X140X15mm, Fan Speed 1000ą10%RPM, Max.
 Air Flow 47.35CFM, zgomot 21dB(A), Hydro Bearing, ''N17''^|12.20|USD|s|12|0|Link|^Promo achizitii saptamanale^|scutit|||

The source xls is UTF-8, if this matters.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Replace \n char in Data

File is pipe delimited with 17 fields. We may get \n char (1 or more \n in one field or multi fileds) in data in any field. Need to replace \n in data with space and not the Ture \n that is line separator. I tried below awk command it did not work as expected. awk '(NR-1)%2{$1=$1}1' RS=\|... (9 Replies)
Discussion started by: rajeshkumare
9 Replies

2. UNIX for Dummies Questions & Answers

Grep or sed to search, replace/insert chars!

HI All Im trying to come up with an approach to finding a string, using a portion of that string to insert it on lines starting with the value "GOTO" appending to end of line after removing PT's ( See example below! ) EXAMPLE: 1. I would like to search for the line that starts with "TLAXIS/"... (7 Replies)
Discussion started by: graymj
7 Replies

3. UNIX for Dummies Questions & Answers

Replacing multiple special chars with single char

Hi I've a string . And i need to replace set of characters with a single character Means .. or . or ... and so on should be replaced with single % character Irrespective of number of dots in between the characters , those should be replaced with single % All the above strings should be... (3 Replies)
Discussion started by: smile689
3 Replies

4. Shell Programming and Scripting

Find and replace all extended char.

Hi Guys, I wand find and replace all Extended ASCII Codes from all my log files. My Log files: /home/Kalr/PPool/Output i have logs file in sub dir. /home/Kalr/PPool/Output/X /home/Kalr/PPool/Output/Y /home/Kalr/PPool/Output/Z My Abc.log file input: Extended ASCII Codes :– ... (4 Replies)
Discussion started by: asavaliya
4 Replies

5. Shell Programming and Scripting

Replace char on text file

Hi , I have problem on replace char on text file #!/bin/bash echo "Enter Third value : " read a sed '2 s/192.160.1.1/cut -d"." -f3/$a/g' tcpip.txt > a.txt I want replace line 2 on tcpip.txt with value a that I input but it's not run Please help me (2 Replies)
Discussion started by: phillipss
2 Replies

6. Shell Programming and Scripting

Help needed to remove a char from a specified sub-string

Hi! I'm having trouble usind sed to remove the char ' from within a database's varchar register's sql. For example, on the following sql: INSERT INTO patrimonio_municipal.patrimonio_municipal_airc_tmp Values('|Estação Elevatória|',|16723|,'|Grandes Reparações|', '|2010-03-26... (3 Replies)
Discussion started by: cremat0rio
3 Replies

7. Shell Programming and Scripting

How to replace any char with newline char.

Hi, How to replace any character in a file with a newline character using sed .. Ex: To replace ',' with newline Input: abcd,efgh,ijkl,mnop Output: abcd efgh ijkl mnop Thnx in advance. Regards, Sasidhar (5 Replies)
Discussion started by: mightysam
5 Replies

8. Shell Programming and Scripting

Replace Junk chars (Sed)

I know this has been asked previously on this forum...But I think I have a different scenario to present. I ahve a file tht looks like this (note:there are control Z and other chars tht are not visible on line with anme bowers) BB7118450 6004718 BIANCALANA =HEI BZ5842819 ... (4 Replies)
Discussion started by: alfredo123
4 Replies

9. Shell Programming and Scripting

replace chars,

:rolleyes: Hi, I want to replace the particular text in the middle of the line. Ex. In the line 40-50 wanted to replace the char as 'x' (7 Replies)
Discussion started by: Jairaj
7 Replies

10. Shell Programming and Scripting

replace ascii chars without loosing it.

Hi, Can some one tell, how to replace ascii non printable TAB from the while to something, then later on replace it back to TAB. Basciallz we do bulk data processing, our processin treats TAB as new field , So I thought we can replace it with something and later on revert it. TIA (4 Replies)
Discussion started by: braindrain
4 Replies
Login or Register to Ask a Question