Help with sed and replacing white spaces with commas
Dear all,
I am in a bit of a quandary. I have 400 text files which I need to edit and output in a very specific way.
Here is a sample text file copied from gedit ... The columns will come out a bit messed up but when I cat <file>, it gives a table with six columns (0-28, tot lob vol, vcsf, scsf, gray, white):
Basically, I need to do the following:
1. Delete the first four rows
2. Delete the first column (numbers 1-28)
3. Paste each row to the end of the previous row, so one file comes out as one super long line
4. Comma-delimit all numbers
5. Add new first field at the beginning of the line containing the file name
6. Output all of this to a new text file which will take each of the 400 or so text files I have as above, convert them each to their own line consisting of steps 1-5.
So, an example output file from just the above file would be:
I know zippo about programming but pasting together various snippets I've found online have come up with the following which does not seem to working exactly right ... my output is joining some numbers, particularly those from the end of one line and the beginning of another.
Code:
#!/bin/sh
#script file placed in the directory above that which contains all the text files
#script run by typing ../script.sh from within the directory containing the files
files=`ls -l *_stats.txt`
# all files end in _stats.txt
for file in ${files}
do
cat ${file} | sed '1,4d'| awk '{print $2, $3, $4, $5, $6}' | sed 's/[:space:]+/,/g' >> output.txt
#read file, delete lines 1-4, print all but the first column, delete all whitespace #between numbers including carriage returns and put them all on one line with #comma-delimiters; then new line with the output of the next file
done
Any help would be most appreciated. It is sad, but I have spent HOURS trying to figure this out and I am pretty sure it would be ridiculously simple for anyone with basic bash scripting knowledge. Please help!
For removing the first 4 rows you can use the sed command as follows
Code:
sed '1,4d' input_file
For removing the first field which contains the numbers (0-28) you can use the sed command as follows
Code:
sed 's/\(^[ ]*[0-9]\{1,2\}\)//g'
Quote:
3. Paste each row to the end of the previous row, so one file comes out as one super long line
I think you want to join all the lines
Heres a sed solution for this
Code:
sed '$!N;s/\n/ /'
Quote:
4. Comma-delimit all numbers
Code:
tr -s ' ' | sed 's/ /,/g'
Here why I am using tr -s means it contains repeated space characters. I am squeezing the repeated characters
into single space and I am converting the space to comma character.
Last edited by thillai_selvan; 03-13-2010 at 05:07 AM..
files=`ls *_stats.txt`
for file in ${files}
do
content=`cat ${file} | sed '1,4d'| sed -r "s/^[0-9]+\s(.+$)/\1/g" | sed -r "s/\s+/,/g" | tr '\n' '\0'`
echo "$file,$content" >> output.txt
done
files=$(ls -1 *_stats.txt)
for file in ${files}
do
cat ${file} | sed '1,4d' | awk '{printf ("'${file}'"==pfile?","$2","$3","$4","$5","$6:"\n""'${file}'"","$2","$3","$4","$5","$6)}{pfile="'${file}'"}' >> output.txt
done
Last edited by malcomex999; 03-13-2010 at 06:32 AM..
Hi;
In following code
find LOG_DIR -type f | while read filename; do echo $filename; done
I want to precede each white space encountered in filename with \ so that when i use $filename for running some commands in do...done,it wont give me an error.
will appreciate ur help in this.... (1 Reply)
SHELL SCRIPT
Hi
I have a file in the following format
Mayank Sushant
Dheeraj Kunal
ARUN Samir
How can i replace the white space in between and replace them with a comma?? The resultant output should be
Mayank,Sushant
Dheeraj,Kunal
ARUN,Samir
i tried using
sed -e... (8 Replies)
Hi,
Newbie here. I have a file that consists of data that I want to convert to a csv file. For example:
Jul 20 2008 1111 / visit home / BlackBerry8830/4.2.2 Profile/MIDP-2.0 Configuration/CLOC-1.1 VendorID/105
Jul 21 2008 22222 / add friend / BlackBerry8830/4.2.2 Profile/MIDP-2.0... (3 Replies)
Hey guys,
I am very much new to shell scripts. So you ppl may feel that i am asking stupid question here. :D
1. I am using command line argument as an input variable. The user gets this value in his mail from client which has commas n spaces (Eg. 12,34,56,789) and the scripts... (5 Replies)
hello all,
i have a comma delimited file.
i want to replace the commas in the file with the tilde symbol using sed.
how can i do this?
thanks. (4 Replies)
I have a file which contains numbers as follows:
1234 9876 6789 5677 3452
9087 4562 1367 2678 7891
I need to remove the empty spaces and add commas between the numbers like:
1234,9876,6789,5677,3452,
9087,4562,1367,2678,7891
Can anyone tell me the command to do... (4 Replies)
Hi,
Can anybody suggest me how to combine two strings with two or more white spaces and assign it to a variable?
E.g.
first=HAI
second=HELLO
third="$first $second" # appending strings with more than one white spaces
echo $third
this would print
HAI HELLO
Output appears... (2 Replies)
I have a variable that calls in a string from txt file. Problem is the string comes with an abundance of white spaces trailing it. Is there any easy way to trim the tailing white spaces off at the end? Thanks in advance. (9 Replies)
hi all...
i have the next question:
i have a flat file with a lot of records (lines). Each record has 10 fields, which are separated by pipe (|). My problem is what sometimes, in the first record, there are white spaces (no values, nothing) in the beginning of the record, like this:
ws ws... (2 Replies)
How would I delete white spaces in a specified file?
Also, I'd like to know what command I would use to take something off a regular expression, and put it onto another.
ie.
.
.
.
expression1 <take_off>
.
.
.
expression2 (put here)
.
.
.
Any help would be great, thanks! (10 Replies)
