Help with data processing, maybe awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with data processing, maybe awk
# 1  
Old 08-06-2010
Help with data processing, maybe awk

I have a file, first 5 columns are very normal, like
Code:
"1107",106027,71400,"Y","BIOLOGY",

,
however, the 6th columns, the user can put comments, anything, just any characters, like new line, double quote, single quote, whatever from the keyboard, like
Code:
"Please load my previous SOM597G course content in the fall of 2009 into this course. I have two sections in this course.
Thank you.
Andy Pam"

, the 7th column is very normal, either "", or "Check with admin".

The thing is how I can extract the comment column out for each record, since it can be any kind of characters.

Thanks a lot!
Code:
"1107",106027,71400,"Y","BIOLOGY","please check with me.
Thanks.
Linda", ""
"1107",106027,71401,"Y","BIOLOGY","thanks", "Check with admin"
"1107",106027,71402,"Y","BIOLOGY", "Please refer to course "fa09 biology",
thanks.",""
"1107",106027,71403,"Y","BIOLOGY", "",""
"1107",106027,71404,"Y","BIOLOGY","Hey,
I am new for this semester.
Thanks.
Doris.",""

want the output to be like this:
Code:
comment1: please check with me.
Thanks.
Linda
comment2: thanks
comment3: Please refer to course "fa09 biology",
thanks.
comment4:
comment5: Hey,
I am new for this semester.
Thanks.
Doris.

# 2  
Old 08-07-2010
Hi
Code:
#sed 's/\"//g' a | awk '/^[0-9]+/{print "comment"i++":",$6;next}{print $1}' FS=,  i=1
comment1: please check with me.
Thanks.
Linda
comment2: thanks
comment3:  Please refer to course fa09 biology
thanks.
comment4:
comment5: Hey
I am new for this semester.
Thanks.
Doris.
#

Guru.
# 3  
Old 08-07-2010
thanks a lot! It is working on this temp file, and I will test on the official file again on Monday. Thank you so much!

Actually, the double quote, comma are allowed in the comments. Need to think about a more secure way for the field separators.
# 4  
Old 08-08-2010
Code:
sed 's/,/|/5' file|sed 's/.*|//;s/,[ \t]*"Check with admin"//;s/,[ \t]*"".*//'

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Data Processing

I have below Data *************************************************** ********************BEGINNING-1******************** directive url is : https://coursera-eu.mokar.com/directives/96df29ff-176a-35f7-8b1b-4ce483d15762 Src urls are :... (8 Replies)
Discussion started by: nikhil jain
8 Replies

2. Shell Programming and Scripting

awk processing of variable number of fields data file

Hy! I need to post-process some data files which have variable (and periodic) number of fields. For example, I need to square (data -> data*data) the folowing data file: -5.34281E-28 -3.69822E-29 8.19128E-29 9.55444E-29 8.16494E-29 6.23125E-29 4.42106E-29 2.94592E-29 1.84841E-29 ... (5 Replies)
Discussion started by: radudownload
5 Replies

3. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

4. Shell Programming and Scripting

Data processing using awk

Hello, I have some bitrate data in a csv which is in an odd format and is difficult to process in Excel when I have thousands of rows. Therefore, I was thinking of doing this in bash and using awk as the primary application except that due to its complication, I'm a little stuck. ... (24 Replies)
Discussion started by: shadyuk
24 Replies

5. UNIX for Dummies Questions & Answers

Genomic data processing

Dear fellow members, I've just joined the forum and am a newbie to shell scripting and programming. I'm stuck on the following problem. I'm working with large scale genomic data and need to do some analyses on it. Essentially it is text processing problem, so please don't mind the scientific... (0 Replies)
Discussion started by: mvaishnav
0 Replies

6. Programming

Data processing

Hello guys! I have some issue in how to processing some data. I have some files with 3 columns. The 1st column is a name of my sample. The 2nd column is a numerical sequence (very big sequence) starting from "1". And the 3rd column is a feature of each line, represented for a number (completely... (2 Replies)
Discussion started by: bfantinatti
2 Replies

7. Shell Programming and Scripting

awk script processing data from 2 files

Hi! I have 2 files containing data that I need to process at the same time, I have problems in reading a different number of lines from the different files. Here is an explanation of what I need to do (possibly with an awk script). File "samples.txt" contains data in the format: time_instant... (6 Replies)
Discussion started by: Alice236
6 Replies

8. Shell Programming and Scripting

How should i know that the process is still processing data

I have some process . How should i know that the process is still processing data or got hanged even though it is showing that it is running in background I know of a command called truss. how should i use this command and determine 1) process is still processing data 2) process got hanged... (7 Replies)
Discussion started by: ali560045
7 Replies

9. UNIX for Dummies Questions & Answers

Data File Processing Help

I need to read contents of directory and create a list of data files that match a certain pattern and process by renaming it and calling a existing .ksh script then archiving off to file another directory. Any suggestions or samples u could point me to on using .ksh perl or other to process... (5 Replies)
Discussion started by: mavsman
5 Replies

10. UNIX for Advanced & Expert Users

data processing

hi i am having a file of following kind: 20015#67143645#143123#4214 62014#67143148#67143159#456 15432#67143568#00143862#4632 54112#67143752#0067143657#143 54623#67143357#167215#34531 65446#67143785#143598#7456 75642#67143546#156146#845 24464#67143465#172532#6544... (5 Replies)
Discussion started by: rochitsharma
5 Replies
Login or Register to Ask a Question