AWK Data Cleaning Post: 302456679

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers AWK Data Cleaning Post 302456679 by carmar87 on Saturday 25th of September 2010 03:25:52 AM

09-25-2010

Registered User

AWK Data Cleaning

Hello,

I am trying to analyze data I recently ran, and the only way to efficiently clean up the data is by using an awk file.
I am very new to awk and am having great difficulty with it. In $8 and $9, for example, I am trying to delete numbers that contain 1.
I cannot find any tutorials that tell me how to do this. I would greatly appreciate if someone could point me in the right direction.

Code:

"RECORDING_SESSION_LABEL"	"DATA_FILE"	"TRIAL_LABEL"	"trial"	"word"	"SAMPLE_BUTTON"	"LEFT_PUPIL_SIZE"	"LEFT_IN_BLINK"	"LEFT_IN_SACCADE"

"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1872.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1874.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1873.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1873.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1873.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1872.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1872.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1872.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1871.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1870.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1872.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1875.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1880.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1882.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1886.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1888.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1888.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1887.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1886.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1885.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1883.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1881.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1880.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1880.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1880.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1880.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1879.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1879.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1878.00	0	1
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1877.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1878.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1879.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1881.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1883.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1885.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1885.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1885.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1886.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1887.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1890.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1892.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1893.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1893.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1893.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1892.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1891.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1890.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1892.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1892.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1893.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1893.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1895.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1897.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1897.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1896.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1895.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1895.00	0	0
"test"	"test.edf"	"Trial: 1"	1	>>>>>	.	1897.00	0	0

Last edited by Scott; 09-25-2010 at 04:36 AM.. Reason: Please use code tags

carmar87

View Public Profile for carmar87

Find all posts by carmar87

9 More Discussions You Might Find Interesting

1. AIX

doing some spring cleaning....

USERS="me you jim joe sue" for user in ${USERS}; do rmuser -p $user usrdir=`cat /etc/passwd|grep $user|awk -F":" '{ print $6 }'` rm -fr `cat /etc/passwd|grep $user|awk -F":" '{ print $6 }'` echo Deleting: $user '\t' REMOVING: $usrdir done This is for AIX ONLY!!! but easily ported to...

2. Shell Programming and Scripting

Data Cleaning in a file

Hi , I have the below source data, I need to clean the data in 2nd,4th,5th columns. Source Data RECORD,CASH_TRANS,BEJING,AUG2011/CASH_TRANS,Y/N150/CASH_TRANS,N/201108 RECORD,CASH_TRANS,INDIA,AUG2011/CASH_TRANS,Y/NC110/CASH_TRANS,N/201108...

3. Shell Programming and Scripting

File cleaning

HI , I am getting the source data as below. Source Data CDR_Data,,,,, F1,F2,F3,F4,F5,F6 5,5,6,7,8,7 6,6,g,,, 7,7,76,,, 8,8,gt,,, 9,9,df ,d,d,d ,,,,,

4. Shell Programming and Scripting

cleaning the file

Hi, I have a file with multiple rows. each row has 8 columns. Column 8 has entries separated by commas. I want to exclude all the rows in which column 8 has more than 3 commas. 1234#0/1 - ABC_1234 3 ATGCATGCATGC HHHIIIGIHVF 1 49:T>C,60:T>C,78:C>A,76:G>T,65:T>G Thanks, Diya

5. Shell Programming and Scripting

Cleaning AWK code

Hi I need some help to clean my code used to get city location. wget -q -O - http://www.ip2location.com/ | grep chkRegionCity | awk 'END { print }' | awk -F"" '{print $4}' It gives me the city but have a leading space. I am sure this could all be done by one single AWK Also if possible...

6. Shell Programming and Scripting

Help with parsing data with awk , eliminating unwanted data

Experts , Below is the data: --- Physical volumes --- PV Name /dev/dsk/c1t2d0 VG Name /dev/vg00 PV Status available Allocatable yes VGDA 2 Cur LV 8 PE Size (Mbytes) 8 Total PE 4350 Free PE 2036 Allocated PE 2314 Stale PE 0 IO Timeout (Seconds) default --- Physical volumes ---...

7. Shell Programming and Scripting

Cleaning output using awk

I have some small problem with my code. data.html <TD class="statuscol2">c</TD> <TD class="statuscol3">18</TD> <TD class="statuscol4"><SPAN TITLE="#04">test4</SPAN></TD> <TD...

8. Shell Programming and Scripting

Cleaning through perl or awk a Stemmer dictionary

Hello, I work under Windows Vista and I am compiling an open-source stemmer dictionary for English and eventually for other Indian languages. The Engine which I have written has spewed out all lemmatised/expanded forms of the words: Nouns, Adjectives, Adverbs etc. Each set of expanded forms is...

9. Shell Programming and Scripting

awk --> math-operation in data-record and joining with second file data

Hi! I have a pretty complex job - at least for me! i have two csv-files with meassurement-data: fileA ......