Processing files using awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Processing files using awk
# 1  
Old 09-15-2010
Processing files using awk

Hi

I have files in our UNIX directory like the below
Code:
-rw-r--r--   1 devinfo    devsupp        872 Sep 14 02:09 IMGBTREE27309_12272_11_1_0_FK.idx0
-rw-r--r--   1 devinfo    devsupp        872 Sep 14 02:09 IMGBTREE27309_12272_11_0_0_PK.idx0
-rw-r--r--   1 devinfo    devsupp        432 Sep 14 02:09 IMGBTREE27309_12272_11_0.dat0
-rw-r--r--   1 devinfo    devsupp          0 Sep 14 02:09 IMGBTREE27309_12272_11_0.dat1
-rw-r--r--   1 devinfo    devsupp       8192 Sep 15 13:24 PMLKUP11200_35_0_27813H64.dat1
-rw-r--r--   1 devinfo    devsupp          0 Sep 15 13:24 PMLKUP11200_35_0_27813H64.idx1
-rw-r--r--   1 devinfo    devsupp        872 Sep 15 13:24 PMLKUP11200_35_0_27813H64.idx0
-rw-r--r--   1 devinfo    devsupp        432 Sep 15 12:44 PMLKUP11200_35_0_27794H64.dat0
-rw-r--r--   1 devinfo    devsupp        872 Sep 15 12:44 PMLKUP11200_35_0_27794H64.idx0
-rw-r--r--   1 devinfo    devsupp        432 Sep 15 13:24 PMJNR11200_52_0_27813.dat0
-rw-r--r--   1 devinfo    devsupp        872 Sep 15 13:24 PMJNR11200_52_0_27813.idx0
-rw-r--r--   1 devinfo    devsupp        872 Sep 15 12:44 PMJNR11200_52_0_27794.idx0
-rw-r--r--   1 devinfo    devsupp        432 Sep 15 12:44 PMJNR11200_52_0_27794.dat0

1. I want only the fields $6, $7, $8, $9 and $5
2. From the field $9, I do not need the extensions ie., I do not need .idx0, .dat0, .dat1 etc.,
3. Then group the files based on $9

So finally my output should look like the below
Code:

Sep 14 02:09 IMGBTREE27309_12272_11_1_0_FK      872
Sep 14 02:09 IMGBTREE27309_12272_11_0_0_PK      872
Sep 14 02:09 IMGBTREE27309_12272_11_0           432 
Sep 15 13:24 PMLKUP11200_35_0_27813H64         9044
Sep 15 12:44 PMLKUP11200_35_0_27794H64         1304
Sep 15 13:24 PMJNR11200_52_0_27813             1304
Sep 15 12:44 PMJNR11200_52_0_27794             1304

I have this script -
Code:
ls -l|awk '{arr[$9]+=$5} END {for (i in arr) {print i,arr[i]}}'

which will just give the total with extensions. After this I am stuck. Can someone please help me on this?

Last edited by Franklin52; 09-16-2010 at 03:32 AM.. Reason: Please use code tags!
# 2  
Old 09-15-2010
Don't forget the code tags.

Code:
rule1: ls -l |awk '{print $6, $7, $8, $9 , $5}'
rule2: ls -l |awk '{split($9,a,".");print $6, $7, $8, a[1] , $5}' 
rule3: ls -l |awk '{split($9,a,".");print $6, $7, $8, a[1] , $5}'  |sort -k3n

# 3  
Old 09-15-2010
Code:
# awk '{split($NF,a,".");x=$6FS $7FS $8FS a[1];if(x!=y){print x,$5};y=x}' file
Sep 14 02:09 IMGBTREE27309_12272_11_1_0_FK 872
Sep 14 02:09 IMGBTREE27309_12272_11_0_0_PK 872
Sep 14 02:09 IMGBTREE27309_12272_11_0 432
Sep 15 13:24 PMLKUP11200_35_0_27813H64 8192
Sep 15 12:44 PMLKUP11200_35_0_27794H64 432
Sep 15 13:24 PMJNR11200_52_0_27813 432
Sep 15 12:44 PMJNR11200_52_0_27794 872

Please use [code] tags when you post code or sample data.
# 4  
Old 09-15-2010
Hi rdcwayx

This script helps a lot and does everything but i also want the bytes to be added in the $9 field. The result of your script:
Code:
ls -l |awk '{split($9,a,".");print $6, $7, $8, a[1] , $5}'  |sort -k3n

comes like the below
Code:
Sep 15 13:24 PMLKUP11200_35_0_27813H64 432
Sep 15 13:24 PMLKUP11200_35_0_27813H64 8192
Sep 15 12:44 PMLKUP11200_35_0_27813H64 872
Sep 15 13:24 PMJNR11200_52_0_27813     872
Sep 15 13:24 PMJNR11200_52_0_27813     432

But I want my output like
Code:
Sep 15 13:24 PMLKUP11200_35_0_27813H64  9496
Sep 15 13:24 PMJNR11200_52_0_27813      1304

I request you to help.

Thanks.

Last edited by Franklin52; 09-16-2010 at 05:51 AM.. Reason: Please use code tags!
# 5  
Old 09-15-2010
Ok, you need sum $5 by group.

but with same filename, $6,$7,$8 (date/time) will be different, which one you'd like to keep?
# 6  
Old 09-16-2010
Code:
ruby -e 'Dir["*"].each{|x| print "#{File.mtime(x)} #{x} #{File.size(x)}\n"}'

# 7  
Old 09-16-2010
Hi rdcwayx

Then I do not need (date&time)$6, $7 and $8 fields. I just need $5 and $9 fields. From the the below

Code:
PMLKUP11200_35_0_27813H64 432
PMLKUP11200_35_0_27813H64 8192
PMLKUP11200_35_0_27813H64 872
PMJNR11200_52_0_27813     872
PMJNR11200_52_0_27813     432

the result should be
Code:
Sep 15 13:24 PMLKUP11200_35_0_27813H64  9496
Sep 15 13:24 PMJNR11200_52_0_27813      1304

Please help.


Thanks.

Last edited by Franklin52; 09-16-2010 at 05:07 AM.. Reason: Please use code tags, thank you!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Passing multiple files to awk for processing in bash script

Hi, I'm using awk command in bash script. I'm able to pass multiple files to awk for processing.The code i can use is as below(sample code) #!/bin/bash awk -F "," 'BEGIN { ... ... ... }' file1 file2 file3 In the above code i'm passing the file names manually and it is fine till my... (7 Replies)
Discussion started by: shree11
7 Replies

2. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

3. Shell Programming and Scripting

Processing multiple files awk

hai i need my single awk script to act on 4 trace files of ns2 and to calculate througput and it should print result from each trace file in a single trace file. i tried with the following code but it doesnt work awk -f awkscript inputfile1 inputfile2 inputfile3 inputfile4>outputfile ... (4 Replies)
Discussion started by: sarathyy
4 Replies

4. Shell Programming and Scripting

processing with awk

I have many lines like the following in a file(there are also other kinds of lines) Host: 72.52.104.74 (tserv1.fmt2.he.net) Ports: 22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/, 179/open/tcp//tcpwrapped/// Ignored State: closed (997) Seq Index: 207 IP ID Seq: All... (9 Replies)
Discussion started by: esolvepolito
9 Replies

5. Shell Programming and Scripting

awk script processing data from 2 files

Hi! I have 2 files containing data that I need to process at the same time, I have problems in reading a different number of lines from the different files. Here is an explanation of what I need to do (possibly with an awk script). File "samples.txt" contains data in the format: time_instant... (6 Replies)
Discussion started by: Alice236
6 Replies

6. Shell Programming and Scripting

get all files from a directory and pass the files for processing

Hi All, I have a directory in which there will be several files. i want to get all the files and pass it to a piece of code for processing on the files. This is the piece of code which does the processing. tr "\n" "|" < (log file name) | tr "$" "\n" > output echo ' ' >>output while... (1 Reply)
Discussion started by: suresh_kb211
1 Replies

7. UNIX for Dummies Questions & Answers

single output of awk script processing multiple files

Helllo UNIX Forum :) Since I am posting on this board, yes, I am new to UNIX! I read a copy of "UNIX made easy" from 1990, which felt like a making a "computer-science time jump" backwards ;) So, basically I have some sort of understanding what the basic concept is. Problem Description:... (6 Replies)
Discussion started by: Kasimir
6 Replies

8. Shell Programming and Scripting

Writing output into different files while processing file using AWK

Hi, I am trying to do the following using AWK program. 1. Read the input data file 2. Parse the record and see if it contains errors 3. If the record contains errors, then write it into Reject file, else, write into usual output file or display it on the screen Here is what I have done -... (6 Replies)
Discussion started by: vidyak
6 Replies

9. Shell Programming and Scripting

awk processing

Hi all Is there a way in awk to know that you are processing your final line of input if you do no know how many lines were in the input to begin with? Thanks (7 Replies)
Discussion started by: pxy2d1
7 Replies

10. Shell Programming and Scripting

Preparing LaTeX files using Sed/AWK for processing with latex2html

As the title states, my issue involves preparing LaTeX documents for processing with latex2html. Within my LaTeX docs I have used a package which allows me to cleanly display source code listings. However the package is not supported by latex2html which, when processed, does not display the... (3 Replies)
Discussion started by: oski
3 Replies
Login or Register to Ask a Question