Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Reading in all files from parent directory (GAWK)
Top Forums Shell Programming and Scripting Reading in all files from parent directory (GAWK) Post 302539618 by gd9629 on Monday 18th of July 2011 08:59:33 AM
Old 07-18-2011
The Format_trial.csv.awk script;

Code:
#!/bin/gawk -f


# This file is to restructure the picarro data into the correct .csv columns for R


# create a header with same headings as variable in table
# also set other variables before parsing data
BEGIN   {
        OFS="," 	# tells awk that the output separator is a comma
        ORS=""  	# tells awk to not print newline after each print command so all records are
					# on the same line until we want a new line "\n"			
		getline}	# removes 1st line of input file ie header so we can replace it with correct one

# rearrange the yyyy-mm-dd | hh:mm:ss date and time to single date column of yyyy/mm/dd hh:mm:ss needed for openair
{print substr($1,9,2) "/" substr($1,6,2) "/" substr($1,1,4) " " substr($2,1,5)}		

# print the rest of variables as columns 
{print (" ", $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20)
}


$1 $2 != prev {
    	
	{print "\n"}	# newline after each 5 seconds of data has been parsed
  
; prev=$1 $2}

Yes this awk script works on one file, it creates a combined .csv file with the correct format. The problem is trying to input all the files at once (with one command) as opposed to having to call in every days file.

Yes the output (for the 'new' combined script) is just an empty 'june.csv' file, maybe it's not recognising the june.dat as the input?

The desired data file is just a data frame containing 20 variables. The awk file was essentially used to combine the data and time columns into one column with the correct format I need to input into an R package.

Cheers guys!

---------- Post updated at 01:59 PM ---------- Previous update was at 01:52 PM ----------

Ah I see where I went wrong. I was using 'IN_ALL' to call in the file and then 'IN_all' to produce the file! My how I'm learning how precise programming can be!

Another quick thing guys, this programme is going to be run everyday to create the .csv file. However, whenever I run it, the data is just added to the old data, so I essentially have the same results twice if I run the programme again, and 3 times if I run it again and so on. Is there anyway to tell awk to write over the old file?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading files in directory

Hi Everyone , have a nice day i need a help on this thing algo is something like in certain path like /root/user1 i have many files , i need a code which could open every file one by one and then each file has contents like this <moid>CcnCounters=CAPv3-Received-Total-Requests, Source =... (3 Replies)
Discussion started by: Dastard
3 Replies

2. Shell Programming and Scripting

gawk - reading two files & re arrange the columns

Hi, I am trying to read 2 files and writing to the 3rd file if I find the same elements in 2 files. my first file is 1 0 kb12124819 766409 1.586e-01 1 0 kb17160939 773886 8.674e-01 1 0 kb4475691 836671 8.142e-01 1 0 ... (2 Replies)
Discussion started by: ezhil01
2 Replies

3. UNIX for Dummies Questions & Answers

Listing files in a non-parent directory

Hi, Edit: The title should really read listing files in a non-parent directory, sorry! Im trying to get one of my Bash scripting assignments done for uni and now I'm stuck. This is probably going to be one of those kick yourself moments but, in my script I have a variable usrDir which... (2 Replies)
Discussion started by: Adzi
2 Replies

4. Shell Programming and Scripting

Find files inside the parent directory only

Hi All, The following find command lists the files which are 45 minutes older. But it searches for the sub directories also. $ find . -type f -mmin +45 -print ./hello.txt ./test/hi.txt ./temp/now.txt ls hello.txt test temp How can i modify the find command in such way that it finds... (4 Replies)
Discussion started by: Tuxidow
4 Replies

5. Shell Programming and Scripting

reading files from a directory.

Can some body help me to code this? go to a specific directory.(/home/abcd/test) file1.txt, file2.txt, ... .. filen.txt read the files in side the folder 'test' and print the content of each file into other folder in the same directory lets say(testresult) with the same file name... (4 Replies)
Discussion started by: rocking77
4 Replies

6. UNIX for Dummies Questions & Answers

Need Help in reading N days files from a Directory & combining the files

Hi All, Request your expertise in tackling one requirement in my project,(i dont have much expertise in Shell Scripting). The requirement is as below, 1) We store the last run date of a process in a file. When the batch run the next time, it should read this file, get the last run date from... (1 Reply)
Discussion started by: dsfreddie
1 Replies

7. Shell Programming and Scripting

How to list all Subdirectories and files with its full path in a parent directory?

How to list all Subdirectories and files with its full path in a parent directory? (1 Reply)
Discussion started by: johnveslin
1 Replies

8. Shell Programming and Scripting

Help on Backing up all the files in the subdirectories under a parent directory

Hi, I am not too familiar with Unix scripting but I have to write code to find all the files under all the sub directories under a parent directory of unix location and move them to the corresponding Windows location. For eg: I have \home\sreenu\Files\ Under neath this I have multiple sub... (3 Replies)
Discussion started by: raj.sreenu
3 Replies

9. Shell Programming and Scripting

Copying files from parent directory only

Hello, Please can someone assist on a issue I am having. I want to find specific files in the parent directory only that have been modified over the last 2 days and copy them to another location. NOTE: The version of AIX I am using does not have MAXDEPTH. I have currently written the... (3 Replies)
Discussion started by: Dolph
3 Replies

10. Shell Programming and Scripting

Find/searching files in subdirectories excluding the fiels in Parent Directory

Hi All, requirement is to find and remove the files from sub directories but it should exclude the files from parent directory. At present i am using the below one but it finds and remove files from both parent and sub directories. find ${PATH} -type f \( -name securitas\* -o -name \*gz... (1 Reply)
Discussion started by: Naveenkk
1 Replies

LEARN ABOUT OSX

unshar

UNSHAR(1)						      General Commands Manual							 UNSHAR(1)

NAME

       unshar - unpack a shar file

SYNOPSIS

       unshar [ options ] [ file ... ]

DESCRIPTION

       Unshar  scans mail messages looking for the start of a shell archive.  It then passes the archive through a copy of the shell to unpack it.
       It will accept multiple files.  If no files are given, standard input is used.

OPTIONS

       Options have a one letter version starting with - or a long version starting with --.  The exception is --help and  --version,  which  does
       not have a short version.

       --version
	      Print the version number of the program on standard output, then immediately exits.

       --help Print a help summary on standard output, then immediately exits.

       -d DIRECTORY  --directory=DIRECTORY
	      Change directory to DIRECTORY before unpacking any files.

       -c  --overwrite
	      Passed  as  an option to the shar file.  Many shell archive scripts (including those produced by `shar' 3.40 and newer) accepts a -c
	      argument to indicate that existing files should be overwritten.

       -e  --exit-0
	      This option exists mainly for people who collect many shell archives into a single mail folder.  With this option, `unshar' isolates
	      each different shell archive from the others which have been put in the same file, unpacking each in turn, from the beginning of the
	      file towards its end.  Its proper operation relies on the fact that many shar files are terminated by a `exit 0' at the beginning of
	      a line.

	      Option -e is internally equivalent to -E "exit 0".

       -E STRING  --split-at=STRING
	      This option works like -e, but it allows you to specify the string that separates archives if `exit 0' isn't appropriate.

	      For  example,  noticing  that  most `.signatures' have a `--' on a line right before them, one can sometimes use `--split-at=--' for
	      splitting shell archives which lack the `exit 0' line at end.  The signature will then be skipped altogether with the headers of the
	      following message.

       -f  --force
	      The same as -c.

SEE ALSO

       shar(1)

DIAGNOSTICS

       Any message from the shell may be displayed.

AUTHORS

       The  shar  and  unshar  programs is the collective work of many authors.  Many people contributed by reporting problems, suggesting various
       improvements or submitting actual code.	A list of these people is in the THANKS file in the sharutils distribution.

REPORTING BUGS

       Report bugs to <bug-gnu-utils@gnu.org>.	Please put sharutils in the subject line.  It helps to spot the message.

								September 10, 1995							 UNSHAR(1)
All times are GMT -4. The time now is 02:21 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy