Sponsored Content
Top Forums Shell Programming and Scripting Output large volume of data to CSV file Post 302956686 by Don Cragun on Friday 2nd of October 2015 04:49:39 AM
Old 10-02-2015
  1. The output from ls -l is a text file; not a CSV file.
  2. The "cannot access directory" errors can only be fixed by running the find command as a user who has permission to read and search /sasem and all of its subdirectories.
  3. It is not clear where the memory allocation error is coming from. If mail can't handle the 9.9+Mb attachment you are trying to attach to your mail; there is nothing we can do to help you.
  4. You said you are making a "a program that output sic the ownership and permission on each directory and file", but your find command only looks at regular files; not directories.
If you are running as a user with permission to read and search the file hierarchy rooted in /sasem, you want to list all files in your output (not just regular files), and mail can handle the size of the attachment you are creating, the following should work:
Code:
#!/bin/bash
monitorf=/sc/sb/monitor.txt
find /sasem -exec ls -ld "{}" + > "$monitorf"
mail -a "$monitorf" -s "monitor" jgk@yahoo.com; billygm@yahoo.com

If you don't have permission to search and read that file hierarchy, there is nothing we can do for you other than suggest that you get permission to run this script as a user who has access. If the attachment is too big for mail to handle, you might be able to use a different tool to enqueue your mail, but mail transport protocols also have message size limits imposed by servers on your machine, the receiving users' machines, and any server between them that will be used to forward your email messages.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Large volume file formatting

Hi, I have a file which is around 193 gb in size. This file has tonnes of spaces and I need to sanitize it. I tried to use awk script to split this file but it gave me an error like line to long... As of now I am using a sed command to search replace the spaces; however its too slow for such a... (2 Replies)
Discussion started by: darshanw
2 Replies

2. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies

3. Shell Programming and Scripting

select data from oracle table and save the output as csv file

Hi I need to execute a select statement in a solaris environment with oracle database. The select statement returns number of rows of data. I need the data to be inserted into a CSV file with proper format. For that we normally use "You have to select all your columns as one big string,... (2 Replies)
Discussion started by: rdhanek
2 Replies

4. Shell Programming and Scripting

Help Manipulating Large Csv File

Hello everyone, I am trying to manipulate a large .csv file where I have output similar to the following - http://imgur.com/TEXD8.png The result that I am looking for would be to consolidate the first column, but combine the second and third column so it still relates to the first. I... (8 Replies)
Discussion started by: xxwohxx
8 Replies

5. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

6. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

7. Shell Programming and Scripting

Save output of updated csv file as csv file itself, part 2

Hi, I have another problem. I want to sort another csv file by the first field. result.csv SourceFile,Airspeed,GPSLatitude,GPSLongitude,Temperature,Pressure,Altitude,Roll,Pitch,Yaw /home/intannf/foto5/2015_0313_090651_219.JPG,0.,-7.77223,110.37310,30.75,996.46,148.75,180.94,182.00,63.92 ... (2 Replies)
Discussion started by: refrain
2 Replies

8. Shell Programming and Scripting

Copy data to CSV file from txt output

Hi Everyone , Below is output from a TL1 , I want just the NE Name: and beside that the Temperature and the voltages in a csv file , Is this possible? > act-user:AB1S2a:ArshadFO:493::**********; AB1S2a 2016-02-07 10:13:24 M 493 COMPLD "ArshadFO:2016-02-07 10-04-55,0" ;... (11 Replies)
Discussion started by: adgjmpt
11 Replies

9. Shell Programming and Scripting

Need a script to parse data and output to csv

I am not too savvy with arrays and am assuming that what I am looking for needs arrays. This is my requirement. So I have the raw data that gets updated to a log as shown below StudentInfo: FullInfo = { Address = Newark Age = 20 Name= John } StudentInfo:... (2 Replies)
Discussion started by: sidnow
2 Replies

10. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to... (0 Replies)
Discussion started by: as7951
0 Replies
getc_putc(8)						      System Manager's Manual						      getc_putc(8)

NAME
getc_putc - program to test hard drive performance. SYNOPSIS
getc_putc [-d dir] [-s size(KiB)] [-m machine-name] [-u uid-to-use:gid-to-use] [-g gid-to-use] DESCRIPTION
This manual page documents briefly the getc_putc, program. This is a simple adjunct to the bonnie++ benchmark. It is used to test various ways of doing IO one byte at a time, usually you don't need to do enough of this for it to be a performance issue for it to matter much which way you do it. But sometimes it's necessary (for example whan parsing IO from a terminal and then launching another process which will take over all IO, such as a simple shell). The real benefits of this are to help settle some arguements about the performance of such things, and to educate novices about how bad per-byte IO really is. OPTIONS
For getc_putc every option is of the form of a hyphen followed by a letter and then the next parameter contains the value. -d the directory to use for the tests. -s the size of the file for byte IO performance measured in kilobytes. NB You can specify the size in mega-bytes if you add 'm' to the end of the number. The default for this test is to test with a 40MiB file. Of the file only 1/32 of it will be used for write() and read() system calls (anything else takes too long), and only 1/4 of it will be used for locked getc() and putc(). -m name of the machine - for display purposes only. -u user-id to use. When running as root specify the UID to use for the tests. It is not recommended to use root, so if you really want to run as root then use -u root. Also if you want to specify the group to run as then use the user:group format. If you spec- ify a user by name but no group then the primary group of that user will be chosen. If you specify a user by number and no group then the group will be nogroup. -g group-id to use. Same as using :group for the -u parameter, just a different way to specify it for compatibility with other pro- grams. -q quiet mode. If specified then some of the extra informational messages will be suppressed. Also the csv data will be the only output on standard out and the plain text data will be on standard error. This means you can run getc_putc -q >> file.csv to record your csv data. OUTPUT
The primary output is plain-text in 80 columns which is designed to fit well when pasted into email and which will work well with Braille displays. The second type of output is CSV (Comma Seperated Values). This can easily be imported into any spread-sheet or database program. For every test the result is a speed in KiB/s. I do not display the CPU time because it presumably is 99% of the power of a single CPU (or something very close to that). AUTHOR
This program, it's manual page, and the Debian package were written by Russell Coker <russell@coker.com.au>. The documentation, the Perl scripts, and all the code for testing the creation of thousands of files was written by Russell Coker, but the entire package is under joint copyright with Tim Bray. SIGNALS
Handles SIGINT and does a cleanup (which may take some time), a second SIGINT or a SIGQUIT will cause it to immidiately die. SIGXCPU and SIGXFSZ act like SIGINT. Ignores SIGHUP. AVAILABILITY
The source is available from http://www.coker.com.au/bonnie++ . See http://etbe.coker.com.au/category/benchmark for further information. SEE ALSO
bonnie++(8), zcav(8) getc_putc(8)
All times are GMT -4. The time now is 03:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy