Large file data handling issue Post: 302731313

10 More Discussions You Might Find Interesting

1. HP-UX

Need to split a large data file using a Unix script

Greetings all: I am still new to Unix environment and I need help with the following requirement. I have a large sequential file sorted on a field (say store#) that is being split into several smaller files, one for each store. That means if there are 500 stores, there will be 500 files. This...

2. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below...

3. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--...

4. Shell Programming and Scripting

UNIX File handling -Issue in reading a file

I have been doing automation of daily check activity for a server, i have been using sqls to retrive the data and while loop for reading the data from the file for several activities. BUT i got a show stopper the below one.. where the data is getting store in $temp_file, but not being read by while...

5. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ...

6. Red Hat

Advice regarding filesystems handling large number of files

Hi All, I have a CentOS operating system installed. I work with really huge number of files which are not only huge in number but some of them really huge in size. Minimum number of files could be 1 million to 2 million in one directory itself. Some of the files are even several Gigabytes in...

7. Shell Programming and Scripting

UNIX file handling issue

I have a huge file semicolon( ; ) separated records are Pipe(|) delimited. e.g abc;def;ghi|jkl;mno;pqr|123;456;789 I need to replace the 50th field(semicolon separated) of each record with 9006. The 50th field can have no value e.g. ;; Can someone help me with the appropriate command.

8. UNIX for Dummies Questions & Answers

File handling issue

Hi All, I am running into an issue. I have a very big file. Wants to split it in smaller chunks. This file has multiple header/ trailers. Also, between each header/trailer there are records. Number of records in each header trailer combination can vary. Also, headers can start with...

9. Shell Programming and Scripting

Output large volume of data to CSV file

I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message when I run the program. The program is not outputting to the csv file. Error: the file access permissions do not allow the specified action cannot...

10. Shell Programming and Scripting

Large File masking incorrectly happening � delimeter issue

The OS version is Red Hat Enterprise Linux Server release 6.10 I have a script to mask some columns with **** in a data file which is delimeted with � , I am using awk for the masking , when I try to mask a small file the awk works fine and masks the required column , but when the file is...

LEARN ABOUT DEBIAN

rawdog

RAWDOG(1)						      General Commands Manual							 RAWDOG(1)

NAME

       rawdog - an RSS Aggregator Without Delusions Of Grandeur

SYNOPSIS

       rawdog [options]

DESCRIPTION

       This manual page briefly documents the rawdog command.

       rawdog  is  an  RSS (and other) feed aggregator, based on Mark Pilgrim's flexible feed parser. rawdog reads articles from a number of feeds
       and writes out a single HTML file, based on a template either provided by the user or generated by rawdog, containing the  latest  articles
       it's seen.

OPTIONS

       This  program follows the usual GNU command line syntax, with long options starting with two dashes (`-'). A summary of options is included
       below. For a complete description, see the README file.

       -u, --update
	      Fetch data from the feeds and store it. This could take some time if you've got lots of feeds.

       -w, --write
	      Write out the HTML output file.

       -f SOMEURL, --update-feed SOMEURL
	      Update the feed pointed to by SOMEURL immediately, even if its period hasn't elapsed since it was last updated.

       -c FILE, --config FILE
	      Read FILE as an additional config file; any options provided in FILE will override those set in  the  main  config  file	(with  the
	      exception  of  "feed",  which is cumulative).  FILE may be an absolute path or a path relative to your .rawdog directory.  Note that
	      $HOME/.rawdog/config will still be read first even if you specify this option.

       -t,  --show-template
	      Print the template currently in use to stdout.

       -a URL, --add URL
	      Add a new feed to the config file.

       -d DIR, --dir DIR
	      Use DIR instead of the $HOME/.rawdog directory.

       --help Provide a brief summary of all the options rawdog supports, and exit.

FILES

       $HOME/.rawdog/config

SEE ALSO

       cron(1).

AUTHOR

       rawdog was written by Adam Sampson <azz@us-lot.org>.

       This manual page was written by Decklin Foster <decklin@red-bean.com>, for the Debian project (but may be used by others).

																	 RAWDOG(1)