|
|
Sponsored Content
Top Forums
UNIX for Dummies Questions & Answers
Speeding up a Shell Script (find, grep and a for loop)
Post 302223668 by Dave Stockdale on Monday 11th of August 2008 04:36:02 AM
08-11-2008
Ok, thanks.
I've made these changes. It still takes a little while to complete, but this is due to the number of PDFs that aren't linked to more than anything else.
After the first time this is run for real, and all of the PDFs not linked to are archived, the process will be much quicker. I'm probably going to add it to a cron or something so that it runs once a week automatically, and I won't have to worry about unused files wasting disk space
Thanks for all your help!
I've made these changes. It still takes a little while to complete, but this is due to the number of PDFs that aren't linked to more than anything else.
After the first time this is run for real, and all of the PDFs not linked to are archived, the process will be much quicker. I'm probably going to add it to a cron or something so that it runs once a week automatically, and I won't have to worry about unused files wasting disk space
Thanks for all your help!
|
|
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a file that is 20 - 80+ MB in size that is a certain type of log file.
It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example:
The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies
2. Shell Programming and Scripting
I'm trying to make a simple search script but cannot get it right. The script should search for keywords inside files. Then return the file paths in a variable. (Each file path separated with \n).
#!/bin/bash
SEARCHQUERY="searchword1 searchword2 searchword3";
for WORD in $SEARCHQUERY
do
... (6 Replies)
Discussion started by: limmer
6 Replies
3. Shell Programming and Scripting
Hi,
I have a text file with data in that I wish to extract, assign to a variable and process through a loop.
Kind of the process that I am after:
1: Grep the text file for the values.
Currently using:
cat /root/test.txt | grep TESTING= | awk -F"=" '{ a = $2 } {print a}' | sort -u
... (0 Replies)
Discussion started by: Spoonless
0 Replies
4. UNIX for Dummies Questions & Answers
Hi all,
I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min).
Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search.
Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies
5. Shell Programming and Scripting
Hello,
I am using sed in a for loop to replace text in a 100MB file. I have about 55,000 entries to convert in a csv file with two entries per line. The following script works to search file.txt for the first field from conversion.csv and then replace it with the second field. While it works fine,... (15 Replies)
Discussion started by: pbluescript
15 Replies
6. Shell Programming and Scripting
This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up?
cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies
7. Shell Programming and Scripting
Dear all,
Please help with the following.
I have a file, let's call it data.txt, that has 3 columns and approx 700,000 lines, and looks like this:
rs1234 A C
rs1236 T G
rs2345 G T
Please use code tags as required by forum rules!
I have a second file, called reference.txt,... (1 Reply)
Discussion started by: aberg
1 Replies
8. Shell Programming and Scripting
HI Guys hoping some one can help
I have two files on both containing uk phone numbers
master is a file which has been collated over a few years ad currently contains around 4 million numbers
new is a file which also contains 4 million number i need to split new nto two separate files... (4 Replies)
Discussion started by: dunryc
4 Replies
9. Shell Programming and Scripting
Hi,
I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Discussion started by: newbie_01
9 Replies
10. Shell Programming and Scripting
Hello experts,
we have input files with 700K lines each (one generated for every hour). and we need to convert them as below and move them to another directory once.
Sample INPUT:-
# cat test1
1559205600000,8474,NormalizedPortInfo,PctDiscards,0.0,Interface,BG-CTA-AX1.test.com,Vl111... (7 Replies)
Discussion started by: prvnrk
7 Replies
LEARN ABOUT CENTOS
groff_mom
GROFF_MOM(7) Miscellaneous Information Manual GROFF_MOM(7) NAME
groff_mom - groff `mom' macros SYNOPSIS
pdfmom [-Tps [pdfroff options]] [groff options] files ... groff [-mom] files ... groff [-m mom] files ... DESCRIPTION
mom is a macro set for groff, designed primarily to format documents for PDF and PostScript output. mom provides two categories of macros: macros for typesetting, and macros for document processing. The typesetting macros provide access to groff's typesetting capabilities in ways that are simpler to master than groff's primitives. The document processing macros provide highly customizable markup tags that allow the user to design and output professional-looking documents with a minimum of typesetting intervention. Files processed with pdfmom (man(1) pdfmom), with or without the -Tps option, produce PDF documents. The documents include a PDF outline that appears in the 'Contents' panel of document viewers, and may contain clickable internal and external links. When -Tps is absent, groff's native PDF driver, gropdf, is used to generate the output. When given, the output is still PDF, but process- ing is passed over to pdfroff, which uses groff's PostScript driver, grops. Not all PDF features are available when -Tps is given; its primary use is to allow processing of files with embedded PostScript images. Files processed with groff -mom (or -m mom ) produce PostScript output by default. mom comes with her own very complete documentation in HTML format. A separate PDF manual, Producing PDFs with groff and mom, covers full mom/PDF usage. FILES
om.tmac - the main macro file mom.tmac - a wrapper file that calls om.tmac directly. </usr/share/doc/groff-1.22.2/html/mom/toc.html> - entry point to the HTML documentation /usr/share/doc/groff-1.22.2/pdf/mom-pdf.pdf - the PDF manual, Producing PDFs with groff and mom /usr/share/doc/groff-1.22.2/examples/mom/*.mom - example files using mom AUTHOR
mom was written by Peter Schaffter <peter@schaffter.ca>. PDF support was provided by Deri James <deri@chuzzlewit.demon.co.uk>. Please send bug reports to the groff-bug mailing list <bug-groff@gnu.org> or directly to the authors. Groff Version 1.22.2 7 February 2013 GROFF_MOM(7)