Sponsored Content
Top Forums Shell Programming and Scripting File processing is very slow with cut command Post 302389881 by bilalghazi on Tuesday 26th of January 2010 09:59:05 AM
Old 01-26-2010
Data File processing is very slow with cut command

Dear All,

I am using the following script to find and replace the date format in a file. The field18 in the file has the following format: "01/26/2010 11:55:14 GMT+04:00" which I want to convert into the following format "20100126115514" for this purpose I am using the following lines of codes:
Code:
while [ 1 ]
do
  read fileLine || break
  newDate=`echo $fileLine | cut -f18 -d "|" | cut -c1-19`
  month=`echo $newDate | cut -c1-2`
  day=`echo $newDate | cut -c4-5`
  year=`echo $newDate | cut -c7-10`
  hour=`echo $newDate | cut -c12-13`
  min=`echo $newDate | cut -c15-16`
  sec=`echo $newDate | cut -c18-19`
  new_Format=`echo "$year$month$day$hour$min$sec"`
  `echo $fileLine | sed -e "s/$month\\/$day\\/$year $hour:$min:$sec GMT+04:00/$new_Format/"  >> $newFileName`
  let recordCount=$recordCount+1
  move=1
done < $fileName

The main problem is that this way of conversion taking too much time, from logs:

Record Count : 6147 Start at Tue Jan 26 18:43:23 GST 2010 End Time: Tue Jan 26 18:51:04 GST 2010

which means that almost 8 minutes for only 6147 records which is not acceptable.

Is there any way of increasing the speed or using any other solution. Any suggestion is appreciated. Thanks

Regards,

Last edited by Scott; 01-26-2010 at 11:07 AM.. Reason: Please use code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Modified dates to a file without the cut command

how can i write the modified dates of all of the files in my directory to a file. i dont want any of the other junk from ls in there. i cant use the cut command (4 Replies)
Discussion started by: cypher
4 Replies

2. Shell Programming and Scripting

Cut too slow

Hi I am using a cut command in the script which is slowing down the performance of the script .can anyone suggest the other ways of doing the same cut command Record_Type=`echo "$line" | cut -c19-20` *******this is slowing down********* i have 4 more cut commands 2 in one loop and 2 in inner... (3 Replies)
Discussion started by: pukars4u
3 Replies

3. SCO

Slow Processing - not matching hardware capabilities

I have been a SCO UNIX user, never an administrator...so I am stumbling around looking for information. I don't know too much about what is onboard in terms of hardware, however; I will try my best. We have SCO 5.07 and have applied MP5. We have a quad core processor with 4 250 GB... (1 Reply)
Discussion started by: atpbrownie
1 Replies

4. Shell Programming and Scripting

Using cut command in a fixed length file

Hi, I have a file which have set of rows and has to create separate files based on the id. Eg: 001_AHaris020 001_ATony030 002_AChris090 002_ASmit060 003_AJhon001 Output: I want three files like 001_A.txt, 002_A.txt and 003_A.txt. 001_A.txt should have ... (4 Replies)
Discussion started by: techmoris
4 Replies

5. Shell Programming and Scripting

cut, sed, awk too slow to retrieve line - other options?

Hi, I have a script that, basically, has two input files of this type: file1 key1=value1_1_1 key2=value1_2_1 key4=value1_4_1 ... file2 key2=value2_2_1 key2=value2_2_2 key3=value2_3_1 key4=value2_4_1 ... My files are 10k lines big each (approx). The keys are strings that don't... (7 Replies)
Discussion started by: fzd
7 Replies

6. Shell Programming and Scripting

Cut Command error cut: Bad range

Hi Can anyone what I am doing wrong while using cut command. for f in *.log do logfilename=$f Log "Log file Name: $logfilename" logfile1=`basename $logfilename .log` flength=${#logfile1} Log "file length $flength" from_length=$(($flength - 15)) Log "from... (2 Replies)
Discussion started by: dgmm
2 Replies

7. Shell Programming and Scripting

Help with File Slow Processing

Hello, Hope you are doing fine. Let me describe the problem, I have a script that calls another script K2Test.sh, this script K2Test.sh (created by another team) takes date as argument and generates approx 1365 files in localcurves directory for given date. Out of these 1365 I am only... (11 Replies)
Discussion started by: srattani
11 Replies

8. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

9. UNIX for Beginners Questions & Answers

Cut command: can't make it cut fields

I'm a complete beginner in UNIX (and not a computer science student either), just undergoing a tutoring course. Trying to replicate the instructions on my own I directed output of the ls listing command (lists all files of my home directory ) to My_dir.tsv file (see the screenshot) to make use of... (9 Replies)
Discussion started by: scrutinizerix
9 Replies

10. Shell Programming and Scripting

Processing too slow with loop

I have 2 files file 1 : contains ALINE ALINE BANG B ON A B.B.V.A. BANG AMER CORG BANG ON MORENA BANG ON MORENAIC BANG ON MORENAICA BANG ON MORENAICA CORP BANG ON MORENAICA N.A file 2 contains and is seprated by ^ delimiter : NATIO MARKET^345432534 (10 Replies)
Discussion started by: nikhil jain
10 Replies
PX_TIMESTAMP2STRING(3)					     Library Functions Manual					    PX_TIMESTAMP2STRING(3)

NAME
PX_timestamp2string -- Convert paradox timestamp into formated string SYNOPSIS
#include <paradox.h> char * PX_timestamp2string(pxdoc_t *pxdoc, double timestamp, const char *format) DESCRIPTION
Creates a string representation of a paradox timestamp as specified in the parameter format. The timestamp must be either retrieved with PX_get_data_double(3) after calling PX_get_record(3) or by simply using the double value in the pxval_t struct as returned by PX_retrieve_record(3). A timestamp contains both time and date information. The format string contains chars which stand for certain values of the date/time. The following table lists all available characters with a meaning. All other characters are being copied unmodified to the output string. If a special character shall not be interpreted it must be preceded with a backslash. Characters and their meaning in date/time format string Character Meaning Y year, numeric, 4 digits y year, numeric, 2 digits m month, numeric n month, numeric, no leading zeroes d day of the month, numeric j T{ day of the month, numeric, no leading zeros T} H hour, numeric, 24 hour format h hour, numeric, 12 hour format G T{ hour, numeric, 24 hour format, no leading zeroes T} g T{ hour, numeric, 12 hour format, no leading zeroes T} i minutes, numeric s seconds, numeric A AM/PM a am/pm L boolean for leap year RETURN VALUE
Returns pointer to string on success and NULL on failure. SEE ALSO
PX_time2string(3), PX_date2string(3) AUTHOR
This manual page was written by Uwe Steinmann uwe@steinmann.cx. PX_TIMESTAMP2STRING(3)
All times are GMT -4. The time now is 03:43 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy