Shell script to extract data in a file

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Shell script to extract data in a file
# 29  
Old 02-16-2017
I'd be very surprised if it really took 6h, except on a very busy host. For a 15MB file with 1101925 lines:
Code:
time sh ou.awk 
real    0m4.469s
user    0m4.400s
sys     0m0.040s

EDIT:
For a comparison with vgersh99's proposal in post#5:
Code:
time awk -f gil.awk DS2_export100217.ldif
real    0m1.713s
user    0m1.076s
sys     0m0.072s

And, last but not least, Corona688's proposal in post#9
Code:
time awk -f mline.awk DS2_export100217.ldif
real    0m0.833s
user    0m0.752s
sys     0m0.072s

Both use ($1 in ARRAY) which is WAY faster than my HDCM ~ "," $1 "," ; a good learning, isn't it?

EDIT2:
Applying our learning:
Code:
time ./ou.awk >output                    
real    0m0.678s
user    0m0.660s
sys     0m0.016s


BTW - these timings are far from being scientifically correct, reliable nor objective - other influences like file buffering etc. might heavily modify the results. They're for getting a feeling only...

Last edited by RudiC; 02-16-2017 at 04:05 PM..
These 2 Users Gave Thanks to RudiC For This Post:
# 30  
Old 02-18-2017
@ RudiC it's really taking such time 4hrs at least. I really have to improve the time. Maybe it's my machine that's also running other stuff. But all the same the script is working perfectly.

@vgersh99 i tried the proposal on post 5, it gave me an error. I'm trying again with a file of the same size, and let you know of the timing and error if there is.

---------- Post updated at 04:23 AM ---------- Previous update was at 04:11 AM ----------

Please guys could you explain to me briefly each line of your code?
# 31  
Old 02-18-2017
Try this "optimized" version:
Code:
awk -F: '
BEGIN           {HD="MSISDN,IMSI,NAM,TS11,TS21,TS22,TS62,BAIC,BAOC,APNID1,APNID2"       # define the fields to extract, and prepare a header for eventual print
                 for (i=HDCnt=split(HD, HDArr, ","); i>0; i--)  X[HDArr[i]]             # create am "associative" array  for later $1 matching 
                 NXTREC = "dn"                                                          # this is where the next record starts
                }


$1 == NXTREC    {if (PR) for (i=1; i<=HDCnt; i++) printf "%s%s", RES[HDArr[i]], i==HDCnt?ORS:OFS
                                                                                        # print old record collected up to now; don't on the first record (Pr still undefined)
                 split ("", RES)                                                        # remoce RES array
                 PR = 1                                                                 # define the PR logical var so all next records are printed
                }

$1 in X         {RES [$1] = $0                                                          # collect data line if $1 is among the requested fields
                }

END             {for (i=1; i<=HDCnt; i++) printf "%s%s", RES[HDArr[i]], i==HDCnt?ORS:OFS
                }                                                                       # see above for the last record

' FS=":" OFS="," DS2_export100217.ldif

and report back the timing...

Last edited by RudiC; 02-18-2017 at 05:36 AM..
This User Gave Thanks to RudiC For This Post:
# 32  
Old 02-19-2017
It's running but the output file is not receiving the output expected. It's taking more time to load the output file.

---------- Post updated 02-19-17 at 07:07 AM ---------- Previous update was 02-18-17 at 11:52 AM ----------

Please another script to extrat the information highlighted in yellow. Can you guys help? i tried with the other script, it's not giving the results. the results should be on the same line. Please!!


Code:
dn: EpsStaInfId=EpsStaInf,serv=EPS,mscId=aaaaaa001aaaaaaaa629100100138702,ou=multiSCs,dc=mtncg
structuralObjectClass: EpsStaticInf
objectClass: EpsStaticInf
entryDS: 1
nodeId: 1
createTimestamp: 20170119123400Z
modifyTimestamp: 20170119123400Z
EpsStaInfId: EpsStaInf
EpsProfileId: 29
EpsOdb: 0
EpsRoamAllow: TRUE
CDC: 1
EpsIndSubChargChars: 123

dn: EpsStaInfId=EpsStaInf,serv=EPS,mscId=aaaaaa001aaaaaaaa629100100165619,ou=multiSCs,dc=mtncg
structuralObjectClass: EpsStaticInf
objectClass: EpsStaticInf
entryDS: 1
nodeId: 1
createTimestamp: 20170211115449Z
modifyTimestamp: 20170211115449Z
EpsStaInfId: EpsStaInf
EpsProfileId: 29
EpsOdb: 0
EpsRoamAllow: TRUE
CDC: 1
EpsIndSubChargChars: 123

# 33  
Old 02-19-2017
What did you try, esp. in the light of the proposals made for your first problem? Show your attempts including error messages.
Don't miss to post the expected output. Does it go to the same result file as the other fields?

If not: for a new request create a new thread.
# 34  
Old 02-19-2017
@RudiC, i tried with the script, it didn't give any results at all. I lunched it, but the output file was empty. So i'm wondering whether the output was still in the buffer or something else..I'm creating a new thread for the new problem right away.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to extract data from csv file

Hi everyone, I have a csv file which has data with different heading and column names as below. Static Data Ingested ,,,,,,,,,,,,Known Explained Rejections Column_1,column_2,Column_3,Column_4,,Column_6,Column_7,,% Column_8,,Column_9 ,Column_10 ,... (14 Replies)
Discussion started by: Vivekit82
14 Replies

2. UNIX for Dummies Questions & Answers

Shell script to extract data from csv file

Hi Guys, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 7 columns having values say column 1,column 2.....column 7 as below along with their values. Name, Address,... (7 Replies)
Discussion started by: Vivekit82
7 Replies

3. UNIX for Dummies Questions & Answers

Shell script to extract data from csv file based on certain conditions

Hi Guys, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 5 columns having values say column 1,column 2.....column 5 as below along with their valuesm.... (1 Reply)
Discussion started by: Vivekit82
1 Replies

4. Shell Programming and Scripting

regular expression with shell script to extract data out of a text file

hi i am trying to extract some specific data out of a text file using regular expressions with shell script that is using a multiline grep .. and the tool i am using is pcregrep so that i can get compatibility with perl's regular expressions for a sample data like this, i am trying to grab... (6 Replies)
Discussion started by: vemkiran
6 Replies

5. Shell Programming and Scripting

need a shell script to extract data from a log file.

If I have a log like : Mon Jul 19 05:07:34 2010; TCP; eth3; 52 bytes; from abc to def Mon Jul 19 05:07:35 2010; UDP; eth3; 46 bytes; from aaa to bbb Mon Jul 19 05:07:35 2010; TCP; eth3; 52 bytes; from def to ghi I will need an output like this : Time abc to def... (1 Reply)
Discussion started by: hitha87
1 Replies

6. Shell Programming and Scripting

Need shell script to extract data from oracle database

shell script (4 Replies)
Discussion started by: frns5
4 Replies

7. Shell Programming and Scripting

Help with shell script to extract data from XML file

Hello Scripting Gurus, I need help with extracting data from the XML file using shell script. The data is in a large XML and I need to extract the id values of all completedworkflows. Here is a sample of it. Input and output data is also in the attached text files. <wfregistry>... (5 Replies)
Discussion started by: yajaykumar
5 Replies

8. Shell Programming and Scripting

shell-script which extract data from log file

give me a shell-script which extract data from log file on a server by giving date and time as input (for both start time and end time) and it will give the logs generated during the given time as output. (4 Replies)
Discussion started by: abhishek27
4 Replies

9. Shell Programming and Scripting

extract data from xml- shell script using awk

Hi, This is the xml file that i have. - <front-servlet platform="WAS4.0" request-retriever="SiteMinder-aware" configuration-rescan-interval="60000"> <concurrency-throttle maximum-concurrency="50" redirect-page="/jsp/defaulterror.jsp" /> - <loggers> <instrumentation... (5 Replies)
Discussion started by: nishana
5 Replies

10. Shell Programming and Scripting

How to extract data using UNIX shell script?

Hello All, I am starting with UNIX. Any help is highly appreciated. How to extract data using UNIX shell script? And how do you export data using UNIX shell scripts into Microsoft Excel format? Thank you. (3 Replies)
Discussion started by: desiondarun
3 Replies
Login or Register to Ask a Question