Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with Creation of Script to Input Separators in Data

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Creation of Script to Input Separators in Data
# 1  
Old 06-08-2009
Help with Creation of Script to Input Separators in Data
Hi all,
I have one problem that is preparing datas so I can run a script to extrat informations for my statistic reports.
I receive some datas, that are informations mixed and I need to separate them to analyse. This is an exemple of datas:
#header: 0MEPS000006000000051620090508001200905070010001613200020090508001
#body: 20421730007384620090507144018000001000000000A000000020283139Somewhere 00000001600
#body: 20421730007414520090507144237000001000000000A000000020283148Somewhere 00000001700
#body: .
#body: .
#body: .
#body: .
#body: .
#body: .
#tail: 9000000020000000000020000000000000000000000000000

Note that this #body, header does not include in this data.

So I receive them like this, but I need them to be like this:
0,MEPS,00000600,00000516,20090526001,20090525001,00016,132,000,20090526001,
2,04,2191,00112407,20090525233138,0000200000,00000,A,0000000502,98032,Somewhere ,020000008,0,0,
2,04,2191,00008894,20090525075812,0000010000,00000,A,0000001301,55245,Somewhere ,100000147,0,0,
.
.
.
.
9,00000059,0000000001790000,000000000000,000000000000,

So in this manner I can run my script:

#!/bin/bash
echo "
serial_nb amount date time ATM_nb location"
awk -F, '$1==2 {print$13,$7,$5,$6,$10,$12}' file_name

which produce me the results I want:

serial_nb amount date time ATM_nb location
000000016 0000010000 20090507 144018 0000000202 Somewhere
000000017 0000010000 20090507 144237 0000000202 Somewhere

So, right now I prepare my datas with the aid of excell, but I rather use an awk command to to this.
I need a command that says: input "," after 2 characters, then input another one after 5, etc etc.
Bottom line, a command that inputs separators acording to lenght.

Thanks for all your help.
# 2  
Old 06-08-2009
This command should place comma's after 2 then after 5,7 and 10 positions, so you should get columns of 2, 5, 7, .. positions:

Code:
awk -v p="2,5,7,10" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos[i]),",")
    b+=pos[i]
  }
  print substr($0,b)
}' file

# 3  
Old 06-09-2009
Quote:
Originally Posted by Franklin52
This command should place comma's after 2 then after 5,7 and 10 positions, so you should get columns of 2, 5, 7, .. positions:

Code:
awk -v p="2,5,7,10" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos[i]),",")
    b+=pos[i]
  }
  print substr($0,b)
}' file

Thanks Franklin for the solution, but while this brings me closer to solving everything to me, there are still some lose ends.
This code you provided me with, only applies to the first line, and it doesn´t stop where I want it.
So I made some changes, but i still have this problem with first line:
The code changed by me:

Code:
#!/bin/bash
#script to separate the fields in MEPS file
for x in `cat $1`
do awk '$1~/^2/ {print}' $1 |awk -v p="1,2,4,8,8,6,10,5,1,10,5,15,9,1,1" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos[i]),",")
    b+=pos[i]
  }
  print substr($0,b)
}'
done



This produces the following result:
Code:
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,


So you see, that it repeats the first line and it produces these series of ,,, ,,, ,,, ,,,,, that I do not want.
I tried to modified the code many times, but I think the problem is that I did not understant all of the code.
So what I´m asking is for you to give me some pointer on how to make this work and some explanation on the original code you provided me.

Thanks in advance.
Last edited by Alexis Duarte; 06-09-2009 at 04:28 PM.. Reason: Forgot the code tags. Sorry about that
# 4  
Old 06-09-2009
Post the input file of the awk command and the desired output within code tags (select the text and press the # button above the edit box).
# 5  
Old 06-09-2009
Corrected the post. Can you check the post again please.
Thanks you
# 6  
Old 06-10-2009
It should be easier if you post a part of the input file of the awk command and the desired output.

Regards
# 7  
Old 06-10-2009
Check please
Quote:
Originally Posted by Franklin52
It should be easier if you post a part of the input file of the awk command and the desired output.

Regards
Hi there,
I´m not sure I understand exactly what you need, but let me give you exactly what I want:
This is my input file:
Code:
20421910011240720090525233138000020000000000A000000050298032Boavista       02000000800
20421910000889420090525075812000001000000000A000000130155245Praia          10000014700
20421910002863720090525103836000001000000000A000000070128094Sal            10000014800
20421910003524020090525112959000001000000000A000000010746168Praia          10000014900
20421910003616520090525113544000001000000000A000000020185311Sao Vicente    10000015000
20421910003807620090525114955000001000000000A000000130155277Praia          10000015100

And this is the output I need:
Code:
2,04,2191,00112407,20090525233138,0000200000,00000,A,0000000502,98032,Boavista       ,020000008,0,0
2,04,2191,00008894,20090525075812,0000010000,00000,A,0000001301,55245,Praia          ,100000147,0,0
2,04,2191,00028637,20090525103836,0000010000,00000,A,0000000701,28094,Sal            ,100000148,0,0
2,04,2191,00035240,20090525112959,0000010000,00000,A,0000000107,46168,Praia          ,100000149,0,0
2,04,2191,00036165,20090525113544,0000010000,00000,A,0000000201,85311,Sao Vicente    ,100000150,0,0
2,04,2191,00038076,20090525114955,0000010000,00000,A,0000001301,55277,Praia          ,100000151,0,0
2,04,2191,00078149,20090525171505,0000010000,00000,A,0000000401,63329,Santa Catarina ,100000152,0,0

But with the code you provide me (with some changes performed, but the same basic codes):
Code:
#!/bin/bash
#script to separate the fields in MEPS file
for x in `cat $1`
do awk '$1~/^2/ {print}' $1 |awk -v p="1,2,4,8,8,6,10,5,1,10,5,15,9,1,1" '
BEGIN { n=split(p,pos,",");b=1 }
{ for(i=1;i<=n;i++) {
    printf("%s%c",substr($0,b,pos[i]),",")
    b+=pos[i]
  }
  print substr($0,b)
}'
done

I receive insted this output:
Code:
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,
2,04,2173,00073846,20090507,144018,0000010000,00000,A,0000000202,83139,Praia          ,000000016,0,0,
 ,  ,    ,     ,,,,,,,,,,,,


So you see, it only applies to the first input line and reapets it self over and over again.

Hope this was clear for you, and if you could help me I would be very grateful.


Best Regards


P.S. this last output (the one that reapets itself) is not from the original file, I present here, but you get the point.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing out data with multiple field separators

I have a large file that I need to print certain sections out of. file.txt /alpha/beta/delta/gamma/425/590/USC00015420.blah.lt.0.01.str:USC00015420Y2017M10BLALT.01 12 13 14 -9 1 -9 -9 -9 -9 -9 1 2 3 4 5 -9 -9 I need to print the "USC00015420" and... (5 Replies)
Discussion started by: ncwxpanther
5 Replies

2. Shell Programming and Scripting

How to get the shell script to read the .txt file as an input/data?

i have written my shell script in notepad however i am struggling to pass the data file to be read to the script the data file is of .txt format. My target is to run the shell script from the terminal and pass 3 arguments e.g. polg@DESKTOP-BVPDC5C:~/CS1420/coursework$ bash valsplit.sh input.txt... (11 Replies)
Discussion started by: Gurdza32
11 Replies

3. Shell Programming and Scripting

Standardization of input source data files using shell script

Hi there, I'm a newbie in unix and am fishing for options related to how raw input data files are handled. The scenario, as I'm sure y'all must be very familiar with, is this : we receive upwards of 50 data files in ASCII format from various source systems - now each file has its own structure... (3 Replies)
Discussion started by: Prat Khos
3 Replies

4. Shell Programming and Scripting

Script asks to input data

Hi, I have three different files about a warehouse's stock status. Each file shows storage locations, stored product names, quantity of the part and at last column, its price. When there is a change in price, I open those files one by one, search related product name at each row and change... (4 Replies)
Discussion started by: baris35
4 Replies

5. Shell Programming and Scripting

adding data in input file if 2nd script output SUCCESS

Hi All, how can i edit my original data and add more data if my 2nd script outputs SUCESS? ex. input file: 1.txt nik,is,the 1script.sh if 2ndscript.sh output SUCCESS then i'm going to edit my input file and add data best,pogi.. sample outputdata. nik,is,the,best,pogi 2ndscript.sh... (3 Replies)
Discussion started by: nikki1200
3 Replies

6. Shell Programming and Scripting

Echo date variable from data input to a script

Hi, I'm trying to make a script which you type the year, select the month and day and then create the date in the format 2010-12-7. #!/bin/bash dia () { echo " Seleccione el dia:" select file in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Salir do... (6 Replies)
Discussion started by: iga3725
6 Replies

7. Shell Programming and Scripting

Script creation (Input needed)

Hi , I m trying to write a script in linux . The problem is the output of command get changed every now and then. Just like top command in linux. How would i manipulate the output of the command ? The command i m talking about gives the real time values of performance of hosted guest... (3 Replies)
Discussion started by: pinga123
3 Replies

8. Shell Programming and Scripting

Need script to take input from file, match on it in file 2 and input data

All, I am trying to figure out a script to run in windows that will allow me to match on First column in file1 to 8th Column in File2 then Insert file1 column2 to file2 column4 then create a new file. File1: 12345 Sam 12346 Bob 12347 Bill File2:... (1 Reply)
Discussion started by: darkoth
1 Replies

9. Shell Programming and Scripting

Multiple input field Separators in awk.

I saw a couple of posts here referencing how to handle more than one input field separator in awk. I figured I would share how I (just!) figured out how to turn this line in a logfile: 90000000000000000000010001 name... (4 Replies)
Discussion started by: kinksville
4 Replies

10. Shell Programming and Scripting

Creation of script,if the data file have more than one entry!!!

1.Daily there will be 14 files in the data directory 2.someday's the 14 files receive more than once r twice with different time stamps....we need to chk the count of the file and if the count of the file is two.we need to combine the both the files. 3. if any duplicate data is there just... (1 Reply)
Discussion started by: bobprabhu
1 Replies
Login or Register to Ask a Question