Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Repeating Multiple Fields

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Repeating Multiple Fields
# 1  
Old 08-21-2014
Repeating Multiple Fields
I am trying to find a way to repeat fields. I am not really sure how to explain it so let me just post a sample and what I want it to look like.

Code:
888123 66232 18 1
                      19 
            44422 11 7
                      23 
881133 66231 33 1
                      34 
983333 11112 11 1
                      34 
            13331 33 11
                      88
                      99 16
                      77
                      11
            33311 33 33

While the sample here is all the same length, the actual values will vary and contain periods, but that shouldn't matter. It will all be separated by spaces but I can put commas or pipes in between if that matters. Here is what the final product would look like.

Code:
1,888123,6623218,1
2,888123,6623219,1
3,888123,4442211,7
4,888123,4442223,7
5,881133,6623133,1
6,881133,6623134,1
7,983333,1111211,1
8,983333,1111234,1
9,983333,1333133,11
10,983333,1333188,11
11,983333,1333199,16
12,983333,1333177,16
13,983333,1333111,16
14,983333,3331133,33

Fields 1, 2 and 4 will change all over the place. So Field 1 should repeat until it is no longer blank (IE it changes), Field 2 repeat until it changes, Field 4 repeat until it changes. Field 3 will always be populated.

Final output would be numbered, then a comma, then Field 1, then a comma, Field2&3 together, then comma and then finally field 4.

I hope I explained that well enough and appreciate any help. The file is about half a million lines.

Thanks

Trying to figure out a way to do it, I think there has to be a separator besides spaces, so the file will look as follows:

Code:
888123|66232|18|1
           |         |19| 
           |44422|11|7
           |         |23|
881133|66231|33|1
           |         |34|
983333|11112|11|1
           |         |34|
           |13331|33|11
           |         |88|
           |         |99|16
           |         |77|
           |         |11|
           |33311|33|33

I can't think of a way of even trying it without a separator, so I recreated the original file like that. Hopefully that makes it easier. Also when I submit this it is moving the text over a bit, but it will all be left justified.
Last edited by DerangedNick; 08-21-2014 at 03:30 PM.. Reason: Notes - Thoughts
# 2  
Old 08-21-2014
Either used delimiters where each line has the same number of delimiters (even if some fields are empty), or use fixed width columms for each field.
# 3  
Old 08-21-2014
All columns are the same width, it is just getting messed up in the code. So each of them are Field 1 | Field 2 | Field 3 | Field 4 even if they are blank the pipes get put in and they will always be separated in the same place, just figured the pipes would give something to delimit on for the blank fields.
# 4  
Old 08-21-2014
So if they are fixed width columns, no delimiters and you know the number of max fields possible in a like you can do something like...
Code:
cat <file-in-question> | while read a
do
field1=`echo ${a} | cut -c1-10`
field2=`echo ${a} | cut c11-15`
.
.
.
done

just substitute the appriopriate column numbers
you will also want to trim leading and trailing spaces
so once you parse each line for the field values, the next step will be to write the new records, so you will have to have a section that checks to see if the parsed fied value is blank and if so you will have to have a mechanism for saving the previous value
# 5  
Old 08-22-2014
If your input file has fixed width fields, like
Code:
888123 66232 18 1
             19
       44422 11 7
             23
881133 66231 33 1
             34
983333 11112 11 1
             34
       13331 33 11
             88
             99 16
             77
             11
       33311 33 33

, and unlike your sample above, try
Code:
cut  -c1-6,8-12,14-15,17-19 --output-delimiter="|" file |
   while IFS=\| read F1 F2 F3 F4
         do [ $F1 ] && P1=$F1
            [ $F2 ] && P2=$F2
            [ $F4 ] && P4=$F4
            echo $((++CNT)),$P1,$P2$F3,$P4
         done
1,888123,6623218,1
2,888123,6623219,1
3,888123,4442211,7
4,888123,4442223,7
5,881133,6623133,1
6,881133,6623134,1
7,983333,1111211,1
8,983333,1111234,1
9,983333,1333133,11
10,983333,1333188,11
11,983333,1333199,16
12,983333,1333177,16
13,983333,1333111,16
14,983333,3331133,33

If the input is delimited by "|", just drop the cut command in front.

---------- Post updated at 11:26 ---------- Previous update was at 11:09 ----------

Try this with your unregular file format:
Code:
while read F1 F2 F3 F4
        do [ $F4 ] && { P1=$F1; P2=$F2; P3=$F3; P4=$F4; }
           [ $F3 ] && [ ! $F4 ] && { P2=$F1; P3=$F2; P4=$F3; }
           [ $F2 ] && [ ! $F3 ] && { P3=$F1; P4=$F2; }
           [ $F1 ] && [ ! $F2 ] && { P3=$F1; }
           echo $((++CNT)),$P1,$P2$P3,$P4
        done < file

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting fields from a file having multiple delimiters

Hi All, I have a file with a single row having the following text ABC.ABC.ABC,Database,New123,DBNAME,F,ABC.ABC.ABC_APP,"@FUNCTION1("ENT1") ,@FUNCTION2("ENT2")",R, I want an output in the following format ABC.ABC.ABC DBNAME ABC.ABC.ABC_APP '@FUNCTION1("ENT1")... (3 Replies)
Discussion started by: dev.devil.1983
3 Replies

2. Shell Programming and Scripting

Deriving unique entries from multiple repeating patterns

Dear All, I have a below one column data.(example) Col1 1 2 . . 25 8 9 25 1 2 . . 25 Where each entry(row) is a number from 1-25, but in place whereever mentioned with . we have all the entries 1-25, but some places where ever no . like in 8 9 25 I have only 3 entries. No I... (14 Replies)
Discussion started by: ks_reddy
14 Replies

3. Shell Programming and Scripting

Sed replace using same pattern repeating multiple times in a line

Sed replace using same pattern repeating multiple times in a line I have text like below in a file: I am trying to replace the above line to following How can I acheive this? I am able to do it if the occurrence is for 1 time: But If I try like below I am getting like this: I have to... (4 Replies)
Discussion started by: sol_nov
4 Replies

4. Shell Programming and Scripting

Matching multiple fields from two files and then some?

Hi, I am working with two tab-delimited files with multiple columns, formatted as follows: File 1: >chrom 1 100 A G 20 …(10 columns) >chrom 1 104 G C 18 …(10 columns) >chrom 2 28 T C ... (4 Replies)
Discussion started by: mbp
4 Replies

5. UNIX for Dummies Questions & Answers

Using sed command to remove multiple instances of repeating headers in one file?

Hi, I have catenated multiple output files (from a monte carlo run) into one big output file. Each individual file has it's own two line header. So when I catenate, there are multiple two line headers (of the same wording) within the big file. How do I use the sed command to search for the... (1 Reply)
Discussion started by: rebazon
1 Replies

6. UNIX for Dummies Questions & Answers

Formatting Multiple fields on 1 line to multiple rows

I'm trying extract a number of filename fields from a log file and copy them out as separate rows in a text file so i can load them into a table. I'm able to get the filenames but the all appear on one line. I tried using the cut command with the -d (delimiter) option but cant seem to make it... (1 Reply)
Discussion started by: Sinbad-66
1 Replies

7. UNIX for Dummies Questions & Answers

Need help with Join on multiple fields

Hi, I need help with the join command I have 2 files that I want to join on multiple fields. I want to return all records from file 1 I also want empty fields in my joined file if there isn't a match in file 2 I have already sorted them so I know they are in the same order. file1 ... (0 Replies)
Discussion started by: shunter0810
0 Replies

8. Shell Programming and Scripting

sort on multiple fields

Hello All I have data in a flat file with numeric and aplha numeric datatypes. Now i have to sort on multiple fileds. Can any body please give me the sort code? i am particularly confused about the sort code like sort -n +0 -1 +1 -2 .... (1 Reply)
Discussion started by: vasuarjula
1 Replies

9. Shell Programming and Scripting

sort on multiple fields

hello all I have a file names xxx with data like 1,2,3,12 1,3,6,12 1,3,5,12 2,4,6,12 6,5,6,12 4,2,7,12 4,1,3,12 I wish to sort this file xxx on first three fields in ascending order. OUPUT should be like 1,2,3,12 1,3,5,12 1,3,6,12 2,4,6,12 (4 Replies)
Discussion started by: vasuarjula
4 Replies

10. Shell Programming and Scripting

join on multiple fields

Is it possible to do a join on multiple fields of two files? I am trying to do something like join -t, -1 2,3 -2 2,3 -o 2.1,2.2,2.3,1.3 filea fileb I want the join to be on columns 2 and 3 of filea and columns 2 and 3 of fileb. What is hapenning is that the second file that I want to do the join... (1 Reply)
Discussion started by: reggiej
1 Replies
Login or Register to Ask a Question