Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Pairing the nth elements on multiple lines iteratively

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Pairing the nth elements on multiple lines iteratively
# 1  
Old 08-01-2015
Pairing the nth elements on multiple lines iteratively
Hello,

I'm trying to create a word translation section of a book. Each entry in the word list will come from a set of linguistically analyzed texts.

Each sentence in the text has the following format. The first element in each line is the "name" of the line (i.e. "A","B","C","D"). The first line is the object language, the second line is a morpheme gloss, the third and fourth lines are stem/word-level translations:

Code:
A word1 word2 word3 word4
B wordW wordX wordY wordZ
C wordA wordB wordC wordD
D wordI wordII wordIII wordIV

What I'd like to do is pull the nth element in 2 or more lines (not counting the line "name"), and output them as a pair (or n-tuple) on the same line, later to be exported as columns to a spreadsheet. So for the above, I'd like:

Code:
word1 wordW wordA wordI
word2 wordX wordB wordII
word3 wordY wordC wordIII
word4 wordZ wordD wordIV

Note that the initial "name" elements occur several thousand times in the file, and I'd like to take care of all lines so named at the same time. Thanks, any ideas?
Last edited by John Lyon; 08-01-2015 at 02:19 PM.. Reason: adding more info
# 2  
Old 08-01-2015
I'm not sure what you mean by:
Quote:
Note that the initial "name" elements occur several thousand times in the file, and I'd like to take care of all lines so named at the same time.
If you're saying that a single "name" can appear more than once in your input file and that when that happens the input lines need to be combined somehow, you need to give us sample input where that condition exists and show us how that is supposed to affect the output.

First, save the following in a file named tester:
Code:
#!/bin/ksh
awk '
{	for(i = 2; i <= NF; i++)
		o[i - 1, NR] = $i
	if(NF > m) m = NF
}
END {	for(i = 1; i < m; i++)
		for(j = 1; j <= NR; j++)
			printf("%s%s", o[i, j], j == NR ? ORS : OFS)
}' "$@"

And, if, and only if, you want to run this on a Solaris/SunOS system, change awk in the script to /usr/xpg4/bin/awk or nawk. Then make the script executable:
Code:
chmod +x tester

And, if you have input files file containing your sample input and file2 containing:
Code:
E e1
F f1 f2 f3 f4 f5
G g1 g2 g3
H h1 h2
I i1 i2 i3 i4 i5 i6 i7

then the command:
Code:
./tester file

produces the output you requested:
Code:
word1 wordW wordA wordI
word2 wordX wordB wordII
word3 wordY wordC wordIII
word4 wordZ wordD wordIV

and the command:
Code:
./tester OFS="," file2

produces the output:
Code:
e1,f1,g1,h1,i1
,f2,g2,h2,i2
,f3,g3,,i3
,f4,,,i4
,f5,,,i5
,,,,i6
,,,,i7

which shows how you can use a comma (or any other character string you want) as the output field separator and that it will correctly align output fields adjusted to account for input files where the number of input columns is not a constant. (If your input always has five input columns and you always want to produce four output rows, you can easily simplify this script some; but I'll leave that as a simple exercise for the reader.)
This User Gave Thanks to Don Cragun For This Post:
John Lyon
# 3  
Old 08-02-2015
Quote:
Originally Posted by John Lyon
Code:
A word1 word2 word3 word4
B wordW wordX wordY wordZ
C wordA wordB wordC wordD
D wordI wordII wordIII wordIV

Base on your data sample try the following one-liner
Code:
awk 'END{for(l=1;l++<NF;)print o[l]}{for(l=I;l++<NF;){o[l]=((o[l])?o[l]FS:S)$l}}' file


Since this is your first post please read and understand the forum rules Smilie
This User Gave Thanks to danmero For This Post:
John Lyon
# 4  
Old 08-02-2015
Thanks to you both for your replies. I was trying to keep it simple, but I should've added more information, I think. Here goes:

The data come from a LaTeX file, which uses a package called "Expex" which formats interlinear analyses of a non-English language.

The following two examples show how the data is laid out. The first line "\gla" is the object language, the second line "\glb" is the underlying form, the third line "\glc" is the morpheme gloss, the fourth line "\glc" is the word translation (the package doesn't allow "\gld" for whatever reason), and the last line "\glft" is the sentence translation. As you see, the number of words varies from example to example, just as natural language sentences may be shorter, or longer.

Each "word" is enclosed in curly brackets in the first two lines (though other sets of curly brackets may be nested within words), but only separated by spaces in the second two lines. The curly brackets are necessary to delimit words in the first two lines since some latex commands (e.g. "\ts" below) require blank spaces after them.

Code:
\gla {itl\'i\textglotstop } {k\textsuperscript{w}uk\textsuperscript{w}} {t\textschwa cx\textsuperscript{w}\'u\texthalflength\texthalflength y.}// 
\glb {itl\'i\textglotstop } {k\textsuperscript{w}uk\textsuperscript{w}} {tc+\ts x\textsuperscript{w}\'uy}//
\glc \textsc{dem} \textsc{rep} \textsc{loc}+go //
\glc from.there they.say came.over.this.way //
\glft `They said he was coming along.' //

\gla {u\textbeltl } {cut} {k\textsuperscript{w}uk\textsuperscript{w}} {al\'a\textglotstop } {lut} {i\textglotstop } {q\'aqx\textsuperscript{w}\textschwa lx} {ka\textglotstop} {cx\textsuperscript{w}uys} {i\textglotstop } {l} {siw\textbeltl k\textsuperscript{w}.} //
\glb {u\textbeltl } {cut} {k\textsuperscript{w}uk\textsuperscript{w}} {al\'a\textglotstop } {lut} {i\textglotstop } {q\'a(\tb)\ts qx\textsuperscript{w}lx} {ki\textglotstop} {c\textendash \ts x\textsuperscript{w}uy\textendash s} {i\textglotstop } {l} {siw\textbeltl k\textsuperscript{w}} //
\glc \textsc{conj} say \textsc{rep} \textsc{dem} \textsc{neg} \textsc{det} fish \textsc{comp.obl} \textsc{cust}\textendash go\textendash \textsc{3sg.poss} \textsc{det} \textsc{loc} water //
\glc and he.said they.say here no the fish where.that they.come the through water //
\glft `Coyote said there will be no fish going through the water here.' //

The \glft line may be ignored, but what I'd like exactly is the following, where "&" denotes a column separator in LaTeX and "\\" indicates a newline. Each line has 4 "words", i.e. the nth word in each of the first four lines in the examples above.

Code:
{itl\'i\textglotstop } & {itl\'i\textglotstop } & \textsc{dem} & from.there \\
{k\textsuperscript{w}uk\textsuperscript{w}} & {k\textsuperscript{w}uk\textsuperscript{w}} &  \textsc{rep} & they.say \\

Etcetera. Once the first example is done, the second example would be appended to the above list. Eventually each line will be sorted alphabetically by the first "column". It'd also be nice to be able to choose which input lines to include in the output, though I'd greatly appreciate any more assistance you could give in obtaining the basic result just outlined. Thanks again.
# 5  
Old 08-02-2015
Please don't give us "Etcetera."! Show us the exact output you are trying to produce from the 11 line sample input you showed us.

We need to see what is supposed to be done in the output when there are unequal numbers of "words" in input lines.

We need to see how the output lines corresponding to groups of input lines are supposed to be separated.

If you want output sorted, you also need to explain MUCH more clearly what the sort key is and explain how sorting on the 1st column of the output is going to maintain groups of associated output lines??? (The sort utility sorts lines; not line groups!)

You have been given sample awk scripts that work with the sample input you originally provided. Have you tried modifying those scripts to work with your (radically) different real input? What did you try? Where did you get stuck?
# 6  
Old 08-02-2015
Thanks for the reply, apologies for being vague, I'm new to all this. To be clear, the following input consists of two example sentences. There are a combined total of 15, curly-bracket enclosed words in the \gla lines of these two examples (3 in the first, 12 in the second):

Code:
\gla {itl\'i\textglotstop } {k\textsuperscript{w}uk\textsuperscript{w}} {t\textschwa cx\textsuperscript{w}\'u\texthalflength\texthalflength y.}//  
\glb {itl\'i\textglotstop } {k\textsuperscript{w}uk\textsuperscript{w}} {tc+\ts x\textsuperscript{w}\'uy}// 
\glc \textsc{dem} \textsc{rep} \textsc{loc}+go // 
\glc from.there they.say came.over.this.way //
\glft `They said he was coming along.' //  

\gla {u\textbeltl } {cut} {k\textsuperscript{w}uk\textsuperscript{w}} {al\'a\textglotstop } {lut} {i\textglotstop } {q\'aqx\textsuperscript{w}\textschwa lx} {ka\textglotstop} {cx\textsuperscript{w}uys} {i\textglotstop } {l} {siw\textbeltl k\textsuperscript{w}.} // 
\glb {u\textbeltl } {cut} {k\textsuperscript{w}uk\textsuperscript{w}} {al\'a\textglotstop } {lut} {i\textglotstop } {q\'a(\tb)\ts qx\textsuperscript{w}lx} {ki\textglotstop} {c\textendash \ts x\textsuperscript{w}uy\textendash s} {i\textglotstop } {l} {siw\textbeltl k\textsuperscript{w}} // 
\glc \textsc{conj} say \textsc{rep} \textsc{dem} \textsc{neg} \textsc{det} fish \textsc{comp.obl} \textsc{cust}\textendash go\textendash \textsc{3sg.poss} \textsc{det} \textsc{loc} water // 
\glc and he.said they.say here no the fish where.that they.come the through water // 
\glft `Coyote said there will be no fish going through the water here.' //

Given this input, this is the initial output I'm looking for:

Code:
{itl\'i\textglotstop } & {itl\'i\textglotstop } & \textsc{dem} & from.there \\
{k\textsuperscript{w}uk\textsuperscript{w}} & {k\textsuperscript{w}uk\textsuperscript{w}} &  \textsc{rep} & they.say \\
{t\textschwa cx\textsuperscript{w}\'u\texthalflength\texthalflength y.} & {tc+\ts x\textsuperscript{w}\'uy} & \textsc{loc}+go &  came.over.this.way \\
{u\textbeltl } & {u\textbeltl } & \textsc{conj} &  and \\
{cut} & {cut} & say & he.said \\
{k\textsuperscript{w}uk\textsuperscript{w}} & {k\textsuperscript{w}uk\textsuperscript{w}} &  \textsc{rep} & they.say \\
{al\'a\textglotstop } &  {al\'a\textglotstop } & \textsc{dem} & here \\
{lut} & {lut} & \textsc{neg} & no \\
{i\textglotstop } & {i\textglotstop } & \textsc{det} & the \\
{q\'aqx\textsuperscript{w}\textschwa lx} & {q\'a(\tb)\ts qx\textsuperscript{w}lx} & fish & fish \\
{ka\textglotstop} & {ki\textglotstop} & \textsc{comp.obl} & where.that \\
{cx\textsuperscript{w}uys} & {c\textendash \ts x\textsuperscript{w}uy\textendash s} &  \textsc{cust}\textendash go\textendash \textsc{3sg.poss} & they.come \\
{i\textglotstop } & {i\textglotstop } & \textsc{det} & the \\
{l} & {l} & \textsc{loc} & through \\
{siw\textbeltl k\textsuperscript{w}} &  {siw\textbeltl k\textsuperscript{w}} & water & water \\

Then, these 15 lines would be sorted, the sort key being the first letter of the first word in each line, so the above 15 lines (corresponding to the total of 15 words in the \gla lines of the two unmodified examples), would be sorted like this:

Code:
{al\'a\textglotstop } &  {al\'a\textglotstop } & \textsc{dem} & here \\
{cut} & {cut} & say & he.said \\
{cx\textsuperscript{w}uys} & {c\textendash \ts x\textsuperscript{w}uy\textendash s} &  \textsc{cust}\textendash go\textendash \textsc{3sg.poss} & they.come \\
{itl\'i\textglotstop } & {itl\'i\textglotstop } & \textsc{dem} & from.there \\
{i\textglotstop } & {i\textglotstop } & \textsc{det} & the \\
{i\textglotstop } & {i\textglotstop } & \textsc{det} & the \\
{ka\textglotstop} & {ki\textglotstop} & \textsc{comp.obl} & where.that \\
{k\textsuperscript{w}uk\textsuperscript{w}} & {k\textsuperscript{w}uk\textsuperscript{w}} &  \textsc{rep} & they.say \\
{k\textsuperscript{w}uk\textsuperscript{w}} & {k\textsuperscript{w}uk\textsuperscript{w}} &  \textsc{rep} & they.say \\
{l} & {l} & \textsc{loc} & through \\
{lut} & {lut} & \textsc{neg} & no \\
{q\'aqx\textsuperscript{w}\textschwa lx} & {q\'a(\tb)\ts qx\textsuperscript{w}lx} & fish & fish \\
{siw\textbeltl k\textsuperscript{w}} &  {siw\textbeltl k\textsuperscript{w}} & water & water \\
{t\textschwa cx\textsuperscript{w}\'u\texthalflength\texthalflength y.} & {tc+\ts x\textsuperscript{w}\'uy} & \textsc{loc}+go &  came.over.this.way \\
{u\textbeltl } & {u\textbeltl } & \textsc{conj} &  and \\

Lines 5/6 and lines 8/9 above are duplicates, so the duplicate entries will be removed from the list, yielding 13 lines:

Code:
{al\'a\textglotstop } &  {al\'a\textglotstop } & \textsc{dem} & here \\
{cut} & {cut} & say & he.said \\
{cx\textsuperscript{w}uys} & {c\textendash \ts  x\textsuperscript{w}uy\textendash s} &  \textsc{cust}\textendash  go\textendash \textsc{3sg.poss} & they.come \\
{itl\'i\textglotstop } & {itl\'i\textglotstop } & \textsc{dem} & from.there \\
{i\textglotstop } & {i\textglotstop } & \textsc{det} & the \\
{ka\textglotstop} & {ki\textglotstop} & \textsc{comp.obl} & where.that \\
{k\textsuperscript{w}uk\textsuperscript{w}} &  {k\textsuperscript{w}uk\textsuperscript{w}} &  \textsc{rep} &  they.say \\
{l} & {l} & \textsc{loc} & through \\
{lut} & {lut} & \textsc{neg} & no \\
{q\'aqx\textsuperscript{w}\textschwa lx} & {q\'a(\tb)\ts qx\textsuperscript{w}lx} & fish & fish \\
{siw\textbeltl k\textsuperscript{w}} &  {siw\textbeltl k\textsuperscript{w}} & water & water \\
{t\textschwa cx\textsuperscript{w}\'u\texthalflength\texthalflength y.}  & {tc+\ts x\textsuperscript{w}\'uy} & \textsc{loc}+go &   came.over.this.way \\
{u\textbeltl } & {u\textbeltl } & \textsc{conj} &  and \\

The result will be an alphabetized vocabulary list, ready to be dropped into a "tabularx" table environment in LaTeX.

I had some luck with danmero's suggestion:
Code:
awk 'END{for(l=1;l++<NF;)print o[l]}{for(l=I;l++<NF;){o[l]=((o[l])?o[l]FS:S)$l}}' file

However, it only worked if (a) all of the extra blank spaces within "words" were removed (since it seems to use blank spaces as a word delimiter), and (b) only one example at a time is modified (since I think it assumes "line names" do not occur multiple times). Both of these issues are my fault, for not being clear during the initial posting about the nature of the data I'm working with. Also, I don't yet know enough about awk to identify what in the above command needs changing. Thanks for your assistance and patience! I hope this helps to clarify.
# 7  
Old 08-03-2015
I'm confused, I thought you said that each <space> character in the 3rd and later lines in each input "sentence" separated "words". So, in the 12th line of your desired output:
Code:
{cx\textsuperscript{w}uys} & {c\textendash \ts x\textsuperscript{w}uy\textendash s} &  \textsc{cust}\textendash go\textendash \textsc{3sg.poss} & they.come \\

why are there two <space> characters in the middle of the single "word" marked in red from the following input line?:
Code:
\glc \textsc{conj} say \textsc{rep} \textsc{dem} \textsc{neg} \textsc{det} fish \textsc{comp.obl} \textsc{cust}\textendash go\textendash \textsc{3sg.poss} \textsc{det} \textsc{loc} water //

I thought I could modify my earlier suggested awk script to handle your new requirements, but since my code thinks there are two additional fields in the 9th line of your sample input, it gets confused and produces the wrong output.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare multiple arrays elements using awk

I need your help to discover missing elements for each box. In theory each box should have 4 items: ITEM01, ITEM02, ITEM08, and ITEM10. Some boxes either have a missing item (BOX02 ITEM08) or might have da duplicate item (BOX03 ITEM02) and missing another one (BOX03 ITEM01). file01.txt ... (2 Replies)
Discussion started by: alex2005
2 Replies

2. UNIX for Dummies Questions & Answers

Getting the lines with nth column non-null

Hi, I have a huge list of archives (.gz). Each archive is about 40MB. A file is generated every minute so if I want to analyze the data for 1 hour I get already 60 files for example. These are text files, ';' separated, each line having about 300 fields (columns). What I need to do is to... (11 Replies)
Discussion started by: Nenad
11 Replies

3. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

4. UNIX for Dummies Questions & Answers

Rename multiple files in shell bash, changing elements order.

Hi, I want to rename several files like this: example: A0805120817.BHN A0805120818.BHN ..... to: 20120817.0805.N 20120818.0805.N ...... How can i do this via terminal or in shell bash script ? thanks, (6 Replies)
Discussion started by: pintolcv
6 Replies

5. Shell Programming and Scripting

Extracting lines after nth LINE from an output

Hi all, Here is my problem for which i am breaking my head for past three days.. I have parted command output as follows.. Model: ATA WDC WD5000AAKS-0 (scsi) Disk /dev/sdb: 500GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type ... (3 Replies)
Discussion started by: selvarajvs
3 Replies

6. Shell Programming and Scripting

How to output all lines following Nth occurrence of string

Greetings experts. Searched the forums (perhaps not hard enough?) - Am searching for a method to capture all output from a log file following the nth occurrence of a known string. Background: Using bash, I want to monitor my Oracle DB alert log file. The script will count the total # of... (2 Replies)
Discussion started by: cjtravis
2 Replies

7. Shell Programming and Scripting

Array output through a for loop problematic with multiple elements.

This code works perfect when using a machine with only one interface online. (Excluding the loopback of course) But when I have other interface up for vmware or a vpn the output gets mixed up. I know I had this working when I was just reading ip's from files so I know it is not a problem with... (8 Replies)
Discussion started by: Azrael
8 Replies

8. UNIX for Dummies Questions & Answers

Finding nth line across multiple files

I have several files (around 50) that have the similar format. I need to extract the 5th line from every file and output that into a text file. So far, I have been able to figure out how to do it for a single file: $ awk 'NR==5' text1.txt > results.txt OR $ sed -n '5p' text1.txt > results.txt... (6 Replies)
Discussion started by: oriqin
6 Replies

9. UNIX for Dummies Questions & Answers

Getting number of lines of nth occurrency

Hi all, I would like to extract the line number of the n-th occurrency of a given string in a file. e.g. xxx yyy xxx zzz xxx the second occurrency of xxx is at line 3. What is the fastest way to do it in bash? Thank you, (8 Replies)
Discussion started by: f_o_555
8 Replies

10. Shell Programming and Scripting

Average of elements throught multiple files

Hi, I got a lot of files looking like this: 1 0.5 6 All together there are ard 1'000'000 lines in each of the ard 100 files. I want to build the average for every line, and write the result to a new file. The averaging should start at a specific line, here for example at line... (10 Replies)
Discussion started by: chillmaster
10 Replies
Login or Register to Ask a Question