Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Combining two perl commands into one

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combining two perl commands into one
# 1  
Old 03-27-2020
Combining two perl commands into one
I made a program that extracts quotes while retaining special inner quotes (in this case an 'x' followed by an apostrophe). The original program is far more complicated than this, but I wanted to make it simple to troubleshoot.


I want to take these two perl commands and have the first command's results be piped into the second commands input but while only running perl once:


Input: ['Say, x'Hix'']
Code:
echo "['Say, x'Hix'']" | perl -lne 'push @a,/\[\N{U+0027}(.*?)(?<!x)\N{U+0027}/g; END{print "@a"};'

Output: Say, x'Hix'

Taking the output of the first command:

Code:
echo "Say, x'Hix'" | perl -pe 's/x\N{U+0027}/\N{U+0027}/g; END{print "@a"};'

Result (proper): Say, 'Hi'



I realise that I could easily just run perl twice and pipe them into eachother, but running perl twice seems inefficient; especially given that this command is ran thousands of times:
Code:
echo "['Say, x'Hix'']" | perl -lne 'push @a,/\[\N{U+0027}(.*?)(?<!x)\N{U+0027}/g; END{print "@a"};' |  perl -pe 's/x\N{U+0027}/\N{U+0027}/g; END{print "@a"};'

I've tried combining both commands in the code below, but it doesn't seem to be taking the output of the first command as input for the second:
Code:
echo "['Say, x'Hix'']" | perl -lne 'push @a,/\[\N{U+0027}(.*?)(?<!x)\N{U+0027}/g; END{print "@a"};' -pe 's/x\N{U+0027}/\N{U+0027}/g; END{print "@a"};'

Result (not what I want):

['Say, 'Hi'']
Say, x'Hix'
Say, x'Hix'


Any ideas?
# 2  
Old 03-28-2020
Hi
Code:
perl -pe 's/\w(?=\x27)|\[\x27|\x27\]//g'

I think that the data still has a more complex form than in the example?

--- Post updated at 14:06 ---

Code:
perl -pe 's/\W*(\w+,\s*).((\x27)\w+)\w.*/$1$2$3/'

Last edited by nezabudka; 03-28-2020 at 07:36 AM..
# 3  
Old 03-28-2020
Quote:
Originally Posted by nezabudka
Hi
Code:
perl -pe 's/\w(?=\x27)|\[\x27|\x27\]//g'

I think that the data still has a more complex form than in the example?

--- Post updated at 14:06 ---

Code:
perl -pe 's/\W*(\w+,\s*).((\x27)\w+)\w.*/$1$2$3/'

Thank you for posting that code. I tried it, but it didn't have any effect. The code is more complex. The program takes the output of Google Translate and sifts through a bunch of garbage to find the translation. I almost think Google makes it complex on purpose to discourage ripping it off the net...

Here is that part of the code. I didn't want to post it initially, as it is quite overwhelming, but I think it's also unfair to leave you guys in the dark:

Code:
text='Il dit, "Ça va?" Elle dit, "Oui, bien sûr!"'
translation=$(wget -U "Mozilla/5.0" -q -O- "http://translate.googleapis.com/translate_a/single?client=gtx&sl=fr&tl=en&dt=t&q=$text");
echo
echo $translation
echo
echo $translation | perl -lne 'push @a,/(?<!\,\[\[)\[\"(.*?)(?<!\\)\"/g;END{print "@a"}'\
                    | perl -CS -pwe ' s/\N{U+005C}\N{U+0022}/\N{U+0022}/g;'

005C = backslash
0022 = quote

The result is: He says, "Are you okay?" She says, "Yes, of course!"


...


If anyone is really bored and wants to see the full code, it is here (free to use, edit, and distribute... as always Smilie):


A simple Linux shell script for translating an .srt file into another language and merging both languages into an .ass file . GitHub


The program allows a person to take an .srt file, translate it into another language, and merge both languages into an .ass file.


So it can be understood a little better, here is a picture of my computer playing a video with a subtitle produced by the program:

Image


This project allows me to learn French by watching movies with both French and English subtitles alongside eachother. Quite a worthwhile venture for a language learner like myself.
This User Gave Thanks to bedtime For This Post:
nezabudka
# 4  
Old 03-28-2020
Or maybe just do throw an agent?
GitHub - soimort/translate-shell: Command-line translator using Google Translate, Bing Translator, Yandex.Translate, etc.
Code:
trans -s fr -t en -b "$text"
He says, "Are you okay?" She says, "Yes, of course!"

--- Post updated at 17:12 ---

Code:
      if [ "$task" == "translating" ]; then
         translation=$(trans -s en -t fr -b "$text")
      else

Code:
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
  [  0%] Dialogue: 0,0:04:23.80,0:04:26.55,en,,0000,0000,0000,,Good morning, James.
06:52:01 Dialogue: 0,0:04:23.80,0:04:26.55,fr,,0000,0000,0000,,Bonjour James.
  [  0%] Dialogue: 0,0:04:26.55,0:04:27.85,en,,0000,0000,0000,,How are you feeling?
06:51:41 Dialogue: 0,0:04:26.55,0:04:27.85,fr,,0000,0000,0000,,Comment allez-vous ?
  [  0%] Dialogue: 0,0:04:27.85,0:04:29.10,en,,0000,0000,0000,,Wait. What?
06:51:21 Dialogue: 0,0:04:27.85,0:04:29.10,fr,,0000,0000,0000,,Attendez. Quelle ?
  [  0%] Dialogue: 0,0:04:29.10,0:04:32.31,en,,0000,0000,0000,,It's perfectly normal to feel confused.
06:51:01 Dialogue: 0,0:04:29.10,0:04:32.31,fr,,0000,0000,0000,,Il est parfaitement normal de se sentir confus.
  [  0%] Dialogue: 0,0:04:32.31,0:04:35.90,en,,0000,0000,0000,,You just spent 120 years in suspended animation.
06:50:41 Dialogue: 0,0:04:32.31,0:04:35.90,fr,,0000,0000,0000,,Vous venez de passer 120 ans en animation suspendue.
  [  0%] Dialogue: 0,0:04:35.90,0:04:37.52,en,,0000,0000,0000,,What?
06:50:21 Dialogue: 0,0:04:35.90,0:04:37.52,fr,,0000,0000,0000,,Quelle ?
  [  0%] Dialogue: 0,0:04:37.52,0:04:39.32,en,,0000,0000,0000,,It's okay, James.
06:50:01 Dialogue: 0,0:04:37.52,0:04:39.32,fr,,0000,0000,0000,,Ça va, James.
  [  0%] Dialogue: 0,0:04:39.32,0:04:41.86,en,,0000,0000,0000,,-It's Jim. -Jim.
06:49:41 Dialogue: 0,0:04:39.32,0:04:41.86,fr,,0000,0000,0000,,-C'est Jim. -Jim.

But I must admit, this did not increase speed. You would thoroughly rewrite the entire script, send the entire dialogs for translation and then parse it along with the original. And only then I think you can run the program in real time and not for entire night
This User Gave Thanks to nezabudka For This Post:
bedtime
# 5  
Old 03-28-2020
Quote:
Originally Posted by nezabudka
Or maybe just do throw an agent?
GitHub - soimort/translate-shell: Command-line translator using Google Translate, Bing Translator, Yandex.Translate, etc.
Code:
trans -s fr -t en -b "$text"
He says, "Are you okay?" She says, "Yes, of course!"

It's a great program, but I prefer my program to be independent. Most people don't want to have to install a script plus another program. Many are shy of scripts to begin with. My program does it all in one easy script and doesn't require installation. In the end, they both bring out the same translation (as far as I can tell, and with non-asian languages), though mine does it in one line and trans takes an entire installation.


But the biggest thing is that I like to know what's going on. trans is far too advanced for me to follow. My simple script is perfect for me and other learners to build on.


That said, if that part of my code should ever fail due to Google changing its algorithm-which could easily happen-trans would be an awesome drop-in replacement until I patched my own. I'd have no qualms using it in such a case. It's a great idea. Thank you!
This User Gave Thanks to bedtime For This Post:
nezabudka
# 6  
Old 03-28-2020
In the Fedora is install like this
Code:
dnf install translate-shell

And thank you
I liked the program, I'll try to work with it. I'm just learning English Smilie
This User Gave Thanks to nezabudka For This Post:
bedtime
# 7  
Old 03-28-2020
Quote:
Originally Posted by nezabudka
Or maybe just do throw an agent?
GitHub - soimort/translate-shell: Command-line translator using Google Translate, Bing Translator, Yandex.Translate, etc.
Code:
trans -s fr -t en -b "$text"
He says, "Are you okay?" She says, "Yes, of course!"

--- Post updated at 17:12 ---

Code:
      if [ "$task" == "translating" ]; then
         translation=$(trans -s en -t fr -b "$text")
      else

Code:
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
  [  0%] Dialogue: 0,0:04:23.80,0:04:26.55,en,,0000,0000,0000,,Good morning, James.
06:52:01 Dialogue: 0,0:04:23.80,0:04:26.55,fr,,0000,0000,0000,,Bonjour James.
  [  0%] Dialogue: 0,0:04:26.55,0:04:27.85,en,,0000,0000,0000,,How are you feeling?
06:51:41 Dialogue: 0,0:04:26.55,0:04:27.85,fr,,0000,0000,0000,,Comment allez-vous ?
  [  0%] Dialogue: 0,0:04:27.85,0:04:29.10,en,,0000,0000,0000,,Wait. What?
06:51:21 Dialogue: 0,0:04:27.85,0:04:29.10,fr,,0000,0000,0000,,Attendez. Quelle ?
  [  0%] Dialogue: 0,0:04:29.10,0:04:32.31,en,,0000,0000,0000,,It's perfectly normal to feel confused.
06:51:01 Dialogue: 0,0:04:29.10,0:04:32.31,fr,,0000,0000,0000,,Il est parfaitement normal de se sentir confus.
  [  0%] Dialogue: 0,0:04:32.31,0:04:35.90,en,,0000,0000,0000,,You just spent 120 years in suspended animation.
06:50:41 Dialogue: 0,0:04:32.31,0:04:35.90,fr,,0000,0000,0000,,Vous venez de passer 120 ans en animation suspendue.
  [  0%] Dialogue: 0,0:04:35.90,0:04:37.52,en,,0000,0000,0000,,What?
06:50:21 Dialogue: 0,0:04:35.90,0:04:37.52,fr,,0000,0000,0000,,Quelle ?
  [  0%] Dialogue: 0,0:04:37.52,0:04:39.32,en,,0000,0000,0000,,It's okay, James.
06:50:01 Dialogue: 0,0:04:37.52,0:04:39.32,fr,,0000,0000,0000,,Ça va, James.
  [  0%] Dialogue: 0,0:04:39.32,0:04:41.86,en,,0000,0000,0000,,-It's Jim. -Jim.
06:49:41 Dialogue: 0,0:04:39.32,0:04:41.86,fr,,0000,0000,0000,,-C'est Jim. -Jim.

You are the first to run the script. Thank you! Smilie


Quote:
But I must admit, this did not increase speed. You would thoroughly rewrite the entire script, send the entire dialogs for translation and then parse it along with the original. And only then I think you can run the program in real time and not for entire night
Good point. And thank you so much for looking at my script.

Here is the issue with sending Google big chucks to translate: My first program did just that; it would send out 50 lines of text at once. It was very, very fast; a movie of 1000 lines could be translated under 10 minutes. The problem came with sorting out what text came from what line when it had to be reinserted.


For example:

Pseudo .srt movie file:


Code:
1.
How are you today?
2.
I am fine.
3.
Great!
4.
Lets go bird-watching.

In order to send the text to Google, you must give it all in one line, so you'd need to insert some type of mark to know where each line began so they could be inserted back in the proper order, so here is how I'd send it:

Code:
How are you today?\nI am fine\nGreat!\nLets go bird-watching.

The problem is that Google would turn the '\n' into something mysterious and ruin it. It might work well for 10 lines or 50, but it would never do all 1000 lines. I then tried:

Code:
How are you today? \n. I am fine \n. Great! \n. Lets go bird-watching.

The text itself would be fine, but the ' \n. ' would be dismantled just as it was before. I'd tried dozens and dozens of combinations: ###, ~~~, ~~~~~~, @@@, (TEXT_BREAK) , ... Every single time Google would do some thing like this:

Code:
How are you today? ### I am fine### Great!# ##Lets go bird-watching.

Note the break in the third '#' and the strange inserted spaces? I think Google knows what's going on and purposely wreaks the pattern so people can't do batch translations easily. I've looked around on reddit and other places and apparently there is no way to tell Google not to touch a certain bit of text (except with a special tag which can only be used when translating websites).

So frustrating.


Quote:
Originally Posted by nezabudka
[/CODE]And only then I think you can run the program in real time and not for entire night
I have movies that are 1000-2000+ lines of text and take 5-12 hours to translate. I've been leaving it on overnight for the past week, and in the morning it's done. I've never had a problem.


Quote:
And thank you
I liked the program, I'll try to work with it. I'm just learning English.
Help yourself. If you have any suggestions, please let me know, here or github.


Parles-tu français ?
Last edited by bedtime; 03-28-2020 at 11:09 AM..
This User Gave Thanks to bedtime For This Post:
nezabudka
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Combining 2 commands

Hello all, I need to send an attachment and text in the body, both in the same Email. Below are two cammand that send the required data in separate Emails. I need to combine them so that I get just 1 Email containing the attachment & text in the body. uuencode ${filename} "${file_}" |... (6 Replies)
Discussion started by: Junaid Subhani
6 Replies

2. Shell Programming and Scripting

How to combining awk commands?

I can achieve two tasks with 2 different awk commands: 1) awk -F";;WORD" '{print $2}' file | sed '/^$/d' #to find surface_word 2) awk -F"bw:|gloss:" '// {print $2}' file | sed '/\//!d; s:/*+*: + :g; s:^+::; s: *+ *$::;' #to find segmentation of surface_word Number 1) finds surface_word... (7 Replies)
Discussion started by: Viernes
7 Replies

3. UNIX for Dummies Questions & Answers

Help with combining the ls and 'file' commands

I have a directory of 3000 files without extensions (Solaris 5.10). I would like to iterate the file names through the 'file' command and output their mime types (most are pdf or jpg, but a very few might be psd or swf which show simply as 'data') So, I would like the output of the 'ls'... (2 Replies)
Discussion started by: pwallace
2 Replies

4. UNIX for Dummies Questions & Answers

Combining resukts of ls commands

Hi, I have a directory with some XML files in it. I can use wildcards to get the list of XMLs I want say I have following XMLs in same dir Employee1.xml Employee2.xml Employee3.xml and Salary1.xml Salary2.xml Salary3.xml apart from other .txt .dat files etc I want to write a unix... (7 Replies)
Discussion started by: dsrookie
7 Replies

5. UNIX for Dummies Questions & Answers

Combining two commands that use sar.

hey can anyone tell me how can i combine these two commands so that it is executed only once, but gives me both the results. IDLE=`sar 30 6 | grep Average | awk '{print $1 $5}' ` sar 30 120 | awk '{print $1" "$5}' >> mailx -m -s "$MSG" xyz@abc.com. (5 Replies)
Discussion started by: Ankur Khatri
5 Replies

6. Shell Programming and Scripting

Combining multiple commands

Hi Guys, I am looking to optimze these 5 SSH lines to a single SSH to get my machine to not hang! lol! cat hosts.lst | xargs -n1 -t -i echo 'home/util/timeout 6 0 ssh -q {} top -b > util/{}.top &' >> r_query_info cat hosts.lst | xargs -n1 -t -i echo 'home/util/timeout 6 0 ssh -q {} uname -r... (5 Replies)
Discussion started by: wick3dsunny
5 Replies

7. UNIX for Advanced & Expert Users

Combining two commands.

Is there anyway to achieve "find /home -name "*.bashrc" 2>/dev/null" and "PS1="\n>"" in the same command? I just wanna add a line to the previous command to change the PS1 variable to ">". (1 Reply)
Discussion started by: raidkridley
1 Replies

8. UNIX for Dummies Questions & Answers

combining commands

Hello all, I am trying to list and count all the files of a particular type in any given directory. I can use the commands separately but when I combine them they do not give an output. The command for counting the files is ls -1 | wc -l and for listing all the file of particular type say... (2 Replies)
Discussion started by: BigTool4u2
2 Replies

9. UNIX for Dummies Questions & Answers

combining sed commands

I would like to change the lines: originalline1 originalline2 to: originalline1new originalline1newline originalline2new originalline2newline To do this, id like to combine the commands: sed 's/^/&new/g' file > newfile1 and sed '/^/ a\\ newline\\ \\ (2 Replies)
Discussion started by: Dave724001
2 Replies

10. UNIX for Dummies Questions & Answers

Combining elements of different commands in history

What is the correct format for a single command that would combine portions of 2 different lines in the command history? I'm using a C shell. Here's a simplified command history to clarify: 4 rm file1 5 ls -ld file2 file3 file4 6 cat file 5 With the above history, what would be the... (5 Replies)
Discussion started by: Dbyte
5 Replies
Login or Register to Ask a Question