Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash to print if keyword not in file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash to print if keyword not in file
# 1  
Old 01-03-2017
Bash to print if keyword not in file
I am trying to create an output file new that contains only the S5-00580 lines from list that are not in analysis_log. My attempt to do this is below.

The new file would be used in the aria2c command to download only new folders. The aria2c command works to download all the files in list, but if they already exist in analysis_log then those lines can be skipped.

Also, all S5-00580-17-Medexome lines are used that text is there and I can not figure out how to ignore lines that have a keyword in them test.... basically exclude all lines that do not end with Medexome.tar.bz2. Thank you Smilie.


Code:
diff -u list analysis_log  | sed -nr 's/^+([^S5-].*)/\1/p' > new

list
Code:
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-19-Medexome_122_059/plugin_out/FileExporter_out.137/R_2016_12_09_14_01_11_user_S5-00580-19-Medexome.tar.bz2
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-18-Medexome_121_057/plugin_out/FileExporter_out.134/R_2016_12_09_11_18_52_user_S5-00580-18-Medexome.tar.bz2
http://xxx.xx.xxx.xxx/output/Home/Auto_S5-00580-17-Medexome_5224_9680c70_120_056/plugin_out/FileExporter_out.125/R_2016_12_07_12_25_50_S5-00580-17-Medexome_5224_9680c70.tar.bz2
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-17-Medexome_119_054/plugin_out/FileExporter_out.122/R_2016_12_05_13_30_48_user_S5-00580-17-Medexome.tar.bz2
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-16-Medexome_118_052/plugin_out/FileExporter_out.119/R_2016_12_05_10_45_37_user_S5-00580-16-Medexome.tar.bz2

analysis_log
Code:
R_2016_11_18_10_45_10_user_S5-00580-17-Medexome
R_2016_11_18_13_19_32_user_S5-00580-16-Medexome

Code:
# verify new files with list call
         line_no=$(awk '{x++} END {print x}' /home/cmccabe/s5_files/downloads/new) # count new files and store as variable
         if [[ -s /home/cmccabe/s5_files/downloads/new ]]; then
     echo "starting download of $line_no new S5 sequencing run"
else
    echo " no new files to analyze, goodbye "
    exit 1
fi

# download all from list
while read new; do
         echo $new
aria2c -x8 -l /home/cmccabe/log.txt -c -d /home/cmccabe/Desktop/NGS/API --use-head=true --http-user "xxxx"  --http-passwd xxxx "$new"
done < /home/cmccabe/s5_files/downloads/new
rm /home/cmccabe/s5_files/downloads/list
rm /home/cmccabe/s5_files/downloads/new

desired output of new only these two lines are printed because the S5-00580 was not in the analysis_log

Code:
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-19-Medexome_122_059/plugin_out/FileExporter_out.137/R_2016_12_09_14_01_11_user_S5-00580-19-Medexome.tar.bz2
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-18-Medexome_121_057/plugin_out/FileExporter_out.134/R_2016_12_09_11_18_52_user_S5-00580-18-Medexome.tar.bz2

# 2  
Old 01-03-2017
Hi, try:
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\.tar\.bz2") p=0}}p' analysis_log list > new

This User Gave Thanks to Scrutinizer For This Post:
cmccabe
# 3  
Old 01-06-2017
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\.tar\.bz2") p=0}}p' /home/cmccabe/analysis_log /home/cmccabe/files/downloads/list > /home/cmccabe/files/downloads/new

awk: cmd. line:1: warning: escape sequence `\.' treated as plain `.'

a new file does get created with the line that is not in analysis_log, so it apperas to be working just not sure what the error means (seems like "Medexome\.tar\.bz2" is causing the error message? Thank you very much Smilie.

---------- Post updated at 06:58 AM ---------- Previous update was at 06:54 AM ----------

if I remove the \, I get no message... but are they need? Thank you Smilie.

So it is "Medexome.tar.bz2"
# 4  
Old 01-06-2017
Hello cmccabe,

As per message it is only warning so definitely program will not be getting impacted. Off course if message is saying you could remove \. to ., yes you could try it out, it shouldn't affect code(though I didn't try it).

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
cmccabe
# 5  
Old 01-06-2017
Thank you both Smilie.
# 6  
Old 01-06-2017
Quote:
Originally Posted by cmccabe
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\.tar\.bz2") p=0}}p' /home/cmccabe/analysis_log /home/cmccabe/files/downloads/list > /home/cmccabe/files/downloads/new

awk: cmd. line:1: warning: escape sequence `\.' treated as plain `.'

a new file does get created with the line that is not in analysis_log, so it apperas to be working just not sure what the error means (seems like "Medexome\.tar\.bz2" is causing the error message? Thank you very much Smilie.

---------- Post updated at 06:58 AM ---------- Previous update was at 06:54 AM ----------

if I remove the \, I get no message... but are they need? Thank you Smilie.

So it is "Medexome.tar.bz2"
Hi, cmccabe, yes they are needed. My suggestion was slightly incorrect. Try this instead:
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~/Medexome\.tar\.bz2/) p=0}}p' analysis_log list

This is a "regex constant" expression..


The double quotes (regex string) would be possible as well, but the dots then would need an extra escape:
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\\.tar\\.bz2") p=0}}p' analysis_log list

\. is necessary, since a . would mean "any character" instead of a literal dot. It would probably work too, but theoretically it could mean a false positive...
This User Gave Thanks to Scrutinizer For This Post:
cmccabe
# 7  
Old 01-06-2017
Thank you very much for your help Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to print the "grep" result as specified keyword order?

I have a content.xls file as given below, NC_020815.1 1891831 1894692 virb4_A0A0H2X8Z4_ 1 954 1945 NC_020815.1 1883937 1886123 vird4_A0A0P9KA26_ 1 729 1379 NC_020815.1 2976151 2974985 virb10_H8FLU5_Ba 1 393 478 NC_020815.1 2968797 2967745 virb6_A0A0Q5GCZ4 5 398 499... (2 Replies)
Discussion started by: dineshkumarsrk
2 Replies

2. UNIX for Beginners Questions & Answers

How to align/sort the column pairs of an csv file, based on keyword word specified in another file?

I have a csv file as shown below, xop_thy 80 avr_njk 50 str_nyu 60 avr_irt 70 str_nhj 60 avr_ngt 50 str_tgt 80 xop_nmg 50 xop_nth 40 cyv_gty 40 cop_thl 40 vir_tyk 80 vir_plo 20 vir_thk 40 ijk_yuc 70 cop_thy 70 ijk_yuc 80 irt_hgt 80 I need to align/sort the csv file based... (7 Replies)
Discussion started by: dineshkumarsrk
7 Replies

3. UNIX for Beginners Questions & Answers

Bash command to find a file and print contents

I need to find a file and print its contents I am trying but it is not working find -path /opt/app-root/src/.npm/_logs -type f -name "*.log" -print Version $ bash -version GNU bash, version 4.4.12(1)-release (x86_64-pc-msys) (1 Reply)
Discussion started by: SVRao19056
1 Replies

4. Shell Programming and Scripting

Print all lines between two keyword if a specific pattern exist

I have input file as below I need to check for a pattern and if it is there in file then I need to print all the lines below BEGIN and END keyword. Could you please help me how to get this in AIX using sed or awk. Input file: ABC ******** BEGIN ***** My name is Amit. I am learning unix.... (8 Replies)
Discussion started by: Amit Joshi
8 Replies

5. Shell Programming and Scripting

Search for a Keyword in file and replace another keyword or add at the end of line

Hi I want to implement something like this: if( keyword1 exists) then check if(keyword2 exists in the same line) then replace keyword 2 with New_Keyword else Add New_Keyword at the end of line end if eg: Check for Keyword JUNGLE and add/replace... (7 Replies)
Discussion started by: dashing201
7 Replies

6. Shell Programming and Scripting

[ksh] finding last file with keyword in it

Hi, In short : I have several log files and I need to find the last file with a certain keyword in it. # ls -1tr logs log_hostX.Jan01_0100.gz log_hostX.Jan01_0105.gz log_hostX.Jan01_0110.gz log_hostX.Jan01_0115.gz log_hostX.Jan01_0120.gz log_hostX.Jan01_0125.gz log_hostX.Jan01_0130.gz... (2 Replies)
Discussion started by: ejdv
2 Replies

7. Shell Programming and Scripting

Reading file contents until a keyword

Hi Guys, I need to read a file until I find a blank line. and in the next iteration I want to continue reading from the line I find a keyword. For ex: my file looks like PDS_JOB_ALIAS CRITERIA_ITEM_TYPE PDS_JOB_CRITERIA_ITEM CRITERIA_ITEM_TYPE First I want to read the file... (2 Replies)
Discussion started by: infintenumbers
2 Replies

8. Shell Programming and Scripting

How to print Specific keyword, by using awk?

How to print Specific keyword, by using awk.? prime:root:I want output. 78 1457 10000 10000 5985 307 10000 10000 10000 10000 3760 692 6656 157 696 (4 Replies)
Discussion started by: ooilinlove
4 Replies

9. Solaris

Keyword search input from a file

Hi, I have a file which got only one column and got some keywords. I have another file where the keywords used in the first file are repeated in the second file. Now I would like to know how many times each keyword from the first file is repeated in the second file. Request your help on... (1 Reply)
Discussion started by: pointers
1 Replies

10. Shell Programming and Scripting

compare two files and search keyword and print output

You have two files to compare by searching keyword from one file into another file File A 23 >pp_ANSWER 24 >aa hello 25 >jau head wear 66 >jss oops 872 >aqq olps ploww oww sss 722 >GG_KILLER ..... large files File B Beta done KILLER John Mayor calix meyers ... (5 Replies)
Discussion started by: cdfd123
5 Replies
Login or Register to Ask a Question