I am trying to create an output file new that contains only the S5-00580 lines from list that are not in analysis_log. My attempt to do this is below.
The new file would be used in the aria2c command to download only new folders. The aria2c command works to download all the files in list, but if they already exist in analysis_log then those lines can be skipped.
Also, all S5-00580-17-Medexome lines are used that text is there and I can not figure out how to ignore lines that have a keyword in them test.... basically exclude all lines that do not end with Medexome.tar.bz2. Thank you .
Code:
diff -u list analysis_log | sed -nr 's/^+([^S5-].*)/\1/p' > new
# verify new files with list call
line_no=$(awk '{x++} END {print x}' /home/cmccabe/s5_files/downloads/new) # count new files and store as variable
if [[ -s /home/cmccabe/s5_files/downloads/new ]]; then
echo "starting download of $line_no new S5 sequencing run"
else
echo " no new files to analyze, goodbye "
exit 1
fi
# download all from list
while read new; do
echo $new
aria2c -x8 -l /home/cmccabe/log.txt -c -d /home/cmccabe/Desktop/NGS/API --use-head=true --http-user "xxxx" --http-passwd xxxx "$new"
done < /home/cmccabe/s5_files/downloads/new
rm /home/cmccabe/s5_files/downloads/list
rm /home/cmccabe/s5_files/downloads/new
desired output of new only these two lines are printed because the S5-00580 was not in the analysis_log
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\.tar\.bz2") p=0}}p' /home/cmccabe/analysis_log /home/cmccabe/files/downloads/list > /home/cmccabe/files/downloads/new
awk: cmd. line:1: warning: escape sequence `\.' treated as plain `.'
a new file does get created with the line that is not in analysis_log, so it apperas to be working just not sure what the error means (seems like "Medexome\.tar\.bz2" is causing the error message? Thank you very much .
---------- Post updated at 06:58 AM ---------- Previous update was at 06:54 AM ----------
if I remove the \, I get no message... but are they need? Thank you .
As per message it is only warning so definitely program will not be getting impacted. Off course if message is saying you could remove \. to ., yes you could try it out, it shouldn't affect code(though I didn't try it).
Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\.tar\.bz2") p=0}}p' /home/cmccabe/analysis_log /home/cmccabe/files/downloads/list > /home/cmccabe/files/downloads/new
awk: cmd. line:1: warning: escape sequence `\.' treated as plain `.'
a new file does get created with the line that is not in analysis_log, so it apperas to be working just not sure what the error means (seems like "Medexome\.tar\.bz2" is causing the error message? Thank you very much .
---------- Post updated at 06:58 AM ---------- Previous update was at 06:54 AM ----------
if I remove the \, I get no message... but are they need? Thank you .
So it is "Medexome.tar.bz2"
Hi, cmccabe, yes they are needed. My suggestion was slightly incorrect. Try this instead:
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~/Medexome\.tar\.bz2/) p=0}}p' analysis_log list
This is a "regex constant" expression..
The double quotes (regex string) would be possible as well, but the dots then would need an extra escape:
Code:
awk -F_ 'NR==FNR{A[$NF]; next} {p=1; for(i in A) { if($0~i || $0!~"Medexome\\.tar\\.bz2") p=0}}p' analysis_log list
\. is necessary, since a . would mean "any character" instead of a literal dot. It would probably work too, but theoretically it could mean a false positive...
This User Gave Thanks to Scrutinizer For This Post:
I need to find a file and print its contents
I am trying but it is not working
find -path /opt/app-root/src/.npm/_logs -type f -name "*.log" -print
Version
$ bash -version
GNU bash, version 4.4.12(1)-release (x86_64-pc-msys) (1 Reply)
I have input file as below I need to check for a pattern and if it is there in file then I need to print all the lines below BEGIN and END keyword. Could you please help me how to get this in AIX using sed or awk.
Input file:
ABC
******** BEGIN *****
My name is Amit.
I am learning unix.... (8 Replies)
Hi
I want to implement something like this:
if( keyword1 exists)
then
check if(keyword2 exists in the same line)
then replace keyword 2 with New_Keyword
else
Add New_Keyword at the end of line
end if
eg:
Check for Keyword JUNGLE and add/replace... (7 Replies)
Hi,
In short : I have several log files and I need to find the last file with a certain keyword in it.
# ls -1tr logs
log_hostX.Jan01_0100.gz
log_hostX.Jan01_0105.gz
log_hostX.Jan01_0110.gz
log_hostX.Jan01_0115.gz
log_hostX.Jan01_0120.gz
log_hostX.Jan01_0125.gz
log_hostX.Jan01_0130.gz... (2 Replies)
Hi Guys,
I need to read a file until I find a blank line. and in the next iteration I want to continue reading from the line I find a keyword.
For ex: my file looks like
PDS_JOB_ALIAS
CRITERIA_ITEM_TYPE
PDS_JOB_CRITERIA_ITEM
CRITERIA_ITEM_TYPE
First I want to read the file... (2 Replies)
Hi,
I have a file which got only one column and got some keywords. I have another file where the keywords used in the first file are repeated in the second file.
Now I would like to know how many times each keyword from the first file is repeated in the second file.
Request your help on... (1 Reply)
You have two files to compare by searching keyword from one file into another file
File A
23 >pp_ANSWER
24 >aa hello
25 >jau head wear
66 >jss oops
872 >aqq olps ploww oww sss
722 >GG_KILLER
..... large files
File B
Beta done
KILLER
John Mayor
calix meyers
... (5 Replies)