Thank you very much for the explanations, I am trying to process it and think it makes sense and will be very helpful later on. I appreciate it .

The awk using the entire one line of index.html returns the above output (which is very close). In each Alignments there is the portion in bold IonXpress_004_R_2016_09_01_10_24_52_user_S5-00580-4-Medexome_Auto_user_S5-00580-4-Medexome_65 that will match the portion in bold in path, /results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52. Thus grouping all the matching strings together in the same order aalready outputting. That way any duplicates can be identified (like line 1 and the last line) and removed. If there is no match found then the next line is processed (nothing needs to happen). I am not sure why there in the index.html, but it looks like the API used to retrieve that file has duplicates in it. In the --Alignments from index.html only the portion up to the second _ is needed. So in IonXpress_004_R_2016_09_01_10_24_52_user_S5-00580-4-Medexome_Auto_user_S5-00580-4-Medexome_65 only IonXpress_004. I will write something as well, but I'm sure it will need work.

Thank you very much for all of your help .

So using the above as an example:

IonXpress_004_R_2016_09_01_10_24_52_user_S5-00580-4-Medexome_Auto_user_S5-00580-4-Medexome_65 -- Alignments from index.html portion in bold matches portion in bold from

/results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52 -- path from index.html so all the user_S5-00580-4-Medexome are grouped together.

Code:

/results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52    -- path from index.html
R_2016_09_01_10_24_52_user_S5-00580-4-Medexome    -- Aligned Reads from index.html
IonXpress_004    -- Alignments from index.html
MEV42    -- Sample Name from index.html
IonXpress_005    -- Alignments from index.html
MEV43    -- Sample Name from index.html
IonXpress_006    -- Alignments from index.html
MEV44    -- Sample Name from index.html

awk attempt

Code:

awk -F'"[]},:]* *"*' -v RS='{' '
{for(i = 2; i < NF - 1; i++) {
if($i == "path" &&
   $(i + 2) == "plugin" &&
   $(i + 3) == "/rundb/api/v1/plugin/49/")
     print $(i+1) "    -- path from " FILENAME
      else if ($i  == "Aligned Reads")
              print $(i+1) | awk '!x[$0]++' "    -- Aligned Reads from " FILENAME
      else if ($i == "Alignments")
              print $(i+1) | awk -F_R_* '{print $1}' "    -- Alignments from " FILENAME  
      else if($i  == "Sample Name")
              print $(i+1) "    -- Sample Name from " FILENAME
    }
}' index.html | awk 'match($0, /_user\([^_]+)/) { print substr( $0, RSTART, RLENGTH )}' > out

1 bold marking removes duplicates in Aligned Reads
2 bold marking parses Alignments using the second _ removing everything after
3 bold marking groups all _user in Alignments with path (I don't think this will work as I just removed the _user from Alignments