while read line
do
if [[ $(awk '/PROJECT_DIR/ && /serial\/lookup/' $line) ]]
then
file_name="lookup"$(echo $line | sed 's/[a-zA-Z]//g')
echo $file_name
mv $line $file_name
fi
done<File_name_temp
But the partitioning of the files is not done properly, I mean starting line of the file is correct but the last lines are overlapping in all files. I guess more attention is required on starting lies only if at all prob arises then we will think of adding condition for the last line also.
But the partitioning of the files is not done properly, I mean starting line of the file is correct but the last lines are overlapping in all files. I guess more attention is required on starting lies only if at all prob arises then we will think of adding condition for the last line also.
Could you please elaborate. Please show input and output. and what overlapping..?
{2010503005|XXGfvertex|1259|0|3869|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|Layout|$[[record kind 85 subkind 0 parts [vector _interp_("mfile:$\{INF_ENTRPRSDWUNFYRETLCRED_MFS\}/m_cdp2_uedw_t_arnge_to_arnge_uld.dat", "dollar_substitution")]]]|3|9||@{0|}}
{30001002|XXparameter|read_metadata|$\{INF_ENTRPRSDWUNFYRETLCRED_DML\}/cdp2_uedw_t_arnge_to_arnge.dml|3|2|f$|@{0|}}
{30001002|XXparameter|!prototype_path|C:\\Program Files\\Ab Initio\\Ab Initio GDE 1_14_5\\Components\\Datasets\\Input_File.mdc|3|2|Pw$|@{0|}}
{30001002|XXparameter|eme_dataset_location|$\{_Projects_SunTrust_inf_inf_EntrprsDwUnfyRetlCred\}/data/mfs/main/m_cdp2_uedw_t_arnge_to_arnge_uld.dat|3|9||@{0|}}
}}@1|@151424|2797588|173000|2818000|56000|50000|39502|m_cdp2_uedw_t_arnge_to_arnge_uld.dat|M K Muralidhar||1|100|-1||6||33537|-1|-1|}}
{2010203004|XXGoport|1260|0|3871|0|{@{}@228000||11000|11000|read|0.0|@@@2160|0|}}
{2010503005|XXGfvertex|1261|0|3874|0|{Represents one file, many files, or a multifile as an input to your graph.|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|protection|0666|12|2|RF$||{0|}}
{30001002|XXparameter|mode|0x0001|1|2|FH$|modes of access|{0|}}
{30001002|XXparameter|Layout|@28|2|RF$||{0|}}
{30001002|XXparameter|read_metadata||7|1|RFl||{0|}}
{30001002|XXparameter|mpcmodtime|1138303912|1|1|Hl|The last modification time of this component's template|{0|}}
{30001002|XXparameter|eme_dataset_location|@3|9|F|Place in the EME to create a dataset corresponding to this file.|{0|}}
}}@0|@0|0|0|0|0|0|0|@@@1|10|-1|@6|@1|-1|-1|}}
{2010203004|XXGoport|1262|0|3876|0|{@{30100001|XXparameter_set|@@@@{{30001002|XXparameter|metadata||7|8|RF=||{0|}}
}}@0|0|0|0|read|0.0|@@@2160|0|}}
{2010501005|XXGpvertex|1263|0|3885|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|transform0|$AI_XFR/cdp2_rdm_dt_core_cnsum_arnge_dim_xfm_rfmt_orig_fico_scor_val.xfr|3|2|f$|@{0|}}
{30001002|XXparameter|out0_metadata|$AI_DML/cdp2_rdm_dt_core_cnsum_arnge_dim_xfm_rfmt_orig_fico_scor_val.dml|3|2|f$|@{0|}}
{30001002|XXparameter|error0_metadata|string('\\n')|3|1|l|@{0|}}
{30001002|XXparameter|log_metadata|record string("\|") node, timestamp, component, subcomponent, event_type; string("\|\\n") event_text; end|3|1|l|@{0|}}
I need to travel through files.
Basically for ur understanding i am explaining this.
I am doing analysis of a graph in unix . graph contains components connected and till now we have taken a graph and divided into components(one text file splitting into different files and name of the components as file names). Now i need to get the flow from each input component to output component(form input file to output file), I need to know in which flow the particular component flows and get connected to output component. For ex:
Input1 is connected to Join and then to Partition by key and then to Reformat and then to FBE and then to Output.(u can ignore the component names , i just said so that through these files it must go).
So We need to take one Inputfile , 'cat' it and then find the output port number(The above number highlighted ) and then it needs to go to XXGFlow file(I have done scripting for getting the file and the above given file is the entire file for temp1.txt) and the need to search the line which has "2844000"(2839000(got this number frm input file)+5000), for this we have
now again 2814000(2809000+5000) need to be searched in XXGFlow file......................... This process continues until the iport number of Output_File is reached in this file the number of Output_file is 1419000 , so in XXGFlow it will be 1424000(1419000+5000). I need this flow file names like
(This is given for example purpose only)
Like this i need for every input file, Here for this particular graph , i have 26 Input_File's , so 26 files of their corresponding flows must be needed.
Praying GOD , so that u understand what i have written
This is second phase and i need to do many phases after this.
Dont ask me why only u r doing +5000 and -5000 , after 1.5 month analysis i found this pattern and even i also dont know y unix is storing in such a way, If u have any idea please let me know.
If u think this is more confusing , i'll expalin u in subparts and we will do it part by part and then accomplish phase2.
Thank q very much for ur patience and if possible pls help me out in this.
Please get back to me for any clarifications
---------- Post updated at 02:06 PM ---------- Previous update was at 02:02 PM ----------
What i mean is:
Code:
Code:
{2010503005|XXGfvertex|234|0|604|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|Layout|$[[record kind 85 subkind 0 parts [vector _interp_("mfile:$\{INF_ENTRPRSDWUNFYRETLCRED_MFS\}/m_cdp2_uedw_t_post_addr_uld.dat", "dollar_substitution")]]]|3|9||@{0|}}
{30001002|XXparameter|read_metadata|$\{INF_ENTRPRSDWUNFYRETLCRED_DML\}/cdp2_uedw_t_post_addr.dml|3|2|f$|@{0|}}
{30001002|XXparameter|key|\{post_addr_id\}|3|2|$|@{0|}}
{30001002|XXparameter|!prototype_path|E:\\program files\\Ab Initio\\Ab Initio GDE 1.15.11.1\\Components\\Datasets\\Input_File.mdc|3|2|Pw$|@{0|}}
{30001002|XXparameter|eme_dataset_location|$\{_Projects_SunTrust_inf_inf_EntrprsDwUnfyRetlCred\}/data/mfs/main/m_cdp2_uedw_t_post_addr_uld.dat|3|9||@{0|}}
}}@1|@625666|493684|648000|514000|56000|50000|38410|m_cdp2_ uedw_t_post_addr_uld.dat|a8sc20||1|10|0||6||33025|-1|-1|}}
{2010203004|XXGoport|235|0|606|0|{@{}@703000|534000|11000|11000|read|0.0|@@@2160|0|}}
{2010503005|XXGfvertex|236|0|609|0|{Represents one file, many files, or a multifile as an input to your graph.|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|protection|0666|12|2|RF$||{0|}}
{30001002|XXparameter|mode|0x0001|1|2|FH$|modes of access|{0|}}
{30001002|XXparameter|Layout|@28|2|RF$||{0|}}
{30001002|XXparameter|mpcmodtime|1256159712|1|1|Hl|The last modification time of this component's template|{0|}}
{30001002|XXparameter|read_metadata||7|1|RFl||{0|}}
{30001002|XXparameter|key||19|2|RF$|Key specifier For Lookup File|{0|}}
{30001002|XXparameter|key.condition|mode lookup|3|15|P?|@{0|}}
{30001002|XXparameter|eme_dataset_location|@3|9|F|Place in the EME to create a dataset corresponding to this file.|{0|}}
}}@0|@0|0|0|0|0|0|0|@@@1|10|-1|@6|@1|-1|-1|}}
{2010203004|XXGoport|237|0|611|0|{@{30100001|XXparameter_set|@@@@{{30001002|XXparameter|metadata||7|8|RF=||{0|}}
}}@0|0|0|0|read|0.0|@@@2160|0|}}
{2010501005|XXGpvertex|238|0|614|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|key|\{rsrc_itm_id\}|3|2|$|@{0|}}
{30001002|XXparameter|transform|out :: join(in0, in1) =
begin
out.arnge_id:: in0.arnge_id;
out.post_addr_id :: in1.post_addr_id;
end;|3|1|l|@{0|}}
{30001002|XXparameter|join_type|Explicit|3|1|l|@{0|}}
{30001002|XXparameter|record_required1|True|3|1|l|@{0|}}
{30001002|XXparameter|out_metadata|record
decimal("\\x01") arnge_id; /* INTEGER NOT NULL*/
decimal("\\x01") post_addr_id = NULL(""); /* INTEGER*/
string(1) newline = "\\n";
end|3|1|l|@{0|}}
{30001002|XXparameter|error0_metadata|string('\\n')|3|1|l|@{0|}}
{30001002|XXparameter|error1_metadata|string('\\n')|3|1|l|@{0|}}
{30001002|XXparameter|log_metadata|record string("\|") node, timestamp, component, subcomponent, event_type; string("\|\\n") event_text; end|3|1|l|@{0|}}
This is one of the Input_File and the part highlighted in red is of another file(file name is JOIN FYI). I dont think that this a prob for further analysis.
---------- Post updated at 02:07 PM ---------- Previous update was at 02:06 PM ----------
What i mean is:
Code:
Code:
{2010503005|XXGfvertex|234|0|604|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|Layout|$[[record kind 85 subkind 0 parts [vector _interp_("mfile:$\{INF_ENTRPRSDWUNFYRETLCRED_MFS\}/m_cdp2_uedw_t_post_addr_uld.dat", "dollar_substitution")]]]|3|9||@{0|}}
{30001002|XXparameter|read_metadata|$\{INF_ENTRPRSDWUNFYRETLCRED_DML\}/cdp2_uedw_t_post_addr.dml|3|2|f$|@{0|}}
{30001002|XXparameter|key|\{post_addr_id\}|3|2|$|@{0|}}
{30001002|XXparameter|!prototype_path|E:\\program files\\Ab Initio\\Ab Initio GDE 1.15.11.1\\Components\\Datasets\\Input_File.mdc|3|2|Pw$|@{0|}}
{30001002|XXparameter|eme_dataset_location|$\{_Projects_SunTrust_inf_inf_EntrprsDwUnfyRetlCred\}/data/mfs/main/m_cdp2_uedw_t_post_addr_uld.dat|3|9||@{0|}}
}}@1|@625666|493684|648000|514000|56000|50000|38410|m_cdp2_ uedw_t_post_addr_uld.dat|a8sc20||1|10|0||6||33025|-1|-1|}}
{2010203004|XXGoport|235|0|606|0|{@{}@703000|534000|11000|11000|read|0.0|@@@2160|0|}}
{2010503005|XXGfvertex|236|0|609|0|{Represents one file, many files, or a multifile as an input to your graph.|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|protection|0666|12|2|RF$||{0|}}
{30001002|XXparameter|mode|0x0001|1|2|FH$|modes of access|{0|}}
{30001002|XXparameter|Layout|@28|2|RF$||{0|}}
{30001002|XXparameter|mpcmodtime|1256159712|1|1|Hl|The last modification time of this component's template|{0|}}
{30001002|XXparameter|read_metadata||7|1|RFl||{0|}}
{30001002|XXparameter|key||19|2|RF$|Key specifier For Lookup File|{0|}}
{30001002|XXparameter|key.condition|mode lookup|3|15|P?|@{0|}}
{30001002|XXparameter|eme_dataset_location|@3|9|F|Place in the EME to create a dataset corresponding to this file.|{0|}}
}}@0|@0|0|0|0|0|0|0|@@@1|10|-1|@6|@1|-1|-1|}}
{2010203004|XXGoport|237|0|611|0|{@{30100001|XXparameter_set|@@@@{{30001002|XXparameter|metadata||7|8|RF=||{0|}}
}}@0|0|0|0|read|0.0|@@@2160|0|}}
{2010501005|XXGpvertex|238|0|614|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|key|\{rsrc_itm_id\}|3|2|$|@{0|}}
{30001002|XXparameter|transform|out :: join(in0, in1) =
begin
out.arnge_id:: in0.arnge_id;
out.post_addr_id :: in1.post_addr_id;
end;|3|1|l|@{0|}}
{30001002|XXparameter|join_type|Explicit|3|1|l|@{0|}}
{30001002|XXparameter|record_required1|True|3|1|l|@{0|}}
{30001002|XXparameter|out_metadata|record
decimal("\\x01") arnge_id; /* INTEGER NOT NULL*/
decimal("\\x01") post_addr_id = NULL(""); /* INTEGER*/
string(1) newline = "\\n";
end|3|1|l|@{0|}}
{30001002|XXparameter|error0_metadata|string('\\n')|3|1|l|@{0|}}
{30001002|XXparameter|error1_metadata|string('\\n')|3|1|l|@{0|}}
{30001002|XXparameter|log_metadata|record string("\|") node, timestamp, component, subcomponent, event_type; string("\|\\n") event_text; end|3|1|l|@{0|}}
This is one of the Input_File and the part highlighted in red is of another file(file name is JOIN FYI). I dont think that this a prob for further analysis.
It looks like early 80's cricket match where WI spacers balling on flat bouncy track
Jokes apart..
Please give me some more details..
1) How many Total files you have?
2) How many input files, and others..? And in which file we need to get what or we need to get from all the files..?
3) 2844000 first line from input file # didn't get how you got that. Give one line showing that number.
4) Below just need to check the line containing XXGflow and 2844000 from which files?
Code:
{2010210004|XXGflow|59|0|117|0|{@{}@384|.5|.5|{12|239000|2844000|259000|2844000|483000|2844000|483000|2824000|708000|2824000|729000|2824000|}39548|529|}}
5) Below just need to check the line containing XXGiport and 2819000 from which files?
Now i have total 101 files but I want this to be done for n number of graphs. So please dont restrict on number of files generated from a particular big file(Graph) So Ques1 is cleared
Is this ok??? so number of files varies form graph to graph. So i need a generalised code which performs my analysis one by one graph
So Ques2 is cleared
This is the Input_File i have given u:
Code:
Code:
{2010503005|XXGfvertex|1259|0|3869|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|Layout|$[[record kind 85 subkind 0 parts [vector _interp_("mfile:$\{INF_ENTRPRSDWUNFYRETLCRED_MFS\}/m_cdp2_uedw_t_arnge_to_arnge_uld.dat", "dollar_substitution")]]]|3|9||@{0|}}
{30001002|XXparameter|read_metadata|$\{INF_ENTRPRSDWUNFYRETLCRED_DML\}/cdp2_uedw_t_arnge_to_arnge.dml|3|2|f$|@{0|}}
{30001002|XXparameter|!prototype_path|C:\\Program Files\\Ab Initio\\Ab Initio GDE 1_14_5\\Components\\Datasets\\Input_File.mdc|3|2|Pw$|@{0|}}
{30001002|XXparameter|eme_dataset_location|$\{_Projects_SunTrust_inf_inf_EntrprsDwUnfyRetlCred\}/data/mfs/main/m_cdp2_uedw_t_arnge_to_arnge_uld.dat|3|9||@{0|}}
}}@1|@151424|2797588|173000|2818000|56000|50000|39502|m_cdp2_uedw_t_arnge_to_arnge_uld.dat|M K Muralidhar||1|100|-1||6||33537|-1|-1|}}
{2010203004|XXGoport|1260|0|3871|0|{@{}@228000|2839000|11000|11000|read|0.0|@@@2160|0|}}
{2010503005|XXGfvertex|1261|0|3874|0|{Represents one file, many files, or a multifile as an input to your graph.|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|protection|0666|12|2|RF$||{0|}}
{30001002|XXparameter|mode|0x0001|1|2|FH$|modes of access|{0|}}
{30001002|XXparameter|Layout|@28|2|RF$||{0|}}
{30001002|XXparameter|read_metadata||7|1|RFl||{0|}}
{30001002|XXparameter|mpcmodtime|1138303912|1|1|Hl|The last modification time of this component's template|{0|}}
{30001002|XXparameter|eme_dataset_location|@3|9|F|Place in the EME to create a dataset corresponding to this file.|{0|}}
}}@0|@0|0|0|0|0|0|0|@@@1|10|-1|@6|@1|-1|-1|}}
{2010203004|XXGoport|1262|0|3876|0|{@{30100001|XXparameter_set|@@@@{{30001002|XXparameter|metadata||7|8|RF=||{0|}}
}}@0|0|0|0|read|0.0|@@@2160|0|}}
{2010501005|XXGpvertex|1263|0|3885|0|{|{30100001|XXparameter_set|@@@@{{30001002|XXparameter|transform0|$AI_XFR/cdp2_rdm_dt_core_cnsum_arnge_dim_xfm_rfmt_orig_fico_scor_val.xfr|3|2|f$|@{0|}}
{30001002|XXparameter|out0_metadata|$AI_DML/cdp2_rdm_dt_core_cnsum_arnge_dim_xfm_rfmt_orig_fico_scor_val.dml|3|2|f$|@{0|}}
{30001002|XXparameter|error0_metadata|string('\\n')|3|1|l|@{0|}}
{30001002|XXparameter|log_metadata|record string("\|") node, timestamp, component, subcomponent, event_type; string("\|\\n") event_text; end|3|1|l|@{0|}}
Now from the above line did u get: 2839000
from every Input_File We can find the same line but with different numbers. In the sense
Code:
Code:
{2010203004|XXGoport|1260|0|3871|0|
This one we can have as a key word. Now frm every Input_File we will get one num then add 5000 to this number
2839000+5000 = 2844000
So Ques3 is cleared
As i already mentioned I have one file with me with the name XXGFlow and it contains the flow of components from one component to another(one file to another) and see my previous post for the file XXGFlow.
So Ques4 is cleared
My main moto at this stage is to get the flow of componet by component(file by file) starting from input_File to Output_File and need respective flow in respective Input_File'number'_f(i mean respective file name_f).
We have the data looks like below in a log file.
I want to generat files based on the string between two hash(#) symbol like below
Source:
#ext1#test1.tale2 drop
#ext1#test11.tale21 drop
#ext1#test123.tale21 drop
#ext2#test1.tale21 drop
#ext2#test12.tale21 drop
#ext3#test11.tale21 drop... (5 Replies)
Hi All,
I have the sales_data.csv file in the directory as below.
SDDCCR; SOM ; MD6546474777 ;05-JAN-16
ABC ; KIRAN ; CB789 ;04-JAN-16
ABC ; RAMANA; KS566767477747 ;06-JAN-16
ABC ; KAMESH; A33535335 ;04-JAN-16
SDDCCR; DINESH; GD6674474747 ;08-JAN-16... (4 Replies)
Hi Friends ,
Please guide me with the code to extract multiple files from one file .
The File Looks like ( Suppose a file has 2 tables list ,column length may vary )
H..- > File Header....
H....- >Table 1 Header....
D....- > Table 1 Data....
T....- >Table 1 Trailer....
H..-> Table 2... (1 Reply)
Dear All,
I have two files but want to extract data from one based on another... can you please help me
file 1
David
Tom
Ellen
and file 2
David|0010|testnamez|resultsz
David|0004|testnamex|resultsx
Tom|0010|testnamez|resultsz
Tom|0004|testnamex|resultsx
Ellen|0010|testnamez|resultsz... (12 Replies)
I will simplify the explaination a bit, I need to parse through a 87m file -
I have a single text file in the form of :
<NAME>house........
SOMETEXT
SOMETEXT
SOMETEXT
.
.
.
.
</script>
MORETEXT
MORETEXT
.
.
. (6 Replies)
Hi Friends,
Newbie to shell scripting. Currently i have used the below to sort data based on filenames and datestamp
$ printf '%s\n' *.dat* | sort -t. -k3,4
filename_1.dat.20120430.Z
filename_2.dat.20120430.Z
filename_3.dat.20120430.Z
filename_1.dat.20120501.Z
filename_2.dat.20120501.Z... (1 Reply)
Hi all,
I am new to scripting and I have a requirement
we have source file as
HEADER 01.10.2010 14:32:37 NAYA
TA0022
TA0000
20000001;20060612;99991231;K4;02;3
20000008;20080624;99991231;K4;02;3
20000026;19840724;99991231;KK;01;3
20000027;19840724;99991231;KK;01;3... (6 Replies)
I have a file as:
I/P File:
Ground Car 2009
Lib 2008
Lib 2003
Ground Car 2009
Ground Car 2003
Car 2005
Car 2003
Car 2005
Sita 2900 2006
Car 2007
I have to split the file into two: - one for names and second for years.
O/p1 (Names):
Ground Car (3 Replies)
I need to write a shell script for below scenario
My input file has data in format:
qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43
qwerty0101CFG 12345... (19 Replies)
I need to split a file based on certain context inside the file. Is there a unix command that can do this? I have looked into split and csplit but it does not seem like those would work because I need to split this file based on certain text. The file has multiple records and I need to split this... (1 Reply)
10-10-2012
Working fine