|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Making Large Connection nodes for Graph
Post 302331161 by anjas on Saturday 4th of July 2009 05:30:00 AM
07-04-2009
Making Large Connection nodes for Graph
Hi power user,
Basically, this thread is a continuation of the previous one
:
https://www.unix.com/shell-programmin...#post302326483
However, I'm going to explain it again.
I have this following data:
file1
aa A
aa B
aa C
bb X
bb Y
bb Z
cc O
cc P
cc Q
. .
. .
. .
. .
and I want to turn them into a connection nodes like this:
file2
A aa A
A aa B
A aa C
B aa C
B aa B
C aa C
X bb X
X bb Y
X bb Z
Y bb Z
Y bb Y
Z bb Z
. . .
. . .
. . .
. . .
I made this relation, to create a graph. The file have more than 6.000.000 lines.
For smaller files (100.000 lines), I have used this following script in the previous thread:
join -o 1.2 0 2.2 -1 1 -2 1 file1 file1 | nawk '!a[$3$2$1];{a[$1$2$3]++}'
join -o 1.2 0 2.2 -1 1 -2 1 file1 file1 | nawk '$1<$3{print;next}{print$3,$2,$1}' | sort -u
nawk '
NR==FNR { c = a[$1]; a[$1] = c?c" "$2:$2; next }
{ c = a[$1]
if (c) {
split(c,b)
for (k in b) {
p = $2<b[k]?$2" "$1" "b[k]:b[k]" "$1" "$2
if (!d[p]++) print p
}
}
}
' file1 file1
For small file, those three kind of scripts could create the network only in less than 10 minutes. However, for files with more than 6.000.000 lines, even after one days, there was no results at all
. Is there any faster way to do it?
Any suggestion, how to create file2 by using perl or awk? Tx
Basically, this thread is a continuation of the previous one
:https://www.unix.com/shell-programmin...#post302326483
However, I'm going to explain it again.
I have this following data:
file1
aa A
aa B
aa C
bb X
bb Y
bb Z
cc O
cc P
cc Q
. .
. .
. .
. .
and I want to turn them into a connection nodes like this:
file2
A aa A
A aa B
A aa C
B aa C
B aa B
C aa C
X bb X
X bb Y
X bb Z
Y bb Z
Y bb Y
Z bb Z
. . .
. . .
. . .
. . .
I made this relation, to create a graph. The file have more than 6.000.000 lines.
For smaller files (100.000 lines), I have used this following script in the previous thread:
join -o 1.2 0 2.2 -1 1 -2 1 file1 file1 | nawk '!a[$3$2$1];{a[$1$2$3]++}'
join -o 1.2 0 2.2 -1 1 -2 1 file1 file1 | nawk '$1<$3{print;next}{print$3,$2,$1}' | sort -u
nawk '
NR==FNR { c = a[$1]; a[$1] = c?c" "$2:$2; next }
{ c = a[$1]
if (c) {
split(c,b)
for (k in b) {
p = $2<b[k]?$2" "$1" "b[k]:b[k]" "$1" "$2
if (!d[p]++) print p
}
}
}
' file1 file1
For small file, those three kind of scripts could create the network only in less than 10 minutes. However, for files with more than 6.000.000 lines, even after one days, there was no results at all
. Is there any faster way to do it? Any suggestion, how to create file2 by using perl or awk? Tx
|
|
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
first off, i am new to unix so please bear with me. i was reading somewhere that if your i-nodes get critical that it can slow your network down. what are i-nodes and when do they become a critical number? this is what mine states:
/ (/dev/root ): 777058 blocks 569290 i-nodes... (4 Replies)
Discussion started by: djatwork
4 Replies
2. UNIX for Dummies Questions & Answers
how do you list all the nodes in unix
:confused: (3 Replies)
Discussion started by: kamisi
3 Replies
3. UNIX for Advanced & Expert Users
Does anyone know something about this? I have no idea what it means and how to do it. but if anyone can give me and explanation and also point me to a website, i'd really appreciate it (5 Replies)
Discussion started by: TRUEST
5 Replies
4. UNIX for Dummies Questions & Answers
Hi Friends,
I'mfacing a problem while doing ftp of a large file.The control session is getting closed after sometime.But data session transfers the file successfully even when the control seeion is lost.I need to make the control session available as long as data session is active.
How can i... (1 Reply)
Discussion started by: rprajendran
1 Replies
5. HP-UX
Hello Friends,
When i trying to transfer a huge amount of files via FTP to a HP-Unx server, I am getting an error "Connection Refused"...!
How can i avoid this error.
Regards,
Prakash K:b: (4 Replies)
Discussion started by: bullz26
4 Replies
6. Shell Programming and Scripting
Hi Power User,
I have this following data:
file1
aa A
aa B
aa C
bb X
bb Y
bb Z
cc O
cc P
cc Q
. .
. .
. .... (4 Replies)
Discussion started by: anjas
4 Replies
7. Solaris
Hi everyone,
I am hoping anyone of you could help me in this weird problem we have in 1 of our Solaris 10 servers. Lately, we have been having some ftp problems in this server. Though it can ping any server within the network, it seems that it can only ftp to a select few. For most servers, the... (4 Replies)
Discussion started by: labdakos
4 Replies
8. UNIX for Dummies Questions & Answers
Hi all. I have two nodes taken different places. They are connected together on a network. So, i have a service, it works on one of nodes and when the node is unavailable the service should will be launched on other node.
Solution: rhel cluster, keepalive, hearbeat...may be Carp
but what if... (2 Replies)
Discussion started by: Flomaster
2 Replies
9. Shell Programming and Scripting
so i have a large file that has many lines in it.
i want to make changes (replace certain things) that are in a range of lines, but NOT the whole file.
how do i do this?
say for instance, if i want to change all the occurrences of the word "ramamama" to "ram cow welcome" in lines 9333 to... (5 Replies)
Discussion started by: SkySmart
5 Replies
10. UNIX for Advanced & Expert Users
How keep running a program n an another computer via a connection ssh when the connection is closed?
Hi everybody,
I am running a program on a supercomputer via my personal computer through a ssh connection. My program take more than a day to run, so when I left work with my PC I stop the connection with the supercomputer and the program stop.
I am wondering if someone know how I can manage... (2 Replies)
Discussion started by: TomTomGre
2 Replies
LEARN ABOUT DEBIAN
rdf::trine::graph
RDF::Trine::Graph(3pm) User Contributed Perl Documentation RDF::Trine::Graph(3pm) NAME
RDF::Trine::Graph - Materialized RDF Graphs for testing isomorphism VERSION
This document describes RDF::Trine::Graph version 1.000 SYNOPSIS
use RDF::Trine::Graph; my $a = RDF::Trine::Graph->new( $model_a ); my $b = RDF::Trine::Graph->new( $model_b ); print "graphs are " . ($a->equals( $b ) ? "the same" : "different"); DESCRIPTION
RDF::Trine::Graph provdes a mechanism for testing graph isomorphism based on graph triples from either a RDF::Trine::Model or a RDF::Trine::Iterator. Isomorphism testing requires materializing all of a graph's triples in memory, and so should be used carefully in situations with large graphs. METHODS
"new ( $model )" "new ( $iterator )" Returns a new graph from the given RDF::Trine::Model or RDF::Trine::Iterator::Graph object. "equals ( $graph )" Returns true if the invocant and $graph represent two equal RDF graphs (e.g. there exists a bijection between the RDF statements of the invocant and $graph). "is_subgraph_of ( $graph )" Returns true if the invocant is a subgraph of $graph. (i.e. there exists an injection of RDF statements from the invocant to $graph.) "injection_map ( $graph )" If the invocant is a subgraph of $graph, returns a mapping of blank node identifiers from the invocant graph to $graph as a hashref. Otherwise returns false. The solution is not always unique; where there exist multiple solutions, the solution returned is arbitrary. "split_blank_statements" Returns two array refs, containing triples with blank nodes and triples without any blank nodes, respectively. "get_statements" Returns a RDF::Trine::Iterator::Graph object for the statements in this graph. "error" Returns an error string explaining the last failed "equal" call. BUGS
Please report any bugs or feature requests to through the GitHub web interface at <https://github.com/kasei/perlrdf/issues>. AUTHOR
Gregory Todd Williams "<gwilliams@cpan.org>" COPYRIGHT
Copyright (c) 2006-2012 Gregory Todd Williams. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.14.2 2012-06-29 RDF::Trine::Graph(3pm)