Comments on: Computing connected graph components via SQL

By: online stock trading advice

online stock trading advice — Mon, 11 Jan 2010 04:36:41 +0000

Hey, I found your blog while searching on Google your post looks very interesting for me. I will add a backlink and bookmark your site. Keep up the good work!

I’m Out! :)

By: david

david — Mon, 24 Nov 2008 21:36:18 +0000

Well, there are graph databases. We’re just not using one for various reasons. :-) (Although most of the existing graph databases don’t look like they’d make operations like “find all connected components” particularly easy)

By: jherber

jherber — Mon, 24 Nov 2008 18:30:00 +0000

sorry - explain and example read the same in haste ;)

i foresee ACID’able subgraphs in our 64bit, large memory future, or at the very least a specific case of stm for graphs.

By: david

david — Mon, 24 Nov 2008 13:42:47 +0000

Yes, I know it will. That’s why I said I’d have to take a look at the explain again. :-) I just wasn’t in front of a database at the time. And explain confirms that it uses the index rather than a table scan.

We could probably keep the graph in memory but I’m not really convinced it’s a good idea, particularly given that we can run this as a once a day process. Maybe at some point it will become a better idea, but for now this seems preferable.

By: jherber

jherber — Mon, 24 Nov 2008 13:34:24 +0000

yah, that’s a fairly large set of object to represent in ruby - especially if you are on a fixed slice. i’m pretty sure the linked-in guys keep their graph in memory, of course they are on the jvm. “explain” command will tell you how mysql plans to process that query. http://dev.mysql.com/doc/refman/5.0/en/explain.html

By: david

david — Mon, 24 Nov 2008 09:34:04 +0000

Well the problem isn’t the performance of the in memory solution so much as the fact that it’s in memory. The graph in question is really on the largish side (I mean sure, it’s only a few tens of thousands of nodes at the maximum in the data sets I’m running with at the moment, but that will easily bump up an order of magnitude or three in a production system). Loading all of that into memory is asking for trouble.

I don’t think the select min(component2) creates a full table scan as long as component2 is sensibly indexed. I’d have to take a look at the explain to be sure though. I didn’t talk about indices in this article, but the example code I linked to should have the right ones.

By: jherber

jherber — Mon, 24 Nov 2008 05:47:11 +0000

the underlying concern of storing the graph and the merge results (ie. the communicating groups) in the database can still be satisfied while performing the actual merge (transitive entailment) outside the db with an in-memory solution. it should be blazing fast ( groups * log (

By: david

david — Sun, 23 Nov 2008 16:05:01 +0000

No, the idea is that we want to find communicating groups.

E.g. suppose A talks to B and B talks to C, but none of them talk to D, even though D talks to E (this isn’t actually how we use this, but it illustrates the point). We want to find the two groups

{A, B, C} and {D, E}.

i.e. despite the fact that A never talks to C directly, C is part of the group of people with which A communicates. However there are no communications reaching from A to D or E, so D and E are in a different group.

What this algorithm does is it assigns a unique number to each person such that people with the same number are in the same group of communications.

By: Slow

Slow — Fri, 21 Nov 2008 19:22:21 +0000

I am a bit slow and am not 100% sure of the goal. Is the idea to consider all nodes with the same external connections (i.e. to everyone but each other) as the same node?

By: david

david — Thu, 20 Nov 2008 08:34:24 +0000

Thanks Phillip. I’m aware of the book and keep meaning to read it, but I don’t think it’s actually relevant to this particular problem: It focuses primarily on directed acyclic graphs, whileas this is an undirected cyclic one.