Transitive closure and recursive Datalog implemented on clusters

Afrati, FN; Ullman, JD

dc.contributor.author	Afrati, FN	en
dc.contributor.author	Ullman, JD	en
dc.date.accessioned	2014-03-01T02:54:03Z
dc.date.available	2014-03-01T02:54:03Z
dc.date.issued	2012	en
dc.identifier.uri	https://dspace.lib.ntua.gr/xmlui/handle/123456789/36564
dc.subject	Datalog	en
dc.subject	map-reduce	en
dc.subject	polynomial fringe property	en
dc.subject	recursion	en
dc.subject	transitive closure	en
dc.subject.other	Binary predicates	en
dc.subject.other	Chain rules	en
dc.subject.other	Computing clusters	en
dc.subject.other	Cost saving	en
dc.subject.other	Datalog	en
dc.subject.other	Datalog programs	en
dc.subject.other	Inter processor communication	en
dc.subject.other	Linear algorithms	en
dc.subject.other	Map-reduce	en
dc.subject.other	Reachability problem	en
dc.subject.other	Recursions	en
dc.subject.other	Recursive algorithms	en
dc.subject.other	Small files	en
dc.subject.other	Transitive closure	en
dc.subject.other	Unique decomposition	en
dc.subject.other	Database systems	en
dc.subject.other	Clustering algorithms	en
dc.title	Transitive closure and recursive Datalog implemented on clusters	en
heal.type	conferenceItem	en
heal.identifier.primary	10.1145/2247596.2247613	en
heal.identifier.secondary	http://dx.doi.org/10.1145/2247596.2247613	en
heal.publicationDate	2012	en
heal.abstract	Implementing recursive algorithms on computing clusters presents a number of new challenges. In particular, we consider the endgame problem: later rounds of a recursion often transfer only small amounts of data, causing high overhead for interprocessor communication. One way to deal with the endgame problem is to use an algorithm that reduces the number of rounds of the recursion. Especially, in an application like transitive closure (""TC"") there are several recursive-doubling algorithms that use a logarithmic, rather than linear, number of rounds. Unfortunately, recursive-doubling algorithms can deduce many more facts than the linear TC algorithms, which could negate the cost savings from the elimination of the overhead due to the proliferation of small files. We are thus led to consider TC algorithms that, like the linear algorithms, have the unique decomposition property that assures paths are discovered only once. We find that many such algorithms exist, and we show that they are incomparable, in that any of them could prove best on some data - - even lower in cost than the linear algorithms in some cases. The recursive-doubling approach to TC extends to other recursions as well. However, it is not acceptable to reduce the number of rounds at the expense of a major increase in the number of facts that are deduced. In this paper, we prove it is possible to implement any Datalog program of right-linear chain rules with a logarithmic number of rounds and no order-of-magnitude increase in the number of facts deduced. On the other hand, there are linear recursions for which the two goals of reducing the number of rounds and maintaining the total number of deduced facts cannot be met simultaneously. We show that the reachability problem cannot be solved in logarithmic rounds without using a binary predicate, thus squaring the number of potential facts to be deduced. We also show that the samegeneration recursion cannot be solved in logarithmic rounds without using a predicate of arity three. © 2012 ACM.	en
heal.journalName	ACM International Conference Proceeding Series	en
dc.identifier.doi	10.1145/2247596.2247613	en
dc.identifier.spage	132	en
dc.identifier.epage	143	en