• Static type checking is not good enough. When declaring a query, there is implicit type information attached. For example, the query select fields ("a" of boolean, "b" of varchar) from "table" gives information about what type fields "a" and "b" have ("boolean" and "varchar" are instances of classes that define the native Scala type for this field as type members). However, the query in the above code does not contain the type information in a form exploitable by the compiler. It would need to be given as type parameters, but the number of fields, and therefore of types, is arbitrary, and Scala does not support classes with an arbitrary number of type parameters. Tuple or Function classes try to simulate it by declaring multiple such classes (Tuple1, Tuple2, etc.) but with a relation, the number of fields can be quite large and declaring, say, 100 classes for all different sizes would be very impractical. Martin proposed it might be an idea to support a mechanism to generate on request a relation (or tuple or function) of the right size, but this is not planned and would not be trivial to do. This problem of type safety will also be true for ScalaDALL, so this remains very much an open problem.
• Automatic type conversions using views is not broad enough. Bug #449 (or a variant of it) prevents the conversion of all the required values. In particular, it is not possible to view a value of type integer (SQL) as a value of type long (Scala) for example, even though it would be perfectly legal. I hope it will be possible to improve this in NSC with the new view system (based on "implicit"). Another sad thing is that views are only one level deep. Of course, multi-level views would be quite a different problem altogether: Cyclicality for one thing would probably be very difficult to manage, and I am not even sure the problem is solvable. It would also make development very unpredictable, but quite a bit more powerful. In practice, this means that if a field is a value, and a value an int, for example, a field is not an int except if this is made explicit by adding a direct view from one to the other.

ScalaDBC, the generic database library for Scala is more or less working. Well, actually it is working as far as I can tell, but I haven't put it through any serious testing (special cases etc.). There are two things that need to be finished though:

1. Automatic type conversion of database types to native Scala types using views. Bug #449 is a problem here, but I can't quite understand how and why.
2. A factory (and I mean a real hard-core factory) to write statement (queries) AST in a nice way. This is already partially working (one can write "selectBag fields ("a", "b" as "c") from ("a" join "b")") but it isn't supporting quite enough of the SQL standard to be useful yet. Currently, I'm stuck with a bug that might be a Scala bug, but I still have to track it down to be sure.

ScalaDALL is the name I am giving to the database library using for-comprehensions and specific optimisations that is the next task after ScalaDBC. I have started some exploratory work for this as I was really getting frustrated with the bugs that where hindering the advance of ScalaDBC. But there isn't anything really working yet.

Otherwise, I also did a little poster that you can find next to my office's door.

The current version of the library is useable, but there are at least two things that I think should be improved if one wants to use it in real applications:

• Allow more user control on the way the wait time before a retry is calculated — as this is key to the performance one will get. This could be done by supporting "pluggable wait policy" for example.
• Implement an even more optimistic policy that does not lock variables at all, but instead calculated the entire locked section as if it was alone, and only once this is done, just before committing, test whether some other thread was messing around with the data.

For them, a database interface library or language is interesting if one of the following behaviours are better than what currently exists. But in all cases, a decrease in performance is a show-stopper.

• Complexity of code is demonstrably reduced. But cleaner code with the same complexity is a no-show. Of course, in that case, they are not really interested in the problem itself but rather the result.
• Performance (in an arbitrary definition) somehow increases. For large systems, and in particular distributed systems, performance becomes an overwhelming problem and offering something that solves or mitigates this would be very useful.
• And a particular instance of performance that interest them particularly: make middle-ware programs simpler. That means that some of the processing done currently in the middle-ware program at run-time might be done at compile time by looking at the queries. An example of such processing is to find the intersection between queries (used for efficient locking of data). Tackling this problem as such is very specialised and not really in my field though.

I have also started implementing the library for the database access that will be optimised à la SLinks. Of course, it is quite uninteresting by itself without the changes to the compiler to support its optimisation. But there is a start for everything. Plenty of work remaining for this though.

Consider the following test environment. 100 threads all execute one transaction that will swap the value of two variables using one support variable (s:=x;x:=y;y:=s). The three used variables for every thread are randomly selected from a pool of 300 variables. That means that the different transactions share some data, but not too much. There are typically 70 transactions that share at least one variable with another transaction. During the execution of the swap function, the threads will each sleep for a randomly chosen time between 0 and 300 ms. This is of course only somewhat realistic (it might correspond to waiting for input or example) but I believe it is also a rather good indication of how well it might work in a multi-processor environment.

In this setting, the policy that locks down the whole system when doing a transaction takes a time around 16'000 ms in average. This is what one would expect as 100 threads times 150 ms (the average time a swap waits) is 15'000 ms plus the locking overhead. On the other hand, the optimistic policy is much faster: around 1000 ms (15 times less). The number of aborts are quite high, but it doesn't seem to be such a problem when compared to the other policy. I did not measure directly the average number of aborts per thread, but I tried to estimate it differently: If the number of aborts allowed for a given transaction is limited to 50, 2-5 percents of transactions will fail (that is, they would have aborted more than 50 times). With a abort limit of 100, the failure of a transaction is too rare to be detected (significantly less than 1%).

Of course, in unfavourable cases, the optimistic algorithm becomes very bad. The good news however is that the optimistic algorithm remains at least as good as the pessimistic one ... until it starts to completely wreak havoc on the system: since the abort mechanism is recursive with no tail call, it starts using up huge amounts of memory and the threads fail because they run out of memory. I will have to improve this.

val x = new atomic.Variable[Int];
val y = new atomic.Variable[Int];
val z = new atomic.Variable[Int];
x.value = 1; y.value = 2;
(for (
  val a <- x.get();
  val _ <- z.put(a);
  val b <- y.get();
  val _ <- x.put(b);
  val c <- z.get();
  val d <- y.put(c);
) yield d) run

The problem now is that the for loop as a monad is something that I find very difficult to get an intuition about. I have a very simple version that locks down the entire program when evaluating the atomic bloc, but this is of course hopelessly inefficient. But when I try to improve from this basic version, I am having troubles grasping how things interact around this comprehension/monad, and getting the right intuition on how to do it. Please, read on for a small explanation on the locking mechanisms I intend to add.

• Locking variables. With this method, the locking would be done on the variable itself. A variable must be locked as soon as it is read or written to guarantee a consistent state during the entire transaction. When a variable that is already locked by another transaction is reached, the transaction is aborted and restarted to prevent deadlocks. Restarting means reverting all variables that have been changed to their pre-change value. The easiest here is to write all changes to a buffer in the variable and commit the buffers when the transaction is completed or leave them alone when the transaction is aborted.
• Versioning variables. When a variable is read, its version number is remembered. When a variable is written, its version number is incremented and a function that will undo the write is generated (this is not easy). At the end, the used variables are locked and their current version numbers are compared with the version numbers remembered. If they concur, the transaction is confirmed. If they don't, it means another thread has been messing with the variables; the transaction is restarted after the undo functions are applied to the variables.

Very shortly: I started my PhD at the laboratoire des méthodes de programmation (LAMP) at the EPFL in Lausanne under the supervision of Prof. Martin Odersky on the fourth of April 2005. Currently, I intend to work on how to support relational database querying in a completely transparent (that is without SQL) yet efficient (that is with SQL) way in modern programming languages. The project might quickly extend to include more general querying for a variety of data repository systems (who said XQuery?). The modern programming language that will be used as a reference here is the LAMP's very own language, Scala.