Scala random notes, mostly on back-translating to Java

March 2017
last update:

My need for Scala understanding is hardly out of curiosity. The more I look at it, the less I am thrilled to get on any bandwagon. This also characterizes my attitude generally about functional programming. It's popular to rave about it; I see it as more difficult to grok when you inherit someone's code and generally pointless as I don't see it living up to its claimed goals of immutability and readability. The complained-of Java boiler plate or verbosity is somewhat resolved by better modularity, methods, superclassing, etc. However, just as heavily @-annotated frameworks annoy me because I can't tell what's going on, I'd rather crawl through a pile of Java code that's an honest portrayal of reality than guess at someone else's mind.

I once was called upon to back-translate some Scala code to Java, which I did willingly since what that code accomplished had high value to the project I was on. These are my notes on back-translating and on Scala topics that I needed to explore in order to understand what I was doing. These notes aren't exhaustive at all since I gained more of a feeling for how to translate than clear understanding from which I could scratch out sure methodologies. Sorry.

A growing worry I think I have confirmed is that Scala can create inter-class relationships that cannot be easily back-translated directly into Java. The code must be rewritten a different way. I don't see this as superiority in Scala, just as a salient difference. It's like this when translating between any two programming (or even natural, spoken) languages.

General notes

Translating Scala code to Java is a little like shopping for furniture from a store to decorate an empty living room based on what you're looking at in a Picasso painting.

There are no Scala-to-Java translators. Indeed, it may be impossible ultimately to write such a thing. There are some Java-to-Scala translation helps. So... what? UML?

I looked into a UML rendering of Scala code. Sadly, the first (and last, so far) time I used UML in IntelliJ, I did not take notes on how to do it. It appears, however, that there is no plug-in for Scala UML for IntelliJ (while there is for Java). There is a project of Scala code that generates DOT at the command line, but you have to use Graphviz, a bit of open-source software that would take quite a bit of effort to figure out how to use. I didn't have time to screw around with all of that. There is an Eclipse plug-in, but it's for generating Scala stubs based on a given model and not for generating UML diagrams from existing Scala code.

So, it appears that there is no help for rendering Scala code in any representational format.

It's generally good to take time out to catch back up on Java 8 constructs because they're (sort of) close to the Scala ones. This is my advice.


Sloppy Scala programming

I had opportunity to engage in a cordial conversation with some Scala guys at the UJUG. I asked them about some troubles I was having and described classes in which code statements were interwoven with method definitions. In Java, of course, that's syntactically impossible. They confirmed that such statements would be part of the initialization of the class in particular. Their opinion was also that this was not good Scala (to toss them around instead of grouping them all near the top of the class.


Scala traits

These are similar to Java's interface. They share interfaces and fields between classes. Classes and objects can extend traits, but like interfaces and abstract classes, traits cannot be instantiated (nor have parameters).


Scala's case class

This is for modeling classes containing immutable data. They are used especially for pattern matching. They have an implicit apply() method such that you construct new additions to them as if constructing an instance of the class:

case class Book( isbn: String )

val frankenstein = Book( "978-0486282114" )

The -> operator

The -> (or goes-to) operation is shorthand for associating a key (left-hand side of the operator) with a value (right-hand side). This is instead of using the constructor or the put() method of the map class.

=> is syntactic sugar for creating instances of functions. In Scala, every function is an instance of a class. For example,

Int => String

...is equivalent to

Function1[ Int, String ]

So, this is a function that takes and argument of type Int and returns a String.

One stackoverflow responder made a suggestion that (and just how accurate it is I don't know, but I sort of see his point) a simplified way of thinking about this "operator" is to think of what you're looking at as:

LEFT  =>  RIGHT

...means, take (what's on the) LEFT and instead do (what's on the) RIGHT." Or, "given LEFT, do RIGHT."


Scala's _ (underscore) operator

When I'm finished looking through these, I'm just as confused if not more. One read that's sometimes useful is Scala _ [underscore] magic.

import xyz._ Wild card—all of xyz is imported
import xyz.{ Predef => _, _ } Exception, everything except Predef
def f[ M[ _ ] ] Higher kinded type parameter
def f( m: M[ _ ] ) Existential type
_ + _ Anonymous function placeholder parameter
m _ Eta expansion of method into method value
m( _ ) Partial function application
_ => 5 Discarded parameter
case _ => Wild card pattern—matches anything
val ( a, _ ) = ( 1, 2 ) ibid
for ( _ <- 1 to 10 ) ibid
f( xs: _* ) Sequence xs is passed as multiple parameters to f( ys: T* )
case Seq( xs @ _* ) Identifier xs is bound to the whole matched sequence
var i: Int = _ Initialization to the default value
def abc_<>! An underscore must separate alphanumerics from symbols on identifiers
t._2 Part of a method name, such as tuple getters

Scala's ternary operator

Scala doesn't implement the C/Java ternary operation. However, it does offer:

if( condition ) true-result else false-result

Scala icons in IntelliJ IDEA


"Companion" objects

...may be found at the bottom (?) of Scala classes and are "analogous to Java having a class with static methods." At the bottom of Stratio's Lucene plug-in code for Cassandra, for instance, class IndexQueryHandler. The highlighted lines and following constitute this companion object:

/*
 * Copyright (C) 2014 Stratio (http://stratio.com)
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *         http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.stratio.cassandra.lucene
.
.
.
class IndexQueryHandler extends QueryHandler with Logging
{
  .
  .
  .
}

/** Companion object for [[IndexQueryHandler]]. */
object IndexQueryHandler
{
  val processResults: Method = classOf[SelectStatement].getDeclaredMethod( "processResults",
                                   classOf[PartitionIterator],
                                   classOf[QueryOptions],
                                   classOf[Int],
                                   classOf[Int] )
  processResults.setAccessible( true )

  /** Sets this query handler as the Cassandra CQL query handler, replacing the previous one. */
  def activate(): Unit =
  {
    this.synchronized
    {
      if( !ClientState.getCQLQueryHandler.isInstanceOf[IndexQueryHandler] )
      {
        try
        {
          val field = classOf[ClientState].getDeclaredField( "cqlQueryHandler" )
          field.setAccessible( true )
          val modifiersField = classOf[Field].getDeclaredField( "modifiers" )
          modifiersField.setAccessible( true )
          modifiersField.setInt( field, field.getModifiers & ~Modifier.FINAL )
          field.set( null, new IndexQueryHandler )
        }
        catch
        {
          case e:
            Exception => throw new IndexException( "Unable to set Lucene CQL query handler", e )
        }
      }
    }
  }
}

A good way to translate that companion object into Java is to make (in this case, the two methods) simply static:

.
.
.
public class OurQueryHandler implements Logging, QueryHandler
{
  .
  .
  .

  public static Method processResults()
  {
    try
    {
      Class< ? > clazz  = SelectStatement.class;
      Method     method = clazz.getDeclaredMethod( "processResults", PartitionIterator.class,
                                                     QueryOptions.class,
                                                     int.class, // (strange, but true!)
                                                     int.class );
      method.setAccessible( true );
      return method;
    }
    catch( NoSuchMethodException e )
    {
      throw new IndexException( "Unable to return reflection of method SelectStatement.processResults()", e );
    }
  }

  public static void activate()
  {
    synchronized( OurQueryHandler.class )
    {
      if( !( ClientState.getCQLQueryHandler() instanceof OurQueryHandler ) )
      {
        try
        {
          Class< ? > clazz          = ClientState.class;
          Field      methodField    = clazz.getDeclaredField( "cqlQueryHandler" );
          Field      modifiersField = Field.class.getDeclaredField( "modifiers" );
          methodField.setAccessible( true );
          modifiersField.setAccessible( true );
          modifiersField.setInt( methodField, methodField.getModifiers() & ~Modifier.FINAL );
          methodField.set( null, new OurQueryHandler() );
        }
        catch( Exception e )
        {
          throw new IndexException( "Unable to set our CQL handler", e );
        }
      }
    }
  }
}

Unit in Scala

Unit is analogous to void.


asScala

This allows you to take

List< String > javaStringList = Array.fromList( "this", "that", "and", "the", "other", "thing" );

...and turn this list into a Scala construct that will function using map() and other functionality:

val scalaStringList: List[String] = javaStringList.asScala

This permits, from this point on, the use of Scala constructs like:

scalaStringList.forEach{ s => ... }

Seq and List in Scala

The following may be a bit confusing...

In Java terms, Scala's Seq would be Java's List, and Scala's List would be Java's LinkedList. However, I have often rendered Scala's List as merely Java's List.

Note that Seq is a trait, which is equivalent to Java's interface, but with the equivalent of up-and-coming defender methods. Scala's List is an abstract class that is extended by Nil and ::, which are the concrete implementations of List.

So, where Java's List is an interface, Scala's List is an implementation.


Multiple .map() calls in initialization

Here's a construct by way of example. There's an instance variable, pools, in a class named TaskQueue. Once initialized, this variable points at a list of task-queue pools. This class is constructed with the number of threads to support. The interspersed comments below recount what's going on.

private val pools =
  // Start with a range from 1 to numThreads, which you can essentially
  // consider a Seq(1, 2, 3, ..., numThreads).
  ( 1 to numThreads )
    // Now, for each value in that range, pass it to this function and store
    // the results in another sequence. The function ignores the input value
    // (which is what the _ means before the arrow) and returns a
    // new ArrayBlockingQueue[Runnable]. After this step, we will have
    // a Seq[ArrayBlockingQueue[Runnable]], with a number of elements
    // equal to numThreads.
    .map( _ => new ArrayBlockingQueue[Runnable]( queuesSize, true ) )
    // Again, for each of these ArrayBlockingQueues, pass it to this
    // new function (captured in the q argument) and create a new Seq
    // of the results. In this case, it is a Seq[ThreadPoolExecutor]
    // where each of the ThreadPoolExecutor objects has been through the
    // initialization code below
    .map( q => new ThreadPoolExecutor( 1, 1, 1, DAYS, q,
                     new BasicThreadFactory.Builder()
                          .namingPattern( "indexer-%d" )
                          .build(),
                     ( task, executor ) =>
                          if( !executor.isShutdown )
                            executor.getQueue.put( task ) ) )
  // The final resulting type of pools should be Seq[ThreadPoolExecutor],
  // and it will have a number of element equal to numThreads.

In Java, this might be something like this:

List< ThreadPoolExecutor > pools = new ArrayList<>();

BasicThreadFactory threadFactoryBuilder = new BasicThreadFactory.Builder()
                                      .namingPattern( "indexer-%d" )
                                      .build();
// create a list...
List< Queue< Runnable > > queues = new ArrayList<>( numThreads );

// ...and fill it with new queues of queueSize and our default access policy...
while( numThreads-- > 0 )
  queues.add( new ArrayBlockingQueue<>( queueSize, DEFAULT_ACCESS_POLICY ) );

// fill the list of executors with initialized executors that use, each, a queue from above...
for( Queue< Runnable > queue : queues )
  pools.add( new ThreadPoolExecutor( CORE_POOL_SIZE, MAX_POOL_SIZE, KEEP_ALIVE_TIME, KEEP_ALIVE_FOR_DAY,
                                  ( BlockingQueue< Runnable > ) queue, threadFactoryBuilder ) );

Second example

The objective is to build a list of FsIndex.

private[this] val indexes: List[FsIndex] = partitions match
{
  case 1 =>
     List( new FsIndex(name, path, analyzer, refreshSeconds, ramBufferMB, maxMergeMB, maxCachedMB ) )
  case n if n > 1 =>
     val root = path.toFile.getAbsolutePath + File.separator
     ( 0 until n )
       .map(root + File.separator + _ )
       .map(Paths.get( _ ) )
       .map(new FsIndex( name, _, analyzer, refreshSeconds, ramBufferMB, maxMergeMB, maxCachedMB ) )
       .toList
  case _ =>
     throw new IndexException( s"The number of partitions should be strictly positive but found $partitions" )
}
if( numPartitions < 1 )
  throw new IndexException( "The number of partitions should be greater than 0" );

List< FsIndex > indices = new ArrayList<>();
final String    ROOT    = path.toFile().getAbsolutePath();

switch( numPartitions )
{
  case 1 :
   indices.add( new FsIndex( name, path, analyzer, refreshSeconds, ramBufferMb, maxMergeMb, maxCachedMb ) );
   break;
  default :
   // build a new index for each set of data we can muster here...
   for( int partition = 0; partition < numPartitions; partition++ )
   {
     indices.add( new FsIndex( name,
                         Paths.get( ROOT + File.separator + partition ),
                         analyzer, refreshSeconds, ramBufferMb, maxMergeMb, maxCachedMb ) );
   }
   break;
}

Operators /: (foldLeft()) and :\ (foldRight())

This is pretty crazy stuff. I read one post in which the guy tried very hard to elucidate this in this way:

Think of /: as a picture of a domino falling right into the first wagon of a train of other dominoes—the way many set them up to see them knock each other over. Every time the calculation-so-far makes contact with a new element in the train, the function is applied.

val nums = List( 4, 2, 9, 3, 1 )
val sum  = ( 0 /: nums ) ( _+_ )

      _ + _     _     _     _     _
     / / | |   | |   | |   | |   | |
    / /  | |   | |   | |   | |   | |
   /0/   |4|   |2|   |9|   |3|   |1|
  / /    | |   | |   | |   | |   | |
 / /     | |   | |   | |   | |   | |
        \____________nums___________/

            _ + _     _     _     _
           / / | |   | |   | |   | |
          / /  | |   | |   | |   | |
 0 + 4 = /4/   |2|   |9|   |3|   |1|
        / /    | |   | |   | |   | |
       / /     | |   | |   | |   | |

                  _ + _     _     _
                 / / | |   | |   | |
                / /  | |   | |   | |
 (0 + 4) + 2 = /6/   |9|   |3|   |1|
              / /    | |   | |   | |
             / /     | |   | |   | |

                        _ + _     _
                       / / | |   | |
                      / /  | |   | |
   (0 + 4 + 2) + 9 = 15/   |3|   |1|
                    / /    | |   | |
                   / /     | |   | |

                              _ + _
                             / / | |
                            / /  | |
     (0 + 4 + 2 + 9) + 3 = 18/   |1|
                          / /    | |
                         / /     | |

                                    _
                                   / /
                                  / /
       (0 + 4 + 2 + 9 + 3) + 1 = 19/
                                / /
                               / /

Another example

( 0L /: indices ) ( _ + _.getNumDocs )

In this example, what's done is the values of getNumDocs() as called on the various indices of a list, with this effect:

List< FsIndex > indices = [ ... ];
long count = 0;

for( FsIndex index : indices )
  count += index.getNumDocs();