Planet Primates

July 22, 2014



Scala graph and problems when declaring custom types

right now I have a package object which has several types declared to make my code a bit easier to read.

    package object FlowGraphType {
      type FlowEdgeType[NodeLabel] = LDiEdge[NodeLabel]
      type FlowGraph = Graph[NodeLabel, FlowEdgeType]

      type DominatorGraph = Graph[NodeLabel, FlowEdgeType]
      type DominatorEdgeType[NodeLabel] = DiEdge[NodeLabel]

      type NodeLabel = Long
      val FlowTaken = 'taken
      val FlowFallthrough = 'fallthrough


This works fine:

    val graph = Graph[NodeLabel, FlowEdgeType]((1L ~+> 2L)(FlowTaken), (2L ~+> 4L)(FlowTaken))

However, when I try to use the generic type to declare the graph, like below, I get the message not found: value FlowGraph

    val graph = new FlowGraph((1L ~+> 2L)(FlowTaken), (2L ~+> 4L)(FlowTaken))

When I try new like this, I get trait Graph is abstract; cannot be instantiated

    val graph = new FlowGraph((1L ~+> 2L)(FlowTaken), (2L ~+> 4L)(FlowTaken), (2L ~+> 3L)(FlowTaken), (3L ~+> 5L)(FlowTaken), (4L ~+> 5L)(FlowTaken), (5L ~+> 6L)(FlowTaken))

I'm a bit new to Scala, so decoding what exactly it going wrong here is a bit difficult for me. My first guess was that the apply method of Graph which is a factory isn't being called when the type is declared in place of the custom type.

Would love any insights into this. Thanks a bunch!

Edit So this allows me to actually use my type with the constructor:

      type FlowGraph = Graph[NodeLabel, FlowEdgeType]
      val FlowGraph = Graph[NodeLabel, FlowEdgeType]_

However, now I must type 'Seq' in order to get the list properly processed. Is there a way to omit the Seq and pass those values to the original varargs?

      val graph1 = FlowGraph(Seq((1L ~+> 2L)(FlowTaken)))

Edit 2 type FlowGraph = Graph[NodeLabel, FlowEdgeType] val FlowGraph = Graph[NodeLabel, FlowEdgeType]_ def FlowGraph(xs: FlowEdgeType[NodeLabel]) : FlowGraph = FlowGraph(Seq(xs: _))

by Steve H. at July 22, 2014 08:58 PM

Is there any comprehensive tutorial on how to use this kind of structural typing?

As seen here, map is defined

in a slightly funky way

For completeness' sake, here's the code:

implicit def FunctionFunctor[R] = new Functor[({type l[a] = R=>a})#l] {
  def map[A, B](fa: R => A)(f: A => B) : R => B = (x => f(fa(x)))

More concretely - new Functor[({type l[a] = R=>a})#l]

I do think I know what's going on, but can't say honestly that I fully understand the concept. And since there is no any hint, I can't event google the term (I simply don't know that to google). Does some tutorial (or review, or whatever) exist where this is explained with a better level of detalization? I'd appreciate even more is someone could explain it right here in the answers.

by cdshines at July 22, 2014 08:55 PM


Combinatory interpretation of lambda calculus

According to Peter Selinger, The Lambda Calculus is Algebraic (PDF). Early in this article he says:

The combinatory interpretation of the lambda calculus is known to be imperfect, because it does not satisfy the ξ-rule: under the interpretation, M = N does not imply λx.M = λx.N (Barendregt, 1984).


  • What kind of equivalence is meant here?
  • Given this definition of equivalence, what is a counter-example of the implication?

by Simon Shine at July 22, 2014 08:52 PM


Clojure Compojure Ring and https

I'm using Clojure (ring and compojure) to build a web app and I don't know where to start with https. I have a registration and login that will need to be secured, then once they're authenticated they'll need to stay in https.

I can't seem to find a good tutorial on setting up https in general or for a clojure app specifically.

I found this answer: How do you enable https and http->https redirects in ring / compojure

Does that mean I can write my compojure app as if there's no https and have nginx sit in front and take care of all that for me?

by user3866249 at July 22, 2014 08:44 PM

Can't resolve casbah as a sbt dependency

I'm getting the following error upon compile:

[error] (*:update) sbt.ResolveException: unresolved dependency: org.mongodb#casbah_2.11;2.7.4-SNAPSHOT: not found

With this build.sbt file:

name := """play-scala"""

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.11.1"

libraryDependencies ++= Seq(

libraryDependencies += "org.mongodb" %% "casbah" % "2.7.4-SNAPSHOT"

// For stable releases
resolvers += "Sonatype releases" at ""

// For SNAPSHOT releases
resolvers += "Sonatype snapshots" at ""

I'm using Typesafe Activator and have tried many combinations of Casbah versions (including releases, instead of snapshots) and resolvers. The installation instructions from the "official" MongoDB Casbah tutorial don't work either. Any ideas?

by Christopher Su at July 22, 2014 08:43 PM

How can I convert a to a string in Scala? [duplicate]

This question already has an answer here:

If I have a instance, how can I convert it to a string in Scala?

val reader = new
// How do I get a String out of reader?

by aknuds1 at July 22, 2014 08:38 PM

Return all the indexes of a particular substring

Is there a Scala library API method (and if not, an idiomatic way) to obtain a list of all the indexes for a substring (target) within a larger string (source)? I have tried to look through the ScalaDoc, but was not able to find anything obvious. There are SO many methods doing so many useful things, I am guessing I am just not submitting the right search terms.

For example, if I have a source string of "name:Yo,name:Jim,name:name,name:bozo" and I use a target string of "name:", I would like to get back a List[Int] of List(0, 8, 17, 27).

Here's my quick hack to resolve the problem:

def indexesOf(source: String, target: String, index: Int = 0, withinOverlaps: Boolean = false): List[Int] = {
    def recursive(index: Int, accumulator: List[Int]): List[Int] = {
      if (!(index < source.size)) accumulator
      else {
        val position = source.indexOf(target, index)
        if (position == -1) accumulator
        else {
          recursive(position + (if (withinOverlaps) 1 else target.size), position :: accumulator)

    if (target.size <= source.size) {
      if (!source.equals(target)) {
        recursive(0, Nil).reverse
      else List(0)
    else Nil

Any guidance you can give me replacing this with a proper standard library entry point would be greatly appreciated.

UPDATE 2014/Jul/22:

Inspired by Siddhartha Dutta's answer, I tighted up my code. It now looks like this:

  def indexesOf(source: String, target: String, index: Int = 0, withinOverlaps: Boolean = false): List[Int] = {
    @tailrec def recursive(indexTarget: Int, accumulator: List[Int]): List[Int] = {
      val position = source.indexOf(target, indexTarget)
      if (position == -1) accumulator
      else {
        recursive(position + (if (withinOverlaps) 1 else target.size), position :: accumulator)
    recursive(index, Nil).reverse

Additionally, if I have a source string of "aaaaaaaa" and I use a target string of "aa", I would like by default to get back a List[Int] of List(0, 2, 4, 6) which skips a search starting inside of a found substring. The default can be overridden by passing "true" for the withinOverlaps parameter which in the "aaaaaaaa"/"aa" case would return List(0, 1, 2, 3, 4, 5, 6).

by chaotic3quilibrium at July 22, 2014 08:29 PM


Recovering a point embedding from a graph with edges weighted by point distance

Suppose I give you an undirected graph with weighted edges, and tell you that each node corresponds to a point in 3d space. Whenever there's an edge between two nodes, the weight of the edge is the distance between the points.

Your goal is to reconstruct the relative positions of the points, given only the available distances (represented by the edge weights). For example, if I gave you $d_{0,1} = d_{0,2} = d_{0,3} = d_{1,2} = d_{1,3} = d_{2,3} = 1$, then you know the points are the vertices of a tetrahedron. You don't know where it is relative to the origin, or its orientation, or if it's been mirrored, but you can tell it's a tetrahedron.

In general, the problem is easy if I give you all of the edge lengths. Just arbitrarily pick a point $p_0$ to be at $(0,0,0)$, then pick a neighboring point $p_1$ and place it at $(d_{0,1},0,0)$, then a common neighbor $p_2$ gets triangulated onto the XY plane, then a final common neighbor $p_3$ gets triangulated into the half-space $z > 0$ and breaks the remaining symmetry (assuming you didn't pick degenerate points). You can use those four points to triangulate all the remaining ones.

On the other hand, when some edge lengths are missing it may not be possible to recover the embedding. For example, if there's a vertex that disconnects the graph when cut, then the two components it would separate if removed can swing around relative to each other.

Which raises the questions:

  • How expensive is it to find a solution?
  • How do you determine if a solution is unique, up to translation/rotation/mirroring? Is 3-connectedness sufficient? Necessary?
  • What sorts of conditions make the problem trivial?
  • If I don't promise the edge weights actually correspond to point distance sin 3d, how expensive is it to determine if an embedding is possible at all?

by Strilanc at July 22, 2014 08:24 PM

Longest Path A*, Admissible Heuristics, and Optimalness

Im working on a modified A Star algorithm, that instead of attempting to find the shortest path, it tries to find the longest past, or in my specific case, the highest weighted path.

I have some number of classes, each with a list of unique nodes, with each node having its own value. Im attempting to determine the best possible combination of a node from each class to determine the combination or path that has the greatest weight. To do this, I am viewing these classes as a tree, with each Level having the nodes contained within a single class.

Then I search through this tree using A Star, or more specifically, im searching through my tree based on a search stack. Once a node is explored, its children are inserted in a sorted order based on their weight (plus their ancestors weight) plus the possible future weight (my heuristic). Then the top of the stack, with the highest value is selected to search next.

To do this I have an overestimating heuristic, that is it never underestimates the the optimal solution.

If I am looking for the highest weight and not the lowest weight, is this heuristic admissible, and thus my algorithm optimal?

PS: A FormalIsh defination of the current algorithm.

Let S = {S1, S2, ... , Sn}

and each Si has a set of items and a NULL, which represents an item not chosen from this set.

Si = {I1, I2, ... , Im, NULL}

Also each item only ever exists in one set, IE Si U Sj = Si + Sj

Each Item, Ii has an associated value, Vi.

The problem is to select a maximum of M offers, one from each set, when summed yield the highest value. I call this selection a Path, Pi. A path can be complete, meaning it has a selection from all S, or it can be partial where only x offers are contained within it. Also M

In addition, there exists a function IsCompatable(path) that returns true if a path is compatible. Whether a path is compatible or is completely arbitrary. The Max valued path must be compatible. This is why I can not trivially select the M largest Items from each set.

In addition each set contains a NULL item, so that an item need not be selected from that set.

The Trivial Algorithm would to make a search tree and generate all possible combinations of items in S, with each path to the trees leaves said to be a path.

let G(P) be the current value (the summed value of each item) of the partial path. Let H(P) be an estimation (heuristic) of the value of the future path. H(P) = The Y values from Y items from Y Si in S. Each item is the item with the maximum value in the Si where i > len(P). Y = M - the current length of the partial path P.

To Find the path with the greatest value, I keep a sorted Queue of partial paths, sorted on their values + their possible future values, ie G(Pi) + H(Pi). I Initialize this Queue by adding paths contained in S1.

While the Queue is not empty or a path has not been found:
    p = Pop the path from the top of Q
    if p is Complete:
        A full path has been found
        return p
    find the possible children of p by adding an item to p from the next set.
    for child in Children:
        if IsCompatable(child):
            add child back to Q in sorted order on G(child) + H(child)

There it is, now is my Heuristic admissible?

by hoshi at July 22, 2014 08:14 PM


How to use custom plugin in multi-module project?

In a multi-module project, with one module implementing a custom SBT plugin with custom TaskKey, how this plugin can be imported for the project settings of another submodule.

If the submodule using the plugin is define in submodule1/build.sbt, then submodule1/project/plugins.sbt is not loaded.

If plugin is registered in project/plugins.sbt it will fails when loading top/aggregate project as plugin is not necessarily already built.

Is there any other way to define a custom task requiring custom dependency so that it can be used by a submodule?

by applicius at July 22, 2014 08:13 PM


Resource listing models with known VC dimension

Is there any reference resource gathering models with known VC dimension? I am looking for an exhaustive list of models with their VC dimension (and ideally the associated proof or a pointer to it).

It would be in the same spirit as the Complexity Zoo, which gathers most of the known complexity classes, or the textbook Computers and Intractability, which catalogues many NP-Complete problems.

by Franck Dernoncourt at July 22, 2014 08:12 PM


Looking for an algorithm to solve a specific Vehicle Routing Problem

Hopefully I have come to the right place...

I am trying to figure out a way to create routes for trucks to complete a list of orders(drops/stops), while minimizing distance traveled.

  • There is only ever 1 company warehouse in the area.
  • The trucks have to deliver based on capacity.
  • Each truck can hold a maximum (usually 18 pallets).
  • Each order will be for a number underneath that maximum.
  • There will be a maximum number of trucks specified.
  • When trucks are finished with their route, they will return to the company warehouse.

I already have all of the orders, the pallets they are requesting, and the distance between each point.

I am an absolute simpleton when it comes to complex problems like these... I am hoping that someone has a simple (relatively) solution, or an article of some sort that could help me down my path.

Thank you all very much for your time!

by Primalpat at July 22, 2014 08:10 PM


call java class in spark pi?

I have been using spark using scala fairly successfully. The driver code uses other scala classes inside RDD.

Now I want to switch to python. Can I still use these external scala class or do I need to rewrite them in python classes?

by user1058511 at July 22, 2014 08:09 PM

Functional Programming Performance

I've recently started to use Scala to solve some programming challenges at Codeforces in order to exercise functional programming skills. Doing so I came across one particular challenge I wasn't able to solve in a way respecting the given execution time limit of 1000ms; the Painting Fence problem.

I tried various different ways, starting with a straight forward recursive solution, trying a similar approach using streams instead of lists, and eventually trying to reduce list manipulations by working a bit more with indices. I ended up having stack overflow exceptions on larger tests that I was able to fix using Scala's TailCall. But still, while the solution correctly solves the problem, it is too slow to complete within 1000ms. In addition to that, there's a C++ implementation shown that is ridiculously fast in comparison (< 50ms). Now I do understand that Scala will be slower compared to C++ in many cases, and I also understand that I could write a more imperative-style solution in Scala that would probably perform a lot better. Still, I'm wondering if I am missing something more fundamental here, because I have a hard time believing that functional programming is that much slower in general (and I am pretty new to functional programming).

Here is my scala code that you can paste in the REPL including the example that takes >1000ms:

import scala.util.control.TailCalls._

def solve(l: List[(Int, Int)]): Int = {

  def go(from: Int, to: Int, prevHeight: Int): TailRec[Int] = {
    val max = to - from
    val currHeight = l.slice(from, to).minBy(_._1)._1
    val hStrokes = currHeight - prevHeight
    val splits = l.slice(from, to).filter(_._1 - currHeight == 0).map(_._2)
    val indices = from :: splits.flatMap(x => List(x, x+1)) ::: List(to)
    val subLists = indices.grouped(2).filter(xs => xs.last - xs.head > 0)

    val trampolines = => tailcall(go(xs.head, xs.last, currHeight)))
    val sumTrampolines = trampolines.foldLeft(done(hStrokes))((b, a) => b.flatMap(bVal => => aVal + bVal)))
    sumTrampolines.flatMap(v => done(max).map(m => Math.min(m, v)))
  go(0, l.size, 0).result

val lst = (1 to 5000).toList.zipWithIndex
val res = solve(lst)

And for comparison, here's a C++ example achieving the same thing written by Bugman (includes some read/write from console that I didn't include in the Scala version above):

#include <iostream>
#include <cstdio>
#include <cstdlib>
#include <algorithm>
#include <vector>
#include <string>
#include <set>
#include <map>
#include <cmath>
#include <memory.h>
using namespace std;
typedef long long ll;

const int N = 1e6+6;
const int T = 1e6+6;

int a[N];
int t[T], d;

int rmq(int i, int j){
    int r = i;
    for(i+=d,j+=d; i<=j; ++i>>=1,--j>>=1){
        if(i&1) r=a[r]>a[t[i]]?t[i]:r;
        if(~j&1) r=a[r]>a[t[j]]?t[j]:r;
    return r;

int calc(int l, int r, int h){
    if(l>r) return 0;

    int m = rmq(l,r);
    int mn = a[m];
    int res = min(r-l+1, calc(l,m-1,mn)+calc(m+1,r,mn)+mn-h);
    return res;

int main(){
    //freopen("input.txt","r",stdin);// freopen("output.txt","w",stdout);

    int n, m;

    for(int i=0;i<n;++i) scanf("%d",&a[i]);

    a[n] = 2e9;
    for(int i=0;i<n;++i) t[i+d]=i;
    for(int i=n+d;i<d+d;++i) t[i]=n;
    for(int i=d-1;i;--i) t[i]=a[t[i*2]]<a[t[i*2+1]]?t[i*2]:t[i*2+1];


    return 0;

At least before I introduced the explicit tail calls, the more functional style seemed more natural to me to solve the problem than a more imperative solution. So I'd be really happy to learn more on what I should be attentive of when writing functional code in order to still get acceptable performance.

by Marc at July 22, 2014 08:07 PM

Function composition dynamically

Let' s say I have a function move.

Now, depending on some inputs (let' s say the input says "Move by 3"), I have to move 3 times.

I would do this like :

move compose move compose move

Is it possible to do function composition dynamically? So that I can build my new function depending on the number of times I have to move.

Thank you

by ccheneson at July 22, 2014 08:05 PM

Portland Pattern Repository


Computing binomial coefficients and factorials modulo a composite number

Does anyone know of a fast algorithm to compute factorials and/or binomial coefficients in general or modulo a composite number in particular (for composite moduli I am interested in the case where the factorization is not necessarily known) I know that for primes and factorials we can apply Wilson's Theorem but I was wondering if there is also a fast method for composite numbers. For binomial coefficients I was thinking of using recursion and the addition rule: $$\binom{n}{x} + \binom{n}{x+1} = \binom{n+1}{x+1}$$ somehow but I'm not sure really how to go about it. Any suggessions? Thanks.

by Ari at July 22, 2014 07:59 PM


Excellent information source on advanced machine learning / data mining based trading?

I did check the related posts, like this one here.

However, given if one already has knowledge in finance, machine learning and statistics, and wants to know something more advanced on machine learning based trading rather than the introductory. What would you recommend?

I am looking for anything related: excellent books, research papers, posts, authors, as long as it is related to machine learning and trading in general, but the thing is, really looking for the sources with quality content rather than the ads sites.

by Flake at July 22, 2014 07:57 PM


Moving an element to the front of a list in Scala

I was wondering if there is a way to find an element in a list and move it to the front of the list in Scala? Is there any easy way to do this other than iterating over the list, then removing that element and then pre-pending it to the front of the list?

by cm22 at July 22, 2014 07:51 PM

Tail recursive factorial in Fortran95

I decided to try implementing the factorial function in Fortran95 (f2py limitation) but my efforts are only yielding two return-type-mismatch errors.

Inspiration for solution

In Haskell, we can do something like

fac_helper n acc =
  if n <= 1
  then acc 
  else fac_helper (n - 1) (acc * n)

factorial n = fac_helper n 1

Attempted solution: fac.f95

recursive function facHelper(n, acc) result(returner)
  if (n <= 1) then
    returner = acc
    returner = facHelper(n - 1, n * acc)
end function facHelper

function factorial(n)
  factorial = facHelper(n, 1)
end function factorial

When GNU Fortran (GCC) 4.8.3 is used on fac.f95 as

gfortran -std=f95 ./fac.f95 -o fac

the result is:

Error: Return type mismatch of function 'fachelper' at (1) (REAL(4)/INTEGER(4))
Error: Return type mismatch of function 'factorial' at (1) (REAL(4)/INTEGER(4))

These errors appear (to the un-familiar Fortraner) as out of touch with the code of which compilation was attempted on. I am certain that there are no real numbers declared or used in the attempted solution. ...?

How would the tail-recursive factorial implementation look in Fortran95?

by user0xf00 at July 22, 2014 07:51 PM


How can I go about applying machine learning algorithms to stock markets?

I am not very sure, if this question fits in here.

I have recently begun, reading and learning about machine learning. Can someone throw some light onto how to go about it or rather can anyone share their experience and few basic pointers about how to go about it or atleast start applying it to see some results from data sets? How ambitious does this sound?

Also, do mention about standard algorithms that should be tried or looked at while doing this.

by zm1 at July 22, 2014 07:50 PM


Evaluating clojure function to a quoted form

I am using a function that takes arguments as follows: (test-function '((gate 1) (gate 3) (gate 2)))

The list argument can contain any number of elements where each element is of the form (gate x) where x can be an integer from 0 to 8. I have a function, generate-gate-list, that generates lists of random length (up to 10) and content, although they are ALWAYS of the form above.

Example output of generate-gate-list: ((gate 2)), (()), ((gate 1) (gate 6)), etc.

I would like to be able to nest generate-gate-list inside test-function so that I can test a bunch of randomly generated lists without generating them all beforehand. In other words, I'd like something like this: (test-function '(generate-gate-list)) except where generate-gate-list has been evaluated. I've tried some sort of macro syntax-quote and unquoting, but that leads to resolved variables, like (user/gate 3) which screws up test-function. Here's my generate-gate-list code:

    (defn generate-gate-list []
(map symbol (repeatedly (rand-int 10) #(random-gate))))

random-gate outputs a gate element as a string, i.e "(gate 3)" or "(gate 2)".

So in short, I'd like (test-function (something-here (generate-gate-list))) or (test-function (modified-generate-gate-list)) to be equivalent to (test-function '((gate 1) (gate 4) (gate 2))) or some other arbitrary output of generate-gate-list. Thanks!

by user3225710 at July 22, 2014 07:46 PM


Finding the most part of common information

Let we have strings $x$ and $y$. I want to find the most part of extracting common information of $x$ and $y$, that is string $z$ with $C(z) + C(x|z) = C(x)$, $C(z) + C(y|z) = C(y)$, $C(z) \to max$ ($C$ is Kolmogorov complexity).

Is this problem (find so $z$) computable? May be $CT(z|x,y) = 0$ ($CT$ is total complexity)

by Alexey at July 22, 2014 07:39 PM



Pip packages not found during Vagrant provision with Ansible

When using vagrant and Ansible to install pip requirements from a .txt file I'm experiencing the following error when running 'vagrant up'

Downloading/unpacking south==0.8.4 (from -r /opt/app/requirements.txt (line 11))
  Storing download in cache at /root/.cache/pip/
  Running egg_info for package south

Downloading/unpacking djangorestframework==2.3.13 (from -r /opt/app/requirements.txt (line 12))
  Could not find any downloads that satisfy the requirement djangorestframework==2.3.13 (from -r /opt/app/requirements.txt (line 12))
No distributions at all found for djangorestframework==2.3.13 (from -r /opt/app/requirements.txt (line 12))
Storing complete log in /root/.pip/pip.log

FATAL: all hosts have already failed -- aborting

This seems to happen to a random package in the requirements, and when I ssh into the Vagrant box and run 'sudo pip install -r requirements.txt' myself, the packages download and install ok.

Has anyone else experienced this issue before or knows what could be causing it?

Here's my VagrantFile

Here's my Ansible playbook.yml


  - name: application playbook
    user: vagrant
    hosts: all
    gather_facts: false
      - name: Install app pip packages
        sudo: true
        pip: name={{item}} state=present
          - jinja2
          - markupsafe

      - name: install app specific pip packages
        sudo: true
        pip: requirements=/opt/app/requirements.txt

More info:

Vagrant 1.4.3
Ansible 1.5.4
VirtualBox 4.3.10

by iforgotmylogin at July 22, 2014 07:34 PM




communicational complexity calculation [on hold]

i need help regarding communication complexity of security purpose algorithm. from my basic knowledge, i think algorithm with lower computational complexity is acceptable like O(n) > O(n^2); but what about communication complexity? can anyone please help regarding this comparison?

**** If i need to choose an algorithm based on only communication complexity(cc); what should i do? for example if algo A has cc of O(n) and algo B has cc of O(n^2)..which one is better?

by user232803 at July 22, 2014 07:23 PM


Getting rsync in Ansible to work with Vagrant

I want to test an Ansible script using Vagrant. Everything works fine until it tries to do a rsync to the remote host:

- name: Install custom dev user settings
  local_action: command rsync -ave ssh roles/common/files/home/{{ }} 
      {{ ansible_ssh_user }}@{{ inventory_hostname }}:/#home/
  with_items: dev_users
  when: "{{ item.custom }} == True"
  - dev_custom
  - dev_users
  - users

However it fails at this point - it seems to be trying to login via password but I don't know why as it should connect to Vagrant via SSH, right (I have elided some information below with ... because it mentioned keys) ?> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-1393178896.64-215542007508316 && chmod a+rx $HOME/.ansible/tmp/ansible-1393178896.64-215542007508316 && echo $HOME/.ansible/tmp/ansible-1393178896.64-215542007508316']
<> REMOTE_MODULE command rsync -ave ssh roles/common/files/home/someUser vagrant@
<> PUT /tmp/tmpm3BnEW TO /home/mark/.ansible/tmp/ansible-1393178896.64-215542007508316/command
<> EXEC /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=...] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-...; /usr/bin/python /home/mark/.ansible/tmp/ansible-1393178896.64-215542007508316/command; rm -rf /home/mark/.ansible/tmp/ansible-1393178896.64-215542007508316/ >/dev/null 2>&1'"'"''
failed: [] => (item={ ... }) => {"failed": true, "item": { ... }, "parsed": false}
invalid output was: [sudo via ansible, key=...] password: 
[sudo via ansible, key=...] password: 
Sorry, try again.
[sudo via ansible, key=...] password: 
[sudo via ansible, key=...] password: 
Sorry, try again.
[sudo via ansible, key=...] password: 
[sudo via ansible, key=...] password: 
Sorry, try again.
sudo: 3 incorrect password attempts

Any idea how to get around this?

I do deploy a custom key to the box via ansible before I do this e.g.:

- name: Place ansible authorized key
  authorized_key: user=root key="{{ lookup('file', 'root/.ssh/') }}"
  - root

by Mark Butler at July 22, 2014 07:15 PM


reference question about portfolio optimization

I know the "classical" modern portfolio theory. However I have quite a lot of different sources. It seems that there is not a book which cover this topic in a rigorous way:

  1. theory
  2. application
  3. examples in c++ / R / matlab / ...

Are there any books, which cover these points?

Recently, we discussed in class a dynamic portfolio optimization problem. Now I would like to know if there is any good literature about this topic? With dynamic portfolio optimization I mean changing constraints over time. Many thanks for sharing your experiences.

by user8 at July 22, 2014 07:13 PM


Set SSH Host IP Address on Rackspace for Ansible

The Question

When using the rax module to spin up servers and get inventory, how do I tell Ansible to connect to the IP address on an isolated network rather than the server's public IP?

Note: Ansible is being run from a server on the same isolated network.

The Problem

I spin up a server in the Rackspace Cloud using Ansible with the rax module, and I add it to an isolated/private network. I then add it to inventory and begin configuring it. The first thing I do is lock down SSH, in part by telling it to bind only to the IP address given to the host on the isolated network. The catch is, that means ansible can't connect over the public IP address, so I also set ansible_ssh_host to the private IP. (This happens when I add the host to inventory.)

- name: Add servers to group
    module: add_host
    hostname: "{{ }}"
    ansible_ssh_host: "{{ item.rax.addresses.my_network_name[0].addr }}"
    groups: launched
  with_items: rax_response.success
  when: rax_response.action = 'create'

This works just fine on that first run of creating and configuring new instances. Unfortunately, the next time I try to connect to these servers, the connection is refused because Ansible is trying at an IP address on which SSH isn't listening. This happens because:

  1. Ansible tries to connect to ansible_ssh_host...
  2. But the inventory script has set ansible_ssh_host to the accessIPv4 returned by Rackspace...
  3. And Rackspace has set accessIPv4 to the public IP address of the server.

Now, I'm not sure what to do about this. Rackspace does allow an API call to update a server and set its accessIPv4, so I thought I could run another local_action after creating the server to do that. Unfortunately, the rax module doesn't appear to allow updating a server, and even if it did it depends on pyrax which in turn depends on novaclient, and novaclient only allows updating the name of the server, not accessIPv4.

Surely someone has done this before. What is the right way to tell Ansible to connect on the isolated network when getting dynamic inventory via the rax module?

by jchamberlain at July 22, 2014 07:08 PM


What is a semantic cognitive map

Based on: J. P. Carvalho, "On the Semantics and the Use of Fuzzy Cognitive Maps in Social Sciences" (WCCI, 2010 -- PDF) and Richard Dagan's web page Cognitive Mapping.

A cognitive map consists of nodes representing concepts and ties indicating relationship between the concepts. In Carvalho section IV, the semantics is used in perspective of the presented model. Dagan's definition of semantic map is that it is a tree like structure branching from a main concept.

Also, the answer Are semantic networks ontologies? further adds to my confusion as to what is the significant different between cognitive map where nodes are concepts and semantic map where also the nodes are concepts.

I am unable to find significant difference between concept map and semantic map. Can somebody please explain in simple language? Thank you

by Ria George at July 22, 2014 06:55 PM


How to configure a fine tuned thread pool for futures?

How large is Scala's thread pool for futures?

My Scala application makes many millions of future {}s and I wonder if there is anything I can do to optimize them by configuring a thread pool.

Thank you.

by user972946 at July 22, 2014 06:53 PM

Elegant way in Scala to render List of objects(columns) with List attribute(cell values) in a grid?

What is the "best" way to render a List of model objects(matrix) in a Scala template such as:

public class Column extends Model {

    public String columnLabel;

    public List cells;

public class Cell extends Model {
    public String rowLabel; //index
    public BigDecimal value;

For the sake of this question, cells.size() and rowLabel are consistent for all column objects. The controller returns a List[Column] to the view. I have tried to convert the List to an Array with a helper:

@matrix(list: List[Column]):Array = @{

    var rowCount = list.head.values.size()
    var colCount = list.size()
    var m = Array.ofDim[String](rowCount,colCount)
    for (i <- 0 to rowCount-1) {
        for ( j <- 0 to colCount-1) {
            matrix(i)(j) = list(j).cells(i).value.toString();
    return m;

and then in the view:

    @for(i <- 1 to currentPage.getList.head.values.size()) {
        <div class="row">
            @for(j <- 1 to currentPage.getList.size()) {
                <div class="col-md-1">@matrix(currentPage.getList)(i)(j)</div>

but of course this is only extracting the matrix values and not the column or row labels.

Is there some Scala array goodness that can be used on the List of Lists? Efficiency is important as the array size will be approx. 20 columns x 2000 rows. Or is it a better approach to have the controller return the matrix rows explicitly rather than try to convert them in the view?

by geeforce at July 22, 2014 06:49 PM

object X is not a member of package Y

I am trying to implement an abstract Java class in Scala and in the eclipse IDE it is not showing errors but, when I try to compile it is throwing an error

The abstract Java class is defined as

package de.hybris.platform.webservices;

import de.hybris.platform.core.model.ItemModel;

public abstract class AbstractCommand<RESOURCE> extends AbstractYResponseBuilder<RESOURCE, Object, Object> implements
        Command<RESOURCE, Object, Object>

Code for Scala class is


import de.hybris.platform.webservices.AbstractCommand
import de.hybris.platform.core.model.product.ProductModel

class ProductCommand extends AbstractCommand[ProductModel] {

  override def execute(resourceModel: ProductModel, requestDTO: Object): ProductModel = null


And finally, the damn error message is

object webservices is not a member of package de.hybris.platform

Help resolving this is much appreciated!

by drunkenRabbit at July 22, 2014 06:39 PM



Habit of making private methods static to increase functional programming

I got in touch with the functional programming paradigm (haskell, scala) and like the concept. I'm trying to incorporate these functional principles in my every day work.

Here an example

public class Functional

  private final Object o1;
  private final Object o2;

  public Functional(Object o1, Object o2)
    this.o1 = o1;
    this.o2 = o2;

   * method has side effects 
  private void method()
    //    o1.someChange();
    //    ...
    //    o2.someChange();

   * method has no side effects - it only uses its parameters
  private static void method(Object o1, Object o2)
    //    o1.someChange();
    //    ...
    //    o2.someChange();

  public void work()
    method(o1, o2);

  public static void main(String[] args)
    Functional f = new Functional(new Object(), new Object());;

I find the static method easier to maintain, also for people who did not write the code, since they just have to look at the method parameters - which can be an advantage in big classes. Another minor advantage is performance, because after compilation static methods get called with invokestatic which is slightly faster.

The public methods are still kept non static, since I don't want to discard OOP/encapsulation. I'm only talking about private static methods.


So what do you think of this apprach? Esp. what are the negativ sides my new habit of making all private methods static - within reason, as long as I don't need more than 3, 4 parameters?

by mike at July 22, 2014 06:31 PM


Detecting coplanarity by given pairwise distances

Consider an undirected weighted graph $G = (V,E)$, where $V \subset \mathbb{R}^3$ so the points are 3D, and the weight of an edge equals the (Euclidean) distance between its endpoints. Note that we're not given the coordinates of the points in V. We may not even be given all pairwise distances, so the graph need not be complete, and it may even be sparse.

Suppose we're given $k$ and told that there are $k$ planes such that all the vertices belong to at least one of those plane. We want to find $k$ such planes, with an added restriction:

In order to determine whether 4 points are coplanar given only their pairwise distances, the most straightforward method is to use the Cayley-Menger determinant. For our problem, this would require the graph to be fairly dense, since we'd need to know most of the pairwise distances to apply Cayley-Menger. The restriction is to find $k$ planes without using the Cayley-Menger determinant.

If this is impossible, can we obtain a proof that states this is impossible? In other words, can we prove that for any such graph $G$ and given $k$, if we have enough information to find $k$ planes for $G$ by some means, then we have enough information to use Cayley-Menger to find $k$ planes?

by cagirici at July 22, 2014 06:27 PM


ScalaTest in Intellij does not print out console messages

I am running Spark tests that use ScalaTest. They are very chatty on the command line using the following command (as an aside the -Dtest= is apparently ignored - all core tests are being run..):

 mvn -Pyarn -Phive test -pl core -Dtest=org.apache.spark.MapOutputTrackerSuite

There are thousands of lines of output, here is a taste:

7:03:30.251 INFO org.apache.spark.scheduler.TaskSetManager: Finished TID 4417 in 23 ms on localhost (progress: 4/4)
17:03:30.252 INFO org.apache.spark.scheduler.TaskSchedulerImpl: Removed TaskSet 38.0, whose tasks have all completed, from pool 
17:03:30.252 INFO org.apache.spark.scheduler.DAGScheduler: Completed ResultTask(38, 3)
17:03:30.252 INFO org.apache.spark.scheduler.DAGScheduler: Stage 38 (apply at Transformer.scala:22) finished in 0.050 s
17:03:30.288 INFO org.apache.spark.ui.SparkUI: Stopped Spark web UI at http://localhost:4041
17:03:30.289 INFO org.apache.spark.scheduler.DAGScheduler: Stopping DAGScheduler

However in IJ only tests Pass/Fail are printed out. So how to view the same chatty INFO level output as on command line?

by javadba at July 22, 2014 06:18 PM

Apache Spark: distinct doesnt work?

Here is my code example:

 case class Person(name:String,tel:String){
        def equals(that:Person):Boolean = == && ==}

 val persons = Array(Person("peter","139"),Person("peter","139"),Person("john","111"))

It returns

 res34: Array[Person] = Array(Person(john,111), Person(peter,139), Person(peter,139))

Why distinct doesn't work?I want the result as Person("john",111),Person("peter",139)

by MrQuestion at July 22, 2014 06:10 PM


How to deal with extreme cases in normal random numbers generation?

In order to generate normal random numbers, one usually generates random numbers following a uniform distribution $Z \sim \mathcal{U}(0,1)$ and then applies the reverse CDF function on them $X=\Phi^{-1}(Z) \sim \mathcal{N}(0,1)$.

However, I encountered a problematic case when one of the generated $Z$ turns out to give exactly 0. Then, you have $X=\Phi^{-1}(Z)=- \infty$.

This is pretty problematic when you generated random samples because it will usually break all your variance/covariance measure basically returning nan or inf when the samples contain infinite number.

How do you usually handle this? Do you check after each generated random number whether the value is 0 or 1 and shift it slightly (or simply dicard it)?

by SRKX at July 22, 2014 06:01 PM


Term Paper on Computer Graphics

Hi reddit,

I am planning out my paper on computer graphics. It'll be 15 pages when it's all said and done.

I am writing to explore the effects of different color choices have on media. Namely with a focus on Video Games. I thought it would be a good idea to lead the paper through a narrative of a graphic artist/designer who is making the color choices for an upcoming release.


I was wondering if you, reddit, could give me some inspiration to write on. I have most of the paper planned out, but if you could throw some buzz words at me (or hopefully 1 sentence) to inspire me for a few details to cover, that would be just great.

Ex: "You could talk about x."

heres what I have so far:

  • Video games

    • VR
      • colors that are more pleasant to the users
      • more or less eye stress (saturation/brightness)
      • best choices for a HUD to not class with the world
        • change the color to darker to contrast a lightened backdrop
    • 3D environments
      • main characters and backgrounds
      • colors use for leading the story along
        • color choices for hinting on where/what to do (leading the user)
  • Graphic Design

    • best color choices for different categories(menu choices)
    • best use of colors for level of importance
    • box art, don't color, spacing + sizing
  • CGI/Films

    • this chapter may be incorporated into the 3D environments chapter
  • Sound and color

    • this encompasses all other topics
    • describe moods (color and sound relationships)
    • Synesthesia

Thanks for your time!

submitted by nsuetv14
[link] [4 comments]

July 22, 2014 06:01 PM


Why does Scala sbt update every time I start it from command prompt?

I am a beginner in Scala and every time I start sbt from command prompt it goes into a long update and it takes several minutes for the sbt to update. I read in the instructions that the sbt was going to update the first time it was started from the command prompt but in my case it is updating every time I start it. I have to wait several minutes for it to update and it takes away some of the joy of learning Scala. So my question is how do I stop sbt from updating every time and is it normal for it to do so?


This is how my sbt command looks and after this it goes into a long update.

enter image description here


enter image description here

by Zindarod at July 22, 2014 06:00 PM

Will upgrading fix the FreeBSD 8.1 portstree?

I have a machine running FreeBSD and the ports tree is accidentaly updated to one of FreeBSD 8 stable. The problem is that my FreeBSD 8.1 is not configured to use PKG for packages so I cannot update or install packages.

I want to upgrade the machines FreeBSD but get the ports tree in working order before I do that.

It seems the only solution is installing an old 8.1 version of the ports tree, but I cannot find any. Can someone tell me where to find this?

Would upgrading to the lastest stable version 8 also solve my problem?

by Lexib0y at July 22, 2014 05:55 PM

metrics clojure ring middleware not recording metrics

In my handler I have the following defined:

(def reg (new-registry))

(def app (-> app-routes 
         (instrument reg)))

If I go to /metrics I only see an empty json object. What am I missing? Also, the documetation is not clear as to how I can report on the console, it seems report-to-console which is in the documentation is not used any more, instead there are reporters but I cannot find any documentation as to how you use reporters.

by shmish111 at July 22, 2014 05:54 PM



LZW TIFF decoding

When decoding tiff files with LZW decompression, the first 9 bits in the encoded bitstream should be "256", the clear-code.

But when I read it I get a 128, which I just can't figure out. I created the file with GDAL.

My code reading the file is:

val res = (for {
  i <- 0 until next
  if (bitSet.get(i + index))
  } yield (1 << i)).sum

The index is the index in the encoded bitstream and next is how many bits I should read (starting with 9).

So my question is why do I read an 128 instead of an 256? When printing the bitstream input the first bit that is set as 1 is bit number 8 (index 7).

The file in question is:


by Johan S at July 22, 2014 05:51 PM

Spark Streaming Window Operation

The following is simple code to get the word count over a window size of 30 seconds and slide size of 10 seconds.

import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.api._

val ssc = new StreamingContext(sc, Seconds(5))

// read from text file
val lines0 = ssc.textFileStream("test")
val words0 = lines0.flatMap(_.split(" "))

// read from socket
val lines1 = ssc.socketTextStream("localhost", 9999, StorageLevel.MEMORY_AND_DISK_SER)
val words1 = lines1.flatMap(_.split(" "))

val words = words0.union(words1)
val wordCounts =, 1)).reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))


However, I am getting error from this line:

val wordCounts =, 1)).reduceByKeyAndWindow(_ + _, Seconds(30), Seconds(10))

. Especially, from _ + _. The error is

51: error: missing parameter type for expanded function ((x$2, x$3) => x$2.$plus(x$3))

Could anybody tell me what the problem is? Thanks!

by user2895478 at July 22, 2014 05:46 PM

How to fix the pattern-matching exhaustive warning?

Some scala code:

val list = List(Some("aaa"), Some("bbb"), None, ...)

list.filter(_!=None).map {
  case Some(x) => x + "!!!"
  // I don't want to handle `None` case since they are not possible
  // case None

When I run it, the compiler complains:

<console>:9: warning: match may not be exhaustive.
It would fail on the following input: None
                  list.filter(_!=None).map {
res0: List[String] = List(aaa!!!, bbb!!!)

How to fix that warning without providing the case None line?

by Freewind at July 22, 2014 05:44 PM



Opposite of Tail-Risk Hedge (Established Vocabulary)

I'm working on a client memo explaining several approaches to equity hedging, and I'm looking for a not-too-technical term for a hedging strategy where I try to keep options near the money, as to have a quickly reacting hedge, expensive, but drastically reduced drawdown (hopefully).

Of course, the opposite would simply be called tail-risk hedge, but what if I tighten the moneyness? I was thinking about core hedge, near-the-money hedge, continuously adjusted hedge, ...

Obviously this is just a marketing buzzword, but is there some established word that will be understood by most? Also, clients are not too technical, so it may well be a fuzzy word, or not 100% accurate.

by phi at July 22, 2014 05:36 PM



def vs lazy val in case class

I have a DAO object which I defined as a case class.

case class StudentDAO(id: Int) {
  def getGPA: Double = // Expensive database lookup goes here
  def getRank: Int = // Another expensive database operation and computation goes here
  def getScoreCard: File = // Expensive file lookup goes here

I would naturally make getGPA and getRank and getScoreCard defs and not vals because I don't want them to be computed before they may be used.

What would be the performance impact if I marked these methods as lazy vals instead of defs? The reason I want to make them lazy vals is: I do not want to recompute the rank each time for a Student with id "i".

I am hoping that this will not be marked as duplicate because there are several questions as below which are mostly about differences:

When to use val, def, and lazy val in Scala?

def or val or lazy val for grammar rules?

`def` vs `val` vs `lazy val` evaluation in Scala

Scala Lazy Val Question

This question is mainly aimed towards the expenses (tradeoffs between CPU vs. memory) in making a method a lazy val for costly operations and what would one suggest over other and why?

EDIT: Thank you for the comment @om-nom-nom. I should have been more clear with what I was looking for.

I read here:

Use of lazy val for caching string representation

that string representation of the object is cached (see @Dave Griffith's answer). More precisely I am looking at the impact of Garbage Collection if I made it a lazy val instead of def

by Venkat Sudheer Reddy Aedama at July 22, 2014 05:32 PM

How to ensure tail recursion consistently

I'd like to make some function be optimized for tail-recursion. The function would emit stackoverflow exception without optimization.

Sample code:

import scala.util.Try
import scala.annotation.tailrec

object Main {
  val trials = 10

  val gcd : (Int, Int) => Int = {
    case (a,b) if (a == b) => a
    case (a,b) if (a > b) => gcd (a-b,b)
    case (a,b) if (b > a) => gcd (a, b-a)

  def main(args : Array[String]) : Unit = {

  def testTailRec() {
    val outputs : List[Boolean] = Range(0, trials) + 6000) map { x =>
      Try( gcd(x, 1) ).toOption.isDefined

  def outputTestResult(source : List[Boolean]) = {
    val failed = source.count(_ == false)
    val initial = source.takeWhile(_ == false).length
    println( s"totally $failed failures, $initial of which at the beginning")

Running it will produce the following output:

[info] Running Main 
[info] totally 2 failures, 2 of which at the beginning

So, first two runs are performed without optimization and are dropped half-way due to the stackoveflow exception, and only later invocations produce desired result.

There is a workaround: you need to warm up the function with fake runs before actually utilizing it. But it seems clumsy and highly inconvenient. Are there any other means to ensure my recursive function would be optimized for tail recursion before it runs for first time?


I was told to use two-step definition

def gcd_worker(a: Int, b: Int): Int = {
      if (a == b) a
      else if (a > b) gcd(a-b,b)
         else gcd(a, b-a)
val gcd : (Int,Int) => Int = gcd_worker(_,_)

I prefer to keep clean functional-style definition if it is possible.

by ayvango at July 22, 2014 05:32 PM


Big Data Update - New Blog and New Web-Based Training

The topic of big data comes up almost every time I meet with current or potential AWS customers. They want to store, process, and extract meaning from data sets that seemingly grow in size with every meeting.

In order to help our customers to understand the full spectrum of AWS resources that are available for use on their big data problems, we are introducing two new resources -- a new AWS Big Data Blog and web-based training on Big Data Technology Fundamentals.

AWS Big Data Blog
The AWS Big Data Blog is a way for data scientists and developers to learn big data best practices, discover which managed AWS Big Data services are the best fit for their use case, and help them get started on AWS big data services. Our goal is to make this the hub for developers to discover new ways to collect, store, clean, process, and visualize data at any scale.

Readers will find short tutorials with code samples, case studies that demonstrate the unique benefits of doing big data on AWS, new feature announcements, partner- and customer-generated demos and tutorials, and tips and best practices for using AWS big data services.

The first two posts on the blog show you how to Build a Recommender witrh Apache Mahout on Amazon Elastic MapReduce and how to Power Gaming Applications with Amazon DynamoDB.

Big Data Training
If you are looking for a structured way to learn more about the tools, techniques, and options available to you as you learn more about big data, our new web-based Big Data Technology Fundamentals course should be of interest to you.

You should plan to spend about three hours going through this course. You will first learn how to identify common tools and technologies that can be used to create big data solutions. Then you will gain an understanding of the MapReduce framework, including the map, shuffle and sort, and reduce components. Finally, you will learn how to use the Pig and Hive programming frameworks to analyze and query large amounts of data.

You will need a working knowledge of programming in Java, C#, or a similar language in order to fully benefit from this training course.

The web-based course is offered at no charge, and can be used on its own or to prepare for our instructor-led Big Data on AWS course.

-- Jeff;

by Jeff Barr ( at July 22, 2014 05:32 PM



Is it possible to insert rcords in lobos migration plugin for clojure?

In clojure, I am using lobos library with "mysql/mysql-connector-java" dependency.
I've created some master tables using lobos migration library in clojure.
Now I need to insert some of the default values through the migration itself.
Is there any way the library provide or should I use any other library to write DB migration.
Please suggest.

Thanks in advance.

by Richa at July 22, 2014 05:16 PM


Find an MST when $w: E \to \{ 1,2\}$

I've been asked the following question:

Given a connected undirected graph $G=(V,E)$ and a weight function $w: E \to \{1,2\}$, suggest an efficient algorithm that finds an MST of the graph.

After a few clarifications, the algorithm should run in time $O(|V|+|E|)$. I believe we can do this because of the fact that there are two types of weight of edges.

I tried the following idea, which did not work (I had a contradicting example):

Run BFS on the graph, and output the BFS tree.

I thought that shortest paths in this case would also mean easiest trees, but this is not the case.

I also tried to make each edge $e$ such that $w(e)=2$ into two edges of weight $1$ but my friend also gave a contradicting example.

Can you help please? This is not a homework question, this is a previous exam question as I am studying for my algorithms exam which is soon.

by TheNotMe at July 22, 2014 05:13 PM



sbt subproject cannot find it's dependencies

I have a project tree consisting of three projects A, B and C

B depends on A, and C depends on both A and B.

A and B are checked out in C's lib/ and both build fine using sbt compile

However, when I compile C, the build of B fails, complaining that it cannot find certain types/packages:

import org.scalatra.sbt._
import sbt.Keys._
import sbt._

object NwbApiBuild extends Build {
  val Organization = "org.nwb"
  val Name = "NWB API"
  val Version = "0.1.0-SNAPSHOT"
  val ScalaVersion = "2.10.3"
  val ScalatraVersion = "2.3.0"

  lazy val active_slick= Project (
    base = file("lib/active-slick")

  lazy val slick_auth= Project (
    base = file("lib/slick-auth")

  lazy val project = Project (
    settings = Defaults.defaultSettings ++ ScalatraPlugin.scalatraWithJRebel ++ Seq(
      organization := Organization,
      name := Name,
      version := Version,
      scalaVersion := ScalaVersion,
      resolvers += Classpaths.typesafeReleases,
      libraryDependencies ++= Seq(
        "org.scalatra" %% "scalatra" % ScalatraVersion,
        "org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
        "ch.qos.logback" % "logback-classic" % "1.0.6" % "runtime",
        "org.eclipse.jetty" % "jetty-webapp" % "8.1.8.v20121106" % "container",
        "org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "container;provided;test" artifacts (Artifact("javax.servlet", "jar", "jar")),
        "com.typesafe.slick" %% "slick" % "2.0.2",
        "mysql" % "mysql-connector-java" % "5.1.31",
        "joda-time" % "joda-time" % "2.3",
        "org.joda" % "joda-convert" % "1.5",
        "com.github.tototoshi" %% "slick-joda-mapper" % "1.1.0",
        "org.json4s" %% "json4s-native" % "3.2.10",
        "org.json4s"   %% "json4s-jackson" % "3.2.7",
        "c3p0" % "c3p0" % ""
  ) aggregate(active_slick, slick_auth) dependsOn(active_slick, slick_auth)

where slick auth has build file

import org.scalatra.sbt._

name := "slick-auth"

version := "0.0.1-SNAPSHOT"

scalaVersion := "2.10.3"

val ScalatraVersion = "2.3.0"

lazy val active_slick = Project(
  base = file("lib/active-slick")

lazy val root = Project(
  settings = Defaults.defaultSettings ++ ScalatraPlugin.scalatraSettings ++ Seq(
    libraryDependencies ++= Seq(
      "com.typesafe.slick" %% "slick" % "2.0.2",
      "org.slf4j" % "slf4j-nop" % "1.6.4",
      "org.scalatest" %% "scalatest" % "2.2.0" % "test",
      "org.scalatra" %% "scalatra" % ScalatraVersion,
      "org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
      "ch.qos.logback" % "logback-classic" % "1.0.6" % "runtime",
      "org.eclipse.jetty" % "jetty-webapp" % "8.1.8.v20121106" % "container",
      "org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "container;provided;test" artifacts (Artifact("javax.servlet", "jar", "jar")),
      "com.typesafe.slick" %% "slick" % "2.0.2",
      "joda-time" % "joda-time" % "2.3",
      "org.joda" % "joda-convert" % "1.5",
      "com.github.tototoshi" %% "slick-joda-mapper" % "1.1.0",
      "org.json4s" %% "json4s-native" % "3.2.10",
      "org.json4s"   %% "json4s-jackson" % "3.2.7",
      "c3p0" % "c3p0" % ""

and active_slick:

name := "active-slick"

version := "0.0.1-SNAPSHOT"

scalaVersion := "2.10.3"

libraryDependencies ++= Seq(
  "com.typesafe.slick" %% "slick" % "2.0.2",
  "org.slf4j" % "slf4j-nop" % "1.6.4",
  "org.scalatest" %% "scalatest" % "2.2.0" % "test",
  "com.h2database" % "h2" % "1.3.166" % "test"

by A.J.Rouvoet at July 22, 2014 04:37 PM



Rendering fields in Play Framework 2 Form when editing existing object

This may seem a pretty basic question. I'm using Play Framework 2.1.5 with Scala 2.10 to develop a CRUD app (it will have more in the future). I'm also using Twitter Bootstrap 3.2.0. I'm not able to get my template to render the values for a form when I am attempting to update an object.

I'm following along the book Play for Scala. I can already create a new object. Now I want to be able to edit the object.

The Product domain object is mapped using Squeryl:

case class Product(ean: Long, name: String, description: String) extends KeyedEntity[Long]

The companion Person object has the appropriate CRUD functionality. Product.findByEan(ean) is a simple search by field query that returns an Option[Person].

The Persons controller has the Form[Person] mapping:

private val productForm = Form(mapping(
    "ean" -> longNumber.verifying("validation.ean.duplicate", Product.findByEan(_).isEmpty),
    "name" -> nonEmptyText,
    "description" -> nonEmptyText)(Product.apply)(Product.unapply))

In my routes

GET        /products                         controllers.Products.list
POST       /products               
GET        /products/new                     controllers.Products.newProduct
GET        /products/$ean<\d{13}>   Long)
GET        /products/$ean<\d{13}>/edit       controllers.Products.edit(ean: Long)

The Products.edit route ought to load the Product using its ean field and prefill out the edit form with the found product.

def edit(ean: Long) = Action { implicit request =>
    Product.findByEan(ean) match {
        case Some(product) =>
  "found product: ean = " + ean)
            val form = productForm.fill(product)
        case None => Redirect(routes.Products.list)

I know for a fact that I'm successfully retrieving the object as the Logger messages clearly show this in the play console.

The editProducts template to render the form is:

@(productForm: Form[Product])(implicit flash: Flash, lang: Lang)
@import helper._
@import helper.twitterBootstrap._

@productsLayout(Messages("products.form")) {

    @helper.form(action = {
                @Messages("products.details", Messages("products.edit"))
        <p><input type="submit" class="btn primary" value='@Messages("products.edit.submit")'</p>

The problem is that that when I attempt to edit my Product object, this form template is not pre-filling out the <input> tag with the value attribute from the retrieved product object. The same template works just fine when I try to create an object but submit a form with validation errors.

  1. Has anyone else experienced this issue?
  2. Does the twitterBootstrap helper assume Bootstrap 2 rather than Bootstrap 3?

by Alan at July 22, 2014 04:26 PM


Ben Horowitz

How to Ruin Your Company with One Bad Process

“Real quick, whole squad on that real sh*t
0 to 100, n***a, real quick”
—Drake, "0 to 100/The Catch Up"

I am a giant advocate for technical founders running their own companies, but one consistent way that technical founders deeply harm their businesses is by screwing up the budgeting process. Yes, the budgeting process. How ridiculous is that? How does it happen and why is it particularly problematic for engineers?

I'll begin by describing how I screwed it up in my company. Our sales were growing so fast that the biggest problem that we faced was that we literally could not handle all the customers that wanted to sign up for Loudcloud. To combat this and enable us to grow, I worked diligently with my team to plan all the activities that we needed to accomplish to expand our capacity and capture the market before the competition. Next, I assigned sub-goals and activities to each functional head. In conjunction with my leadership team, I made sure that each goal was measurable and supported by paired metrics as well as lagging and leading indicators. I then told the team to figure out what it would take to accomplish those goals and return with their requirements for headcount and program dollars. Finally, I made adjustments to their requests based on industry benchmarks (mostly reductions) to get to a plan that I thought made sense.

Here’s the basic process:

  1. Set goals that will enable us to grow
  2. Break the goals down so that there is clear ownership and accountability for each goal by a specific team
  3. Refine goals into measurable targets
  4. Figure out how many new people are required to hit the targets
  5. Estimate the cost of the effort
  6. Benchmark against the industry
  7. Make global optimizations
  8. Execute

Unless you are an experienced manager, you may not even see what’s wrong with this process, but it very nearly led to my company's demise. In fact, the above process is completely upside-down and should only be followed if you wish to bloat your company to the brink of bankruptcy and create a culture of chaos. 

a16zsummerschool.jpgWhen I asked my managers what they needed, I unknowingly gamified the budgeting process. The game worked as follows: The objective was for each manager to build the largest organization possible and thereby expand the importance of his function. Through the transitive property of status, he could increase his own importance as well. Now you may be thinking, "That wouldn't happen in my company. Most of my staff would never play that game." Well, that's the beauty of the game. It only takes one player to opt in, because once someone starts playing, everybody is going in -- and they are going in hard. 

Gameplay quickly becomes sophisticated as managers develop clever strategies and tactics to improve their chances for winning. One common game technique is to dramatically expand the scope of the goals: "When you said that you wanted to increase our market presence, I naturally assumed that you meant globally. Surely, you wouldn’t want me to take a U.S.-centric view." To really motivate the CEO, another great technique involves claiming dire circumstances if the company fails to achieve its metrics: "If we don't increase sales by 500% and our top competitor does, we will fall behind. If we fall behind, we will no longer be No. 1. If we’re not No. 1, then we won’t be able to hire the best people, command the best prices, or build the best product, and we will spin into a death spiral." Never mind the fact that there is almost no chance that your competitor will grow 500% this year. 

Another subtle problem with this process is that when I asked my team what they needed to achieve their goals, they naturally assumed they would get it. As a result, my team deeply socialized their ideas and newly found money with their teams. This has the added gaming benefit of inextricably tying their demands to company morale. When the VP of marketing asked me for 10 headcount and $5M in program expenses, then shared that plan with his team, it changed the conversation. Now a major cutback to his plan would alarm his team because they had just spent two weeks planning for a much more positive scenario. “Wow, Ben greatly reduced the plan. Should I be looking for a job?” This kind of dynamic put pressure on me to create a more expansive expense plan than was wise. Multiply this by all my managers and I was on my way to burning up all my cash and destroying my culture. 

My core problem was that my budgeting process did not have any real constraints. We were private and did not have a specific profit target that we needed to hit and we had plenty of cash in the bank. Drawing the line on expenses seemed rather arbitrary. In the absence of a hard constraint, I had no constraint. 

An excellent constraining principle when planning your budget is the preservation of cultural cohesion. The enemy of cultural cohesion is super-fast headcount growth. Companies that grow faster than doubling their headcount annually tend to have serious cultural drift, even if they do a great job of onboarding new employees and training them. Sometimes this kind of growth is necessary and manageable in certain functions like sales, but is usually counterproductive in other areas where internal communication is critical like engineering and marketing. If you quadruple your engineering headcount in a year, you will likely have less absolute throughput than if you doubled headcount. As an added bonus, you will burn way more cash. Even worse, you will lose cultural consistency as new people with little guidance will come in with their own way of doing things that doesn’t match your way of doing things. Note that this does not apply to you if you have very small numbers. It's fine to grow engineering from one to four people or from two to eight. However, if you try to grow from 50 to 200, you will cause major issues if you are not extremely careful.

Starting with the cultural cohesion principle, a far better way to run the budgeting process is to start with the constraints. Some useful constraints are:

  • Run rate increase – Note that I say "run rate increase" and not "spend increase". You should set a limit on the amount by which you are willing to increase what you are spending in the last month of the coming year vs. the previous year. 
  • Earnings/Loss – If you have revenue, another great constraint is your targeted earnings or loss for the year. 
  • Engineering growth rate – Unless you are making an acquisition and running it separately or sub-dividing engineering in some novel way, you should strive not to more than double a monolithic engineering organization in a 12-month period.
  • Ratio of engineering to other functions – Once you have constrained engineering, then you can set ratios between engineering and other functions to constrain them as well. 

 After applying the global constraints, the following steps will provide a better process:

  1. Take the constrained number that you created and reduce it by 10-25% to give yourself room for expansion, if necessary.
  2. Divide the budget created above in the ratios that you believe are appropriate across the team.
  3. Communicate the budgets to the team.
  4. Run your goal-setting exercise and encourage your managers to demonstrate their skill by achieving great things within their budgets.
  5. If you believe that more can legitimately be achieved in a group with more money, then allocate that manager extra budget out of the slush fund you created with the 10-25%.

At this point, some readers may think that I've lost my mind. As a technologist, you know that the worst thing that you can do is over-constrain the problem before you start. You'll kill creativity and prevent yourself from getting a truly great outcome. That's precisely why I, as an engineer, struggled with this process: the human factors muck up the logic. Specifically, local incentives, if not properly managed, will sharply motivate human behavior and defeat the global goals. 

It’s critical to recognize this so that you don’t turn your agile, small company into a slow, big company before its time.

by Ben Horowitz at July 22, 2014 04:22 PM


What are the implication of a negative risk-free rate on SML?

What happens to the Security market line (within the CAPM model) when the risk-free rate turns negative?

by user9502 at July 22, 2014 04:20 PM


PartialFunction with recursion [duplicate]

This question already has an answer here:

I have written this code sample:

val fact: PartialFunction[Int, Int] = {
  case 0 => 1
  case n if n > 0 => n * fact(n-1)

And it fails to compile with an error: forward reference extends over definition of value fact.

So my question is whether a partial function can be recursive? May be there is another syntax for that?


This error seems to appear only when trying this in Scala Worksheet.

by Ilya I at July 22, 2014 04:19 PM



ELF Research Projects

Some cool ELF manipulation research projects.


by lattera at July 22, 2014 04:12 PM



Known time complexity advantage of quantum algorithms over classical algorithms

I know that this question may depend on how one formulates each complexity class, but in general, what time complexity advantage does quantum algorithms have over classical algorithms?

by EMP at July 22, 2014 04:09 PM

Definition of Planar 3-SAT

What is the exact definition of Planar 3-SAT? And how can I find a good reference for it, and also for the proof of it's NP-completeness? I need the precise definition? I am a little confused by the results that I found in Google!

Best regards, Elham

by user24175 at July 22, 2014 04:08 PM

Planet Emacsen

Emacs Redux: Travel back and forward in git history

I recently discovered an extremely cool package called git-timemachine that allows you to step though the git history of the file you’re currently editing in Emacs.

Using it is pretty simple:

  • visit a git-controlled file and issue M-x git-timemachine (or bind it to a keybinding of your choice)
  • use the following keys to navigate historic version of the file:
    • p visit previous historic version
    • n visit next historic version
    • w copy the hash of the current historic version
    • q exit the time machine buffer

Here you can see git-timemachine in action:


This package is bundled with Prelude.

by Bozhidar Batsov at July 22, 2014 04:02 PM

High Scalability

Sponsored Post: Apple, Asana, FoundationDB, Cie Games, BattleCry, Surge, Dreambox, Chartbeat, Monitis, Netflix, Salesforce, Cloudant, CopperEgg, Logentries, Gengo, Couchbase, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
    • Software Developer in Test. The iOS Systems team is looking for a Quality Assurance engineer. In this role you will be expected to work hand-in-hand with the software engineering team to find and diagnose software defects. The ideal candidate will also seek out ways to further automate all aspects of our existing process. This is a highly technical role and requires in-depth knowledge of both white-box and black-box testing methodologies. Please apply here
    • Senior Software Engineer -iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of a fast-paced environment make your heart leap? Do you want your technical abilities to be challenged every day, and for your work to make a difference in the lives of millions of people? If so, the iOS Systems Carrier Services team is looking for a talented software engineer who is not afraid to share knowledge, think outside the box, and question assumptions. Please apply here.
    • Software Engineering Manager, IS&T WWDR Dev Systems. The WWDR development team is seeking a hands-on engineering manager with a passion for building large-scale, high-performance applications. The successful candidate will be collaborating with Worldwide Developer Relations (WWDR) and various engineering teams throughout Apple. You will lead a team of talented engineers to define and build large-scale web services and applications. Please apply here.
    • C++ Senior Developer and Architect- Maps. The Maps Team is looking for a senior developer and architect to support and grow some of the core backend services that support Apple Map's Front End Services. Ideal candidate would have experience with system architecture, as well as the design, implementation, and testing of individual components but also be comfortable with multiple scripting languages. Please apply here.

  • Cie Games, small indie developer and publisher in LA, is looking for rock star Senior Game Java programmers to join our team! We need devs with extensive experience building scalable server-side code for games or commercial-quality applications that are rich in functionality. We offer competitive comp, great benefits, interesting projects, and exceptional growth opportunities. Check us out at

  • BattleCry, the newest ZeniMax studio in Austin, is seeking a qualified Front End Web Engineer to help create and maintain our web presence for AAA online games. This includes the game accounts web site, enhancing the studio website, our web and mobile- based storefront, and front end for support tools.

  • FoundationDB is seeking outstanding developers to join our growing team and help us build the next generation of transactional database technology. You will work with a team of exceptional engineers with backgrounds from top CS programs and successful startups. We don’t just write software. We build our own simulations, test tools, and even languages to write better software. We are well-funded, offer competitive salaries and option grants. Interested? You can learn more here.

  • Asana. As an infrastructure engineer you will be designing software to process, query, search, analyze, and store data for applications that are continually growing in scale. You will work with a world-class team of engineers on deploying and operating existing systems, and building new ones for problems that are unique to our problem space. Please apply here.

  • Operations Engineer - AWS Cloud. Want to grow and extend a cutting-edge cloud deployment? Take charge of an innovative 24x7 web service infrastructure on the AWS Cloud? Join DreamBox Learning’s creative team of engineers, designers, and educators. Help us radically change education in an environment that values collaboration, innovation, integrity and fun. Please apply here.

  • Chartbeat measures and monetizes attention on the web. Our traffic numbers are growing, and so is our list of product and feature ideas. That means we need you, and all your unparalleled backend engineer knowledge to help up us scale, extend, and evolve our infrastructure to handle it all. If you've these chops:, come join the team!

  • The Core Application Performance team is seeking talented and experienced software engineers to focus on system reliability and performance, developing solutions for our multi-tenant, on-demand cloud computing system. Ideal candidate is an experienced Java developer, likes solving real-world performance and scalability challenges and building new monitoring and analysis solutions to make our site more reliable, scalable and responsive. Please apply here.

  • Sr. Software Engineer - Distributed Systems. Membership platform is at the heart of Netflix product, supporting functions like customer identity, personalized profiles, experimentation, and more. Are you someone who loves to dig into data structure optimization, parallel execution, smart throttling and graceful degradation, SYN and accept queue configuration, and the like? Is the availability vs consistency tradeoff in a distributed system too obvious to you? Do you have an opinion about asynchronous execution and distributed co-ordination? Come join us

  • Human Translation Platform Gengo Seeks Sr. DevOps Engineer. Build an infrastructure capable of handling billions of translation jobs, worked on by tens of thousands of qualified translators. If you love playing with Amazon’s AWS, understand the challenges behind release-engineering, and get a kick out of analyzing log data for performance bottlenecks, please apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.

Fun and Informative Events

  • OmniTI has a reputation for scalable web applications and architectures, but we still lean on our friends and peers to see how things can be done better. Surge started as the brainchild of our employees wanting to bring the best and brightest in Web Operations to our own backyard. Now in its fifth year, Surge has become the conference on scalability and performance. Early Bird rate in effect until 7/24!

Cool Products and Services

  • A third party vendor benchmarked the performance of three databases: Couchbase Server, MongoDB and DataStax Enterprise. The databases were benchmarked with two different workloads (read-intensive / balanced) via YCSB on dedicated servers. Read.

  • Now track your log activities with Log Monitor and be on the safe side! Monitor any type of log file and proactively define potential issues that could hurt your business' performance. Detect your log changes for: Error messages, Server connection failures, DNS errors, Potential malicious activity, and much more. Improve your systems and behaviour with Log Monitor.

  • The NoSQL "Family Tree" from Cloudant explains the NoSQL product landscape using an infographic. The highlights: NoSQL arose from "Big Data" (before it was called "Big Data"); NoSQL is not "One Size Fits All"; Vendor-driven versus Community-driven NoSQL.  Create a free Cloudant account and start the NoSQL goodness

  • Finally, log management and analytics can be easy, accessible across your team, and provide deep insights into data that matters across the business - from development, to operations, to business analytics. Create your free Logentries account here.

  • CopperEgg. Simple, Affordable Cloud Monitoring. CopperEgg gives you instant visibility into all of your cloud-hosted servers and applications. Cloud monitoring has never been so easy: lightweight, elastic monitoring; root cause analysis; data visualization; smart alerts. Get Started Now.

  • Whitepaper Clarifies ACID Support in Aerospike. In our latest whitepaper, author and Aerospike VP of Engineering & Operations, Srini Srinivasan, defines ACID support in Aerospike, and explains how Aerospike maintains high consistency by using techniques to reduce the possibility of partitions. 

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

by Todd Hoff at July 22, 2014 04:00 PM



NEXPTIME-completeness with more time for reductions

One thing that surprised me when learning about complexity theory is that for a complexity class C, we tend to define C-complete using polynomial time reductions, even when C is a very large complexity class.

This definitely makes sense for P and NP, but how about EXPTIME and NEXPTIME?

So my question is this: What is the subset of NEXPTIME comprised of problems exp-time reducible from every other problem in NEXPTIME? If EXPTIME != NEXPTIME, is this class distinct from both NEXPTIME and NEXPTIME-complete?

by Kurt Mueller at July 22, 2014 03:59 PM



A Tour of Transit

This tour introduces the Transit format via transit-js, an implementation of the Transit format for JavaScript.

Transit is a format and set of libraries for conveying values between applications written in different programming languages.


by jamesmacaulay at July 22, 2014 03:48 PM


Planet Theory

The Burden of Large Enrollments

This week I'm at the CRA Snowbird conference, the biennial meeting of CS chairs and other leaders in the field. In 2012 many of the discussion focused on MOOCS. This year the challenge facing most CS chairs are booming enrollments in CS courses. A nice problem to have, but a problem nevertheless.

Last night we had a broad discussion about the burgeoning number of students. Ed Lazowska showed his NCWIT slides giving anecdotal evidence. It's too early to get a complete survey of CS departments but hardly anyone in the audience felt that enrollments were not going up by double (or triple) digit percentages. Not only an increase in majors, but a number of non-majors, minors or others, who take upper level courses.
At Georgia Tech it seems every engineering student wants to minor in CS.

We all have to deal with the increase in the short term. Depending on the institution, we have more classes, larger classes, enrollment caps, increases in instructors, PhD lecturers, teaching postdocs, automated grading, undergrad and MS TAs and having other departments take up some of the load. All of this could hurt the undergrad experience but with careful planning we can mitigate those effects.

What's driving the increase? Some suggested a change in the ubiquitous view of computing and the more positive opinion of geeks in society. More likely though, the driver is jobs, the great need for computer scientists and not just from computing companies, and the limited jobs for those graduating with non-technical majors.

Is the big shifts in enrollment a permanent change or just another part of the boom and bust cycle in CS? More than a theoretical questions, a permanent change makes for a better argument with deans and provosts to increase faculty sizes in computer science. There are some signs of "this time is different," such as the increase in non-majors in upper level courses, but it's an argument that will have a hard sell unless the increase is sustained for several years. One good argument: If you draw a linear line through enrollments since the 70's you get a consistent 10% yearly increase averaged over booms and busts.

by Lance Fortnow ( at July 22, 2014 03:42 PM



How do you install an external Java library such as caldav4j in my leiningen managed app?

Goes without saying that I'm new to clojure. As a learning project, I'd like to use the caldav4j library to pull down work calendar events however I don't know how to properly pull in an external library that is not found via "lein search".

Update: Sharing my working project.clj file for reference:

(defproject caldav-sync "0.1.0-SNAPSHOT"
  :description "Extract and synchronize calendar entries locally"
  :url ""
  :license {:name "Eclipse Public License"
            :url ""}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [org.osaf/caldav4j "0.7"]]
  :repositories [["caldav4j" ""]])

by jonEbird at July 22, 2014 03:40 PM


ClojureScript debugging tips?

I'm having a fun time coding a small single-page app with Om and core.async, but I'm having trouble getting insight into my code when it goes wrong.

With source maps, I can view the entire ClojureScript call stack at a breakpoint, but I can't inspect the values at each level of the call stack. Viewing the JS values is sometimes useful, but frequently my execution is paused deep in the ClojureScript internals, and it's difficult to figure out what JS values are directly relevant to the ClojureScript in question.

With the browser REPL, I can execute arbitrary ClojureScript, including functions in my current namespace -- but it is hard to recreate the entire execution context of my bug. In Om, for instance, it's hard to debug a component, because it relies on an app argument which is actually a short-lived cursor created from my app state. (I run into a similar problem when trying to debug Ring wrappers; how do I generate a full-featured Ring request/response while playing in the REPL?)

My goal is to run the actual code, pause at a breakpoint, and then inspect the ClojureScript values in play. Is this possible with the current technology?

Currently, I'm just using a bunch of (.log js/console "Doing something with %o [%s]" obj (pr-str obj)) statements... but that's tedious. Right now I feel like I'm going to be slower in CLJS than in JS, even after climbing the learning curve. A stronger debugging story would help a lot.

submitted by iron_ball
[link] [2 comments]

July 22, 2014 03:33 PM


What is the intuition behind the fact that Modified duration = Macaulay Duration / (1+r)?

I understand the derivation of both:take dP/dR and divide by P which will give you both 1) modified duration OR 2) macaulay duration / (1+r)


(notice the weighted average time built into the function from taking the derivative - the math makes sense)

My question is about intuition: how can discounting the weighted average time to maturity by an extra period be equal to the sensitivity of %price to %yield? Perhaps using continuously compounded returns can help in the intuition?

Should I memorize the fact and move on and not require intuition behind it

by qabbonds at July 22, 2014 03:09 PM


How can you prove that all halting probabilites are normal real numbers?

Wikipedia claims that any halting probability (Chaitin's constant) is a normal number.

Since Chaitin's constant is uncomputble, how is a proof the the normalcy of the number possible? Computable numbers like pi and e are thought to be normal, yet there is no proof. It seems surprising to me that we have a proof of Chaitin's constant being normal although it is an uncomputable number, especially since "Chaitin's constant" isn't actually one number, but an infinite set of numbers (since it depends on the program encoding used).

by Sintrastes at July 22, 2014 03:09 PM


Search in a partial ordering defined by tuples of numbers

This is a graph theory and partial ordering problem. Consider a set of triples {(di,ai,ci)}i=1...N, which specify edges between two nodes A and B, d denotes a departure time, a an arrival time and c a cost. Technically there can be multiple incomparable costs, for example in $ and % chance that your goods arrive safely.

An example of such a situation could be these 5 edges:

Example graph between nodes A and B

(I did not draw the time axis on scale)

There are five edges, I, II, III, IV and V. The costs are denoted at the edges, they are either 100, 10 or 1. Edge V is drawn red to easily discriminate it from the other edges, since it crosses through them. However, aside from that it is not different.

Given such edges, a few things are important:

  • An edge is only interesting if it departs after we arrive at A, for example in the image we can arrive at (a), then edge I is not an option anymore. The same goes for edges II and V if we arrive at (b), etc.
  • Given an arrival and its set of interesting edges, and edge ei is dominated if there exists an edge ej with ej< ei. This is the case iff aj ≤ ai ^ cj ≤ ci ^ (aj < ai v cj < ci). In layman terms, cost and arrival need to be at least as small, but one must be strictly smaller. This is a partial ordering, some edges are incomparable.
  • Given a arrival in A, I want to find all relevant, undominated (or Pareto) edges.

We can enumerate all the edges:

  • I: (0 , 1, 100)
  • II: (0.5, 2, 10)
  • III: (2 , 3, 100)
  • IV: (2.5, 4, 10)
  • V: (1.5, 5, 1)

Putting them in a partial ordering, using a directed acyclic graph (dag), we get this:

Directed acyclic graph of partial ordering

An arrow denotes the <-relation between edges. Note that an edge in this case is a node in the dag: confusing, I know. When I say edge, I mean a node in the dag above.

I added an edge (-∞, -∞), which is always irrelevant, but creates a nice 'root' of the dag. This way we sort of have a tree, where leaves sometimes merge without creating cycles. I also only denote the arrival at B and cost, needed to create the dag, but technically there is also a departure are A for each of the edges.

If we were to arrive at A at -1, we can simply return all the children of (-∞, -∞) as relevant and undominated, but for another arrival we may need to traverse the tree differently (in a more complicated fashion). I was thinking of this:

marker := map from edge to bool # marks whether edge has been traversed
for all edges: marker(edge) := false # none have been traversed

traverse(root, arrival):
    if(marker(root)) return []  #already been at this node
    marker(root) := true;
    if(departure(root) > arrival) return [root]; # return singleton list, all children will be dominated by this edge.

    # this node is irrelevant, but possible some children are relevant.
    l = [];
    for each child of root:
        append traverse(child, arrival) to l
    return l

Traverse returns a list of undominated, relevant edges. However, Is this an efficient way to tackle this problem? Is there a better solution to this problem? Is this problem known under a known name?

by Herbert at July 22, 2014 03:08 PM


Select no fields from table in Korma

I'm trying to do a join across a number of tables (three plus a join table in the middle). I think korma is lazily evaluating the last join. What I'm trying to do is to add a condition that restricts the results of the first table in the join, but I'm only interested in fields from the last table in the join.

So for example say I've got clubs, people and hobbies tables, and a people-to-hobbies join table for the last two. Each club can have many people, and each person can have many hobbies.

I'm trying to get the full details of all the hobbies of people who belong to a specific club, but I don't want any fields from the club table. The join table means that korma will create two queries, one to get all the people that are in a specific club, and another to retrieve the hobbies for that person via the people-to-hobbies join table.

My korma query looks something like this:

(select clubs
  (with people
    (with hobbies
      (fields :hobby-name :id)
      (where { "korma coders"}))))

The problem is that I haven't specified which fields I want from clubs and people, and the default is to select *. How can I include no fields from these tables? Is this possible, or does the fact that the hobbies are lazily loaded mean korma has to return some results in the first query (which gets me a filtered list of people), so that when I come to interrogate it later for the hobbies, it has the ids it needs to run the second query?

by Conan at July 22, 2014 03:07 PM

Planet Emacsen

Emacs Redux: Become an Emacs Redux patron

Securing funding for OS projects and blogs (outside of displaying annoying ads) is a hard endeavor. While I’d love to raise enough cash to work on CIDER, projectile and Emacs itself for an entire year through some wildly successful crowdfunding campaign, that’s unlikely to ever happen.

That said, I recently discovered an interesting alternative to one-off crowdfunding campaigns called gittip. Self-described as sustainable crowdfunding (meaning people will get microdonations (say half a buck) from their patrons each week) it seems like a reasonable way to raise a modest sum of money so you could work on open-source projects more than your regular schedule might permit.

So, if you like Emacs Redux and the Emacs projects in which I’m involved like:

  • Prelude - An Emacs distribution built on top of GNU Emacs 24
  • Projectile - Project Interaction Library for Emacs, that stays out of your way
  • clojure-mode - A major mode for programming in Clojure
  • cider - A Clojure programming environment for Emacs
  • guru-mode - An annoying companion on your journey to Emacs mastery
  • rubocop-emacs - Emacs integration for RuboCop
  • zenburn-emacs - The Zenburn color theme, ported to Emacs
  • solarized-emacs - The Solarized color theme, ported to Emacs
  • puppet-mode - A mode for editing Puppet 3 manifests
  • swift-mode - A mode for editing Swift code

consider supporting my work via gittip.

Support via Gittip

by Bozhidar Batsov at July 22, 2014 03:04 PM


Call‑by‑name will succeeds where call‑by‑value may fails: some example cases?

I've landed to SML pages, comparing call‑by‑name and call‑by‑value, asserting the former always succeed while the latter may fails. As this seems counter intuitive to me, I feel at least an example case would be useful.

Call-by-value, call-by-name and call-by-need

Theorem (Church Rosser 1)

For a purely functional language, if call-by-value evaluation and call-by-name evaluation both yield a well-defined result then they yield the same result.

Theorem (Church Rosser 2)

If a well-defined result exists for an expression then the call-by-name evaluation strategy will find it where, in some cases, call-by-value evaluation will not.

What make it counter‑intuitive to me, is that call‑by‑name may re‑evaluate expressions multiple times, which I don't understand how it can make it succeed when call‑by‑value and its one‑time evaluation, would fails. If, as the context is FP and its referential transparency, both expression are supposed to evaluated to the same, how evaluating multiple time may make something succeed or alternatively potentially fails if the same expression is evaluated only once? Unless there are side‑effect somewhere?… is this related to side effects and non‑pure FP?

As a reminder, here is what would look like call‑by‑value and call‑by‑name with SML (if there is an error here, feel free to tell), added call‑by‑need to help not confuse one with the others:

(* This, is supposed to sometime fails… *)

fun call_by_value a =
   a + a  (* `a` is evaluated only once *)

(* …when this is supposed to succeed *)

fun call_by_name a =
   a () + a ()  (* `a` is evaluated for each of its reference *)

(* None of the above should be confused with call‑by‑need: *)

fun call_by_need a =
   let val a' = a ()  (* memoization *)
   in fn () => a' + a'  (* to be evaluated when needed *)

val _ = print (Int.toString (call_by_value 2) ^ "\n")
val _ = print (Int.toString (call_by_name (fn () => 2)) ^ "\n")
val r = call_by_need (fn () => 2) ()  (* memoization *)
val _ = print (Int.toString r ^ "\n")

by Hibou57 at July 22, 2014 03:03 PM



Das Verfahren zu Josef S ist vorbei. Josef S wurde ...

Das Verfahren zu Josef S ist vorbei. Josef S wurde schuldig gesprochen, in allen Anklagepunkten. Er wurde zu zwölf Monaten Haft verurteilt, davon acht bedingt. Die übrigen vier hat er schon in U-Haft abgesessen. So hat sich das Gericht aus der Sache rausgemogelt, ohne eine Schuld einzugestehen oder die Farce dieses Verfahrens zuzugeben. Eine schwere Enttäuschung, dass die Richter nicht die Größe hatten, angesichts dieser lächerlichen Beweislage einen Freispruch auszusprechen.

Das Urteil fußte auf den Aussagen eines einzelnen, sich bei Nachfragen mehrfach selbst widersprechenden, Polizeibeamten. Wieso würde der sich wahllos einen Unschuldigen aus der Menge picken, fragt der Richter. Irgendwas wird schon dran sein.

July 22, 2014 03:02 PM


Examples where the uniqueness of the solution makes it easier to find

The complexity class $\mathsf{UP}$ consists of those $\mathsf{NP}$-problems that can be decided by a polynomial time nondeterministic Turing machine which has at most one accepting computational path. That is, the solution, if any, is unique in this sense. It is thought highly unlikely that all $\mathsf{UP}$-problems are in $\mathsf{P}$, because by the Valiant-Vazirani Theorem this would imply the collapse $\mathsf{NP}=\mathsf{RP}$.

On the other hand, no $\mathsf{UP}$-problem is known to be $\mathsf{NP}$-complete, which suggests that the unique solution requirement still somehow makes them easier.

I am looking for examples, where the uniqueness assumption leads to a faster algorithm.

For example, looking at graph problems, can a maximum clique in a graph be found faster (though possibly still in exponential time), if we know that the graph has a unique maximum clique? How about unique $k$-colorability, unique Hamiltonian path, unique minimum dominating set etc.?

In general, we can define a unique-solution version of any $\mathsf{NP}$-complete problem, scaling them down to $\mathsf{UP}$. Is it known for any of them that adding the uniqueness assumption leads to a faster algorithm? (Allowing that it still remains exponential.)

by Andras Farago at July 22, 2014 03:01 PM

Planet Clojure


We are pleased to announce today the initial release of Transit.

Transit is a format and set of libraries for conveying values between applications written in different programming languages. The key objectives of Transit are to support:

  • Sending values between applications
  • written in different programming languages
  • without requiring schemas/context
    • i.e., to be self-describing at the bottom
  • with extensibility
  • and good performance
  • with reach to the browser

JSON currently dominates similar use cases, but it has a limited set of types, no extensibility, and is verbose. Actual applications of JSON are rife with ad hoc and context-dependent workarounds for these limitations, yielding coupled and fragile programs.

On the other hand, the reach of JSON is undeniable. High performance parsers are widely available. Thus Transit is specified as an encoding to and from both JSON and MessagePack, a binary JSON-like format with widely available parsers. In particular, both formats have parsers written in C for languages like Ruby and Python that reach to C for performance.

Transit supports a minimal but rich set of core types:

  • strings
  • booleans
  • integers (to 64 bits w/o truncation)
  • floats
  • nil/null
  • arrays
  • maps (with arbitrary scalar keys, not just strings)

Transit also includes a wider set of extension types:

  • timestamps
  • UUIDs
  • URIs
  • arbitrary precision integers and decimals
  • symbols, keywords, characters
  • bytes
  • sets
  • lists
  • hypermedia links
  • maps with composite keys

Transit is extensible - users can define extension types in exactly the same way as the included extension types.

The emphasis of Transit is on communication between programs, thus it prioritizes programmatic types and data structures over human readability and document orientation. That said, it does have a readable verbose mode for JSON.

Transit is self describing using tags, and encourages transmitting information using maps, which have named keys/fields which will repeat often in data. These overheads, typical of self-describing formats, are mitigated in Transit by an integrated cache code system, which replaces repetitive data with small codes. This yields not only a reduction in size, but also an increase in performance and memory utilization. Contrast this with gzipping, which, while it may reduce size on the wire, takes time and doesn't reduce the amount of text to be parsed or the number of objects generated in memory after parsing.

We are shipping an 0.8 version of the Transit spec, which has extensive documentation for implementors, as well as interoperable implementations for:

We welcome feedback and suggestions on the transit-format list.

I'd like to thank the team at Cognitect that built Transit:

  • Tim Ewald - Team Lead
  • Brenton Ashworth
  • Timothy Baldridge
  • Bobby Calderwood
  • David Chelimsky
  • Paul deGrandis
  • Benoit Fleury
  • Michael Fogus
  • Yoko Harada
  • Ben Kamphaus
  • Alex Miller
  • David Nolen
  • Russ Olsen

I hope you find Transit useful, and look forward to your feedback.



by "Relevance Inc.'s blog" at July 22, 2014 03:00 PM


"Shared" Connection on Monger Clojure

I am start with monger today, the docs use examples like this:

(let [conn (mg/connect)
      db   (mg/get-db conn "monger-test")
      coll "documents"]
  (mc/insert db coll {:first_name "John"  :last_name "Lennon"})
  (mc/insert db coll {:first_name "Ringo" :last_name "Starr"})

  (mc/find db coll {:first_name "Ringo"}))

All documentation examples use that structure. Always connect to MongoDB, then use db on mc/insert... . The question is, how I can put this code on a function and use it in my functions that execute mongodb queries, without repeat below code all time:

(let [conn (mg/connect)
          db   (mg/get-db conn "monger-test")
          coll "documents"] ...


by elf at July 22, 2014 02:54 PM




Scala for Android - Curious type mismatch error

In my Scala Android project I'm getting the curious error:

type mismatch; found :[Boolean] required:[Boolean]

when calling 'removeAccount' in the code below:

val accountManager = android.accounts.AccountManager.get(this)
for (account <- accountManager.getAccounts) {
      new AccountManagerCallback[Boolean] {
        override def run(result: AccountManagerFuture[Boolean]) {
          try {
            result.getResult(3, TimeUnit.SECONDS)
          } catch {
            case e: Exception => Log.d(getClass.getName, "exception: ", e)
     }, new Handler())

Any ideas on what's going wrong and what can I do to placate the type checker?

by Damian Helme at July 22, 2014 02:41 PM



What's the appropriate way to use Flask with zeromq in production?

I have a Flask app that accepts HTTP requests. When certain HTTP requests come in, I want to trigger a message on a zeromq stream. I'd like to keep the zeromq stream open all the time. I'm wondering what the appropriate way to do this is. Since it is recommended to use gunicorn with Flask in production, doesn't that mean that there will be multiple instances of the Flask app, and if I put the zeromq connection in the same place as the Flask app, only one of those will be able to connect, and the others will fail.

by repalviglator at July 22, 2014 02:30 PM

Clojure : Stuck in writing migration in lobos library, when I am trying to add 2 fields with "timestamp" datatype

In clojure, I am using lobos library with "mysql/mysql-connector-java" dependency.
While writing the migrations to create table, there are 2 fields as "created" and "modified" with the datatype as "timestamp".
When I am not passing any default value for the fields then the library is adding attributes as "on update CURRENT_TIMESTAMP" and default as "CURRENT_TIMESTAMP" for the first field.
i.e. If the "created" will be added first then the default is getting set for it otherwise it is getting set for "modified" field.
I don't want to set the default and ON_UPDATE for created field.
Also the fields are added in some random order in the table.
Can I maintain a sequence to add fields?
Please help me.

by Richa at July 22, 2014 02:23 PM


Keep It (Really) Stupid Simple!

It’s 2014 a year full with creative web design trends but does that really helps? Minimalist design can be seen at every corner with more or less elements to make us think that is good for us too. However, is not difficult to create such landing page/website? This showcase may answer this question.


by WDPMag at July 22, 2014 02:19 PM


how should I handle polling in ZMQ in node.js?

I'm working on building a Paranoid Pirate in node.js. Starting with some python code:

poller = zmq.Poller()

interval = INTERVAL_INIT

heartbeat_at = time.time() + HEARTBEAT_INTERVAL

worker = worker_socket(context, poller)
cycles = 0
while True:
    socks = dict(poller.poll(HEARTBEAT_INTERVAL * 1000))

    # Handle worker activity on backend
    if socks.get(worker) == zmq.POLLIN:
        #  Get message
        #  - 3-part envelope + content -> request
        #  - 1-part HEARTBEAT -> heartbeat
        frames = worker.recv_multipart()

The Go sample code also uses a Poller.

The problem that I'm running into is that zeromq.node doesn't seem to use Pollers. Is the solution in node to just use a callback on message? how should I handle the POLLIN state?

by jcollum at July 22, 2014 02:18 PM


Daniel Lemire

Potentially bogus freelancing advice

For nearly 20 years, I have been freelancing: selling my skills as a free agent. I do very little freelancing these days, but I still regularly give advice to students and colleagues. Is it wise? I let you be the judge: here are my favorite bits of wisdom.

There are bad and good clients.

In retail, the more clients you have, the better. The economics of freelancing are different. That is because you are selling finite ressources (your time and your energy) and every transaction depletes your ressources.

A bad client might waste your talents and skills in a dead-end project. A bad client might use 80% of your energy and contribute less than 20% of your income. A bad client might drop you in favour of a cheaper alternative without thinking twice. A bad client might feel abusive, put you in a bad mood. (Being in a bad mood is bad for business on the long run.) A client might take your business in the wrong direction.

As a freelancer, it is entirely reasonable to turn down work. It is often the strategic thing to do, even if you have nothing else lined up. Think about an actor offered the leading role in a major movie that is bound to be a failure.

Ultimately, the great thing about being a freelancer is the freedom to turn down work. It is not only good business, but it is also what sets you apart from employees.

Everything is negotiable.

When I started freelancing, some clients would put forward rules or policies. These rules were invariably convenient to my clients.

For example, it is common to have bounds on how much consultants can charge. A few times over the years, even recently, a client told me that I could not charge over $50 an hour, as a rule. Whatever the rule or the policy, it is always a matter of negotiation. Do not worry, clients will make “exceptions” if you are worth it.

Intellectual property is another important point… when freelancing, you should not sign away your rights lightly. For example, if you are doing a programming job, consider that giving the client the copyright of your work might prevent you from reusing the same code in other projects. A much more reasonable default stance is to license your work to the client.

In all cases, remember that there are bad and good clients. If a client refuses to negotiate in good faith, he may not be a good client to you.

Do not watch your clock.

Because it is a widespread practice, clients almost always want you to charge by the hour. Often, they want to know ahead of time how many hours you will charge.

Charging by the hour is more of a metaphore. In practice, you should charge by the value provided. That is, suppose that you can solve a problem in 5 minutes but that 99.9% of world experts would take 5 days… then it makes no sense to charge 5 minutes. Similarly, if a client wants you to do some work that most people can do in a few minutes but that will take you days,
you should not charge the client a lot.

A more reasonable approach is to charge flat fees or the equivalent. A flat fee could include a service such as “being available at a moment’s notice”. If it reassures the client, you can translate the flat fee into a fixed number of hours and an hourly rate.

Whatever you are charging, you should not worry about the time you spend on projects too much. Your main worry should be to provide something valuable to your clients.

by Daniel Lemire at July 22, 2014 02:11 PM


How to use Node.js cluster with ZeroMQ reply workers

The basic question is "how to setup a ZeroMQ REPLY socket as a worker?".

In effect, how to swap out the default example using an HTTP server with a ZeroMQ REPLY server, eg.:

var cluster = require('cluster'),
    zmq = require('zmq');

if (cluster.isMaster) {
else {
  // Using `http.createServer(..).listen(5555);` works perfectly

  // However, the following does not:
  var socket = zmq.socket('rep');
  socket.bind( "tcp://*:5555", function (err) { console.log(, err); } );

The first worker logs 'undefined' (ie. no error), whereas the second worker logs an error: Address already in use.

From the "How it works" docs, the bit that seems appropriate is here (emphasis added):

When you call server.listen(...) in a worker, it serializes the arguments and passes the request to the master process. If the master process already has a listening server matching the worker's requirements, then it passes the handle to the worker. If it does not already have a listening server matching that requirement, then it will create one, and pass the handle to the worker.

How to "match requirements" with a ZeroMQ REPLY socket?

by cayuu at July 22, 2014 02:11 PM

Create a scala function to define Gatling HTTP request chains

I'm just a newbie in Scala and gatling.

When I define

Object A{
  val asset_sort = exec(http("Sort by Displays")
   .queryParam("""sort""", """video_starts""")

 val device_sort = and so on ... variables.

Now I want to create a Scala function which return me different variable by passing certain params.

I tried something like this

val asset_sort = generateWebRequest("Sort by Displays", "video_starts", asset_details_map)

  def generateWebRequest(requestName:String, sort:String, queryParamMap:HashMap):ChainBuilder = {

    return exec(http(requestName)
     .queryParam("""sort""", sort)

But it throws error

i_ui\lib\AssetDetail.scala:47: class HashMap takes type parameters
12:50:36.708 [ERROR] i.g.a.ZincCompiler$ -   def generateWebRequest(requestName:String, sort:String, qu
eryParamMap:HashMap):ChainBuilder = {
12:50:36.710 [ERROR] i.g.a.ZincCompiler$ -

by swapnil at July 22, 2014 02:03 PM



Let UNAMBIGUOUS-3DM be defined by analogy to UNAMBIGUOUS-SAT, i.e. as a promise problem version of three-dimensional matching where we may assume there is no more than one solution.

  1. Is there a randomised reduction from SAT, as there is for UNAMBIGUOUS-SAT, so that a polynomial-time algorithm for UNAMBIGUOUS-3DM would imply NP=RP?

    For example, it would suffice for there to be a parsimonious reduction from SAT to 3DM, but I haven't found one in the literature.

  2. If so, does it still work when restricting to planar instances of 3DM, i.e. where the bipartite graph associated with the 3DM instance is planar?

by Robin Houston at July 22, 2014 01:59 PM


installing JDK8 on Windows XP - advapi32.dll error

I downloaded JDK8 build b121 and while trying to install I'm getting the following error:

the procedure entry point RegDeleteKeyExA could not be located in the dynamic link library ADVAPI32.dll

The operating system is Windows XP, Version 2002 Service Pack 3, 32-bit.

by yashhy at July 22, 2014 01:52 PM


What are the axioms, inference rules, and (formal) semantics of a lamba calculus?

Wikipedia says that a lambda calculus is a formal system. It defines a lambda calculus by giving its alphabet, and inductively describing what is inside its formal language.

  1. Since a lambda calculus is a formal system, what are its

    • axioms and
    • inference rules?
  2. Does a lambda calculus have semantics? If yes, how does an interpretation of a lambda calculus look like as a mapping from what subset to another?

by Tim at July 22, 2014 01:49 PM


Planet Theory

“How Might Quantum Information Transform Our Future?”

So, the Templeton Foundation invited me to write a 1500-word essay on the above question.  It’s like a blog post, except they pay me to do it!  My essay is now live, here.  I hope you enjoy my attempt at techno-futurist prose.  You can comment on the essay either here or over at Templeton’s site.  Thanks very much to Ansley Roan for commissioning the piece.

by Scott at July 22, 2014 01:27 PM



Play json parse missing field as empty array

Using play-json 2.3 How do i parse this

{i: 0}

into this

case class A(i: Int, s: Seq[Int])

i would very much like to reuse Json.format macro

at the moment it gives me "missing path s"

UPD: i ended up writing a custom format and pimping it onto JsPath:

implicit class JsPathExtensions(path: JsPath) {
  //return mzero if the path is missing
  def lazyFormatNullableM[T: Monoid](f: => Format[T]): OFormat[T] = OFormat(path.lazyReadNullable(f).map(_.orZero), new OWrites[T] {
    override def writes(o: T) = path.lazyWriteNullable(f).writes(o.some)

by OlegYch at July 22, 2014 01:19 PM


How much RAM needed for preventing use swap?

At my system always available some "Free" and "Inactive" memory, but sometimes system move couple megabytes to swap. Per night it maybe around 100MB, but for a few days in swap could be more than 4000MB.

How can I determine required memory for my system?

by moronizzz at July 22, 2014 01:13 PM


Scala type parameter inference fails for Nothing

scala> class A[T]
defined class A

scala> class B[T](a: A[T])
defined class B

scala> val b = new B(new A[Int])
b: B[Int] = B@1ff8982d

Great! I can create instances of B from instances of A. With one exception. For instances of A[Nothing] the type inference fails.

scala> val b = new B(new A[Nothing])
<console>:9: error: type mismatch;
 found   : A[Nothing]
 required: A[T]
Note: Nothing <: T, but class A is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)
       val b = new B(new A[Nothing])

Specifying the type manually works.

scala> val b = new B[Nothing](new A[Nothing])
b: B[Nothing] = B@3aad5958

Should I file a bug for it, or is it an intentional trap to scare away programmers that lack sufficient resolve for Scala?

by Daniel Darabos at July 22, 2014 01:02 PM

update the last element of List

I have a List

val first = List("A","B","C","D")

and I want to create a new list from it but change the last element only:

val newLastVal = "E"
val second = List("A","B","C","E")

can't figure this one out! Thanks in advance

by user3231690 at July 22, 2014 12:58 PM

Indexing Scala Case Classes in GridGain

I am trying to use Scala Case Classes as the object in Gridgain caches. But I cant get it to work. It works perfectly fine if I create exactly the same object as a java-class.

This is my testcase:

class GridGainTest extends FunSuite {
  val grid = GridGain.start("gridgain_mini.xml")
  private val theCache = grid.cache[Long, TestData2]("theCache")

  test("That the basics is working") {
    val tx = theCache.txStart()
    theCache.put(1, new TestData2(1, "Hello", 0))
    theCache.put(2, new TestData2(2, "World", 1))

    val q = theCache.queries().createSqlQuery(classOf[TestData2], "parent = ?")

    val qRes1 = q.execute(new java.lang.Long(1)).get()

    assert(qRes1.size() == 1)

case class TestData(@BeanProperty @GridCacheQuerySqlField(index = true) id: Long,
                    @BeanProperty @GridCacheQuerySqlField(index = true) name: String,
                    @BeanProperty @GridCacheQuerySqlField(index = true) parent: Long)

public class TestData2 {
    @GridCacheQuerySqlField(index = true)
    public final long id;
    @GridCacheQuerySqlField(index = true)
    public final String name;
    @GridCacheQuerySqlField(index = true)
    public final long parent;

    public TestData2(long id, String name, long parent) { = id; = name;
        this.parent = parent;

With TestData2 the test passes, but with TestData it fails on the assert due to size = 0.

I have tried several combinations of property-types (java.lang.Long etc) and annotation-combinations in the case class but nothing seems to work. I must miss something fundamental here, but a few hours of tests from my side obviously doesnt get me all the way.

Any hints?

by Jonas Oum at July 22, 2014 12:43 PM

cond in Clojure with thousands of clauses

Running the following code in Clojure gives a StackOverflow Error:

 (= 1 2) 1
 (= 2 3) 2
 (= 3 4) 3


 (= 1022 1023) 1022
 (= 1023 1024) 1023
 :else 1024)

I would like to create a function/macro that can handle a huge number of clauses without creating a stack that overflows.

Please advise as to how I might attempt this.

by user2179977 at July 22, 2014 12:38 PM



What are practical applications of AI Planning? [on hold]

note: any tips toward making this more constructive will be highly appreciated.

I've dealt with ai-related problems, from searching algorithms, to computer vision, to machine learning. However none of which required planning, just algorithms following algorithms.

I'm not sure how effective will it be to implement AI Planning in practical problems, so I'm curious to know if there's a significant effect in doing so. Like at what point does the programmer/designer realize we need to apply AI Planning to solve this.

by echad at July 22, 2014 12:21 PM



Scala :- Gatling :- Concatenation of two Maps stores last value only and ignores all other values

I have a two Maps and I want to concatenate them.

I tried almost all example given here Best way to merge two maps and sum the values of same key? but it ignores all values for key metrics and only stores last value.

I have downloaded scalaz-full_2.9.1-6.0.3.jar and imported it import scalaz._ but it won't works for me. How can I concate this two maps with multiple values to same keys ?

Edit :-

Now I tried

  val map = new HashMap[String, Set[String]] with MultiMap[String, String]
  map.addBinding("""report_type""" , """performance""")
  map.addBinding("""start_date""" ,start_date)
  map.addBinding("""end_date""" , end_date)
  map.addBinding("metrics" , "plays")
  map.addBinding("metrics", "displays")
  map.addBinding("metrics" , "video_starts")
  map.addBinding("metrics" , "playthrough_25")
  map.addBinding("metrics", "playthrough_50")
  map.addBinding("metrics", "playthrough_75")
  map.addBinding("metrics", "playthrough_100")

  val map1 = new HashMap[String, Set[String]] with MultiMap[String, String]
  map1.addBinding("""dimensions""" , """asset""")
  map1.addBinding("""limit""" , """50""")

And tried to conver this mutable maps to immutable type using this link as

val asset_query_string = map ++ map1
val asset_query_string_map =(asset_query_string map { x=> (x._1,x._2.toSet) }).toMap[String, Set[String]]

But still I get

i_ui\config\config.scala:51: Cannot prove that (String, scala.collection.immutable.Set[String]) <:< (St
ring, scala.collection.mutable.Set[String]).
11:10:13.080 [ERROR] i.g.a.ZincCompiler$ - val asset_query_string_map =(asset_query_string map { x=> (x
._1,x._2.toSet) }).toMap[String, Set[String]]

by swapnil at July 22, 2014 12:10 PM

Planet Emacsen

Irreal: Fast Toggling Emacs Modes

Over at Endless Parentheses, which I’ve mentioned before, Artur Malabarba has a really good idea. He shows how to define a custom key map that he uses to toggle modes that he uses frequently. He binds the commands to the prefix【Ctrl+x t】, an unused key sequence in stock Emacs. Then he has a third key that mnemonically suggest the mode he wants to toggle. Thus,【Ctrl+x t c】toggles column-number-mode and【Ctrl+x t r】toggles read-only-mode and so on.

Oddly, none of the shortcuts that he proposes are ones I care about but I do have a list of my own

l linum-mode
o org-mode
p paredit-mode
t text-mode
v visual-line-mode

As Malabarba says, these aren’t keys I’ll use everyday but they are mnemonic enough that I’ll be able to remember then.

Doubtless you have your own list of modes you’d like to be able to toggle easily and Malabarba’s method makes it easy to bind them to an easily remembered key.

A very useful post, I think. I’m going to implement the above list in my own init.el as soon as I finish writing this.

by jcs at July 22, 2014 12:08 PM


How to provision in parallel and easily manage a multi-tier OpenStack appliction using Ansible


I am trying to provision a multi-tier distributed application on OpenStack private cloud using the Ansible provided modules for Nova and Quantum. This application has a number of moving parts:

  • a Hadoop cluster
  • a MongoDB cluster
  • a web application that makes use of the above

Based on these tiers, I want to:

  • provision a specified number of hosts in several logical groups relating to each tier
  • use different instance flavors based on characteristics of the associated tier
  • create each group in parallel to minimize duration

One thing to note is that the number of instances should be configurable at provisioning time, much the same way that is done for AWS EMR.


Is there a way to perform the following sketch of actions in Ansible assuming a tier group of 5:

< start all 5 at once >
   < start up openstack host >
   < assign it a floating ip >
   < capture the floating ip >
< end when all 5 have a floating ip >
< wait for all 5 floating ips to have an open SSH port >
< use all floating ips within a group to issue more actions >

The final step is important as it allows for a join point to apply further processing to a named group.


There is a thread that discusses parallelism in some detail, but fails to offer any resolution of the final step in the sketch outlined above.

For the benefit of the reader, I have compiled a small set of related examples that are similar to what I am trying to achieve in that they make use of the Nova and Quantum module actions (which alone are quite scarce). However, they do not seem to leverage parallelism or coordinate multiple groups of hosts:

by btiernay at July 22, 2014 11:53 AM

Planet Theory

TR14-091 | One time-travelling bit is as good as logarithmically many | Ryan O&#39;Donnell, A. C. Cem Say

We consider computation in the presence of closed timelike curves (CTCs), as proposed by Deutsch. We focus on the case in which the CTCs carry classical bits (as opposed to qubits). Previously, Aaronson and Watrous showed that computation with polynomially many CTC bits is equivalent in power to PSPACE. On the other hand, Say and Yakary?lmaz showed that computation with just 1 classical CTC bit gives the power of “postselection”, thereby upgrading classical randomized computation (BPP) to the complexity class BPP path and standard quantum computation (BQP) to the complexity class PP. It is natural to ask whether increasing the number of CTC bits from 1 to 2 (or 3, 4, etc.) leads to increased computational power. We show that the answer is no: randomized computation with logarithmically many CTC bits (i.e., polynomially many CTC states) is equivalent to BPP_path. (Similarly, quantum computation augmented with logarithmically many classical CTC bits is equivalent to PP.) Spoilsports with no interest in time travel may view our results as concerning the robustness of the class BPP_path and the computational complexity of sampling from an implicitly defined Markov chain.

July 22, 2014 11:48 AM


When should I use function currying in Python?

When should I write my functions in curried form? does not match my thought, need to correct myself.

As part of my learning link, this is what I understand from function currying. Below is one example:

def curry2(f):
    """Returns a function g such that g(x)(y) == f(x, y)

    >>> from operator import add
    >>> add_three = curry2(add)(3)
    >>> add_three(4)
    def g(x):
        def h(y):
            return f(x, y)
        return h
    return g

In any application, if I know that the number of arguments are fixed (say 2 arguments) and function name is normalise_range(say), then I will define def normalise_range(x, y): function and use it in my application directly by calling normalise_range(x, y).

In any application, if I know that, the number of arguments are fixed (say 2 arguments), but the function name is varying (can be normalise_range/average/I don't know..), then I will use def curry2(f): as shown above, which will accept all functions that take two arguments (fixed).

My question:

  1. Is my understanding correct?
  2. If yes, can we think of currying for functions of variable number of arguments?

by overexchange at July 22, 2014 11:19 AM


Finding a node that is minimal w.r.t. maximum distance to any other node [on hold]

Given a graph and an edge in the graph, I want to find a point on this edge that makes the maximum distance from all the nodes to this point minimized. Is there any graph-theory based algorithm that can solve such problem rather than brute force?

For example we have a graph:

1 2 10
2 3 10
3 4 1
4 1 5

In every line the first two values indicate the node number, and the 3rd one is the corresponding edge value. Now I want to find a point on the edge 1-2, the answer is a point with distance of 2 to node 1 (8 to node 2). And 1-p distance is 2, 2-p is 8, 3-p is 1+5+2=8, 4-p is 5+2=7. The maximum path is 8, which is the minimal among all the possible points p on edge 1-2.

by user3153970 at July 22, 2014 11:14 AM


How to avoid adding a 'root' scala sbt project in IntelliJ when defining github dependencies?

I'm using IntelliJ with SBT plugin and I've added the following lines to build.sbt in order to add a dependency to a private github repository:

lazy val g = RootProject(uri("ssh://"))

lazy val root = project in file("myproject") dependsOn g

After running sbt, my referenced project is successfully cloned, but build fails due to output path clashes between my base project and a root project that is automatically added each time I refresh sbt after modifying build.sbt.

by mi.iwanowski at July 22, 2014 11:12 AM

What is coming up for scala in 2.10? [closed]

Obviously there is the reflection library but what else can we expect to see in the next release of scala? Any language changes? Is there a website where I can see a roadmap?

by oxbow_lakes at July 22, 2014 11:08 AM


If Halting problem is decidable, are all RE languages decidable? [on hold]

Assume the halting problem was decidable. Is then every recursively enumerable languagerecursive?

by umang at July 22, 2014 11:04 AM


How to find the "hard" probability distribution on the input for recursive boolean functions?


Decision tree complexity or query complexity is a simple model of computation defined as follows. Let $f:\{0,1\}^n\to \{0,1\}$ be a Boolean function. The deterministic query complexity of $f$, denoted $D(f)$, is the minimum number of bits of the input $x\in\{0,1\}^n$ that need to be read (in the worse case) by a deterministic algorithm that computes $f(x)$. Note that the measure of complexity is the number of bits of the input that are read; all other computation is free.

We define the bounded error or Las Vegas randomized query complexity of $f$, denoted $R_0(f)$, as the minimum number of input bits that need to be read in expectation by a zero-error randomized algorithm that computes $f(x)$. A zero-error algorithm always outputs the correct answer, but the number of input bits read by it depends on the internal randomness of the algorithm. (This is why we measure the expected number of input bits read.)

We define the bounded error or Monte Carlo randomized query complexity of $f$, denoted $R_2(f)$, to be the minimum number of input bits that need to be read by a bounded-error randomized algorithm that computes $f(x)$. A bounded-error algorithm always outputs an answer at the end, but it only needs to be correct with probability $\geq$ $1 - \delta$ ($2/3$, say).

Work on Recursive Boolean Functions:

There has been a line of work on the decision tree complexity of recursive boolean functions as mentioned below. The techniques focus on applying Yao's Lemma and using the distributional perspective guaranteed by it. This means we define a probability distribution on the inputs and the cost incurred by the best algorithm for this distribution gives a lower bound on the randomized decision tree complexity of the function. The worst possible distribution will give the actual randomized decision tree complexity.

The techniques in these works focus on giving a lower bound on the cost incurred by reading the "minority" bits (or vertices in the function tree) of the input via some form of induction. Another direction of attack could be to find the most "hard" distribution.

Some Notions

We define: The distribution $D^*$ on an input set $I$ is hard for a given function $f$, if $\forall D$ on $I$, $C(A,D) \leq C(A^*, D^*) $, where $C(A,D)$ is the expected cost (i.e. number of input bits read on expectation) incurred by the deterministic decision tree $A$ when the input follows the probability distribution is $D$. where $A^* = \operatorname{argmin}_A C(A, D^*), A = \operatorname{argmin}_A C(A, D)$

A distribution $D_1 < D_2$, if $C_m(D_1) < C_m(D_2)$, where $C_m(D_i) = C(A_i, D_i)$, and $A_i = \operatorname{argmin}_A C(A, D_i)$ . In other words $D_2$ is harder than $D_1$ means the best possible algorithm for $D_2$ does worse than the best possible algorithm for $D_1$. Note: The algorithm must be correct in the whole domain, and not just in the support of the distributions. For the base case of a recursive boolean function like say 2 bits or 4 bits, it is often easy to show a certain distribution to be hard. Often it is an easy observation or an obvious fact. In many cases, it may seem natural that the "hard" distribution is the recursive extension of the hard distribution. However, this may not be true in general, especially if the function is not symmetric over the input bits and rather skewed i.e. not all input bits are equally important to infer the value of the function on certain inputs (or a subset thereof).


Is there any work on how to approach the problem of finding the "hard" distribution of the recursive function, from that of the base case function?

Is there any interesting connection of this problem with any other problems? Any comments are welcome.


[1] M. Saks, A. Wigderson Probabilistic Boolean Decision Trees and the Complexity of Evaluating Game Trees Proceedings of the 27th Foundations of Computer Science, pp. 29-38, October 1986.

[2] Frédéric Magniez, Ashwin Nayak, Miklos Santha, and David Xiao. Improved bounds for the randomized decision tree complexity of recursive majority. In Luca Aceto, Monika Henzinger, and Jiri Sgall, editors, ICALP (1), volume 6755 of Lecture Notes in Computer Science, pages 317–329. Springer, 2011.

[4] Nikos Leonardos. An improved lower bound for the randomized decision tree complexity of recursive majority. In Fedor V. Fomin, Rusins Freivalds, Marta Z. Kwiatkowska, and David Peleg, editors, Proceedings of 40th International Colloquium on Automata, Languages and Programming, volume 7965 of Lecture Notes in Computer Science, pages 696{708. Springer, 2013.

by Jardine at July 22, 2014 11:03 AM


Play Framework: Netty fails to bind after recompile

I am working on a project in the Play Framework using Akka and Memcached . When I change the java source in the IDE, play automatically re compiles and reruns/reloads. There is code in our project that looks for a free port for Akka. It starts at 40000. Tries to open and close a socket on the port. On success, it constructs a configuration for Akka including found available ports. But later this initialization fails throwing this exception:

 Caused by: Failed to bind to: /

 at org.jboss.netty.bootstrap.ServerBootstrap.bind(
 at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:377)
 at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:374)
 at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
 at scala.util.Try$.apply(Try.scala:161)
 at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
 at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
 at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
 at akka.dispatch.BatchingExecutor$
 at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
 at scala.concurrent.forkjoin.ForkJoinTask.doExec(
 at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(
 at scala.concurrent.forkjoin.ForkJoinPool.runWorker(

 Caused by: Address already in use

 at Method)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(
 at java.util.concurrent.ThreadPoolExecutor$

Strange thing is that this code works on Ubuntu, but does not on Mac OSX (Maverics). Any idea would be most appreciated.

by user3187264 at July 22, 2014 11:02 AM


Habt ihr das auch gehört, wie würdelos die fiesen ...

Habt ihr das auch gehört, wie würdelos die fiesen Rebellen mit den Leichen der Flugzeugabschussopfer umgegangen sind? Fies und Würdelos! Das hat Kerry aus Washington diagnostiziert und die Weltpresse hat es unhinterfragt berichtet.

Auch der ORF hat dann mal ihren Korrespondenten vor Ort angerufen, und wollte wissen, ob die fiesen Rebellen endlich aufgehört haben, die Leichen so würdelos zu behandeln. Das Interview lief dann nicht ganz so, wie sich die ORF-Redakteure sich das vorgestellt hatten.

July 22, 2014 11:02 AM


Scala Ordered Mixin Base Class

In Scala 2.10.2, I have an abstract base class for a major-minor style version number:

abstract class MajorMinorVersion( 
    private [version] val major : Int,
    private [version] val minor : Int
) extends Ordered[MajorMinorVersion] 
    override def toString() : String = "%d.%d".format( major, minor )

    def compare( that : MajorMinorVersion ) = that.major == major match
        case false => major - that.major
        case true => minor - that.minor

Several custom file types I'm reading have a version number in this format:

case class FooFileVersion( maj : Int, min : Int ) extends MajorMinorVersion( maj, min )

case class BarFileVersion( maj : Int, min : Int ) extends MajorMinorVersion( maj, min )

This works nicely in that:

scala> FooFileVersion( 1, 3 ) < FooFileVersion( 1, 4 )
res0: Boolean = true

The problem with this approach is that I'd like comparison of FooFileVersion and BarFileVersion instances to throw an error, because comparing them doesn't make sense, but currently we have:

scala> FooFileVersion( 1, 3 ) < BarFileVersion( 1, 4 )
res0: Boolean = true

What's the best way to get around this? Is it possible to have the Ordered trait mixed in to the base class, or will I need to mix it in to the inheriting classes, and define the compare method there? That would be a solution, I guess, but it would be nice to be able to just abstract the comparison away into a base class.

I'm sure there's a relatively simple solution somewhere I'm just missing. Thanks in advance!

by paulmdavies at July 22, 2014 10:57 AM


Limit on no. of emails I can RECEIVE on my free gmail account

I would like to know what's the limit on number of emails I can receive on my Gmail account( a free gmail account that ends with in a day! Thanks in advance. Please provide help ASAP.

by Stupid Amigo at July 22, 2014 10:53 AM

What is the signal going from the computer to the screen?

Im sorry if it is a weird question, but I cant seem to find an answer for this. What does the signal going to the screen look like in more abstract sense? Does it code the values of each pixel in a huge matrix, is it a computational stream that instructs the screen what to present?

In other words, can I grab this signal by a "listening" device for example and deconstruct it with algorithms? Not just grab the signal and present it on another screen. For example, if it is a matrix of pixel values, then I can run face recognition algorithms for example, to see if the signal sent to the computer is a picture of face or not.

It might also depend on the type of screen? Say that it is a LCD screen.

edit: To elaborate, if I wanted to take the signal sent to the screen and scan it for text (NSA style), could I? If so, what should I know of the signal that Im collecting to actually do this? Disclaimer: Im not interested in spying on anyone.

by Sevenius at July 22, 2014 10:47 AM


Trying to get a lazy-contains? function in Clojure to return results consistant with contains? when dealing with a map

Here's a use of the standard 'contains?' function in Clojure-

(contains? {:state "active", :course_n "law", :course_i "C0"} :state)

and it returns the expected


I used the following

Clojure: Idiomatic way to call contains? on a lazy sequence

as a guide for building a lazy-contains? as this is what I need for my present use-case.

The problem I'm facing is that for a map these alternatives are not returning the same answer, giving either a false or a nil response. I've tried looking at the source for contains? and it's slow going trying to understand what's happening so I can correct the lazy-contains? appropriately (for the record Clojure is essentially my first programming language, and my exposure to Java is very limited).

Any thoughts or ideas on how I might approach this? I tried every variant on the linked question I could.

Thanks in advance.

by RatavaWen at July 22, 2014 10:30 AM


Does a NFA accept context free languages? [on hold]

A NFA accepts regular languages (Type-3). So because of the chomsky hierarchy every regular language is also context-free. So a NFA accepts context free languages. Am I wrong? Moreover a NFA accepts through definition every language.

by user22709 at July 22, 2014 10:21 AM

TCP & UDP time taken/retransmission calculation

What i know is that, TCP provides reliable point-to point communication, whereas UDP doesn't establish a connection before sending data, it just sends.

For this question, assuming that for UDP-IPv4 communication the overhead is 66bytes, and TCP-IPv4 overhead is 78 bytes.

And for the calculation in IPv6, i came out with two option.

Option 1:

Version- 4bits Traffic Class- 8bits Flow label- 20 bits Payload Length- 16 bits Next header- 8 bits Hop Limit- 8 bits Source Address- 128 bits Destination Address- 128 bits

Total = 320 bits = 40 bytes

TCP over IPv6: 20 bytes (TCP) + 40 bytes (IPv6) = 60 bytes

Option 2: By using the overhead of IPv4 (78 bytes) + TCP (20 bytes) = 98 Bytes

The reason why i have option 2 is because IPv6 is 20 bytes larger than IPv4.

So i'm not sure which option is valid. Can anyone enlighten me?

by June at July 22, 2014 10:20 AM


How to Embed a jLine element within TextArea in Scala Swing

I want to embed the interactive shell within a TextArea element of swing application. Why I can't do the following:

object embedInteraction extends SimpleSwingApplication {  
  def top = new MainFrame { contents = gui }      
  val gui = new BoxPanel(Orientation.Vertical) {
     //Other GUI Elements
  def Terminal { 
    val comGetter = new jline.console.ConsoleReader()
    while(!comGetter == "q"){ //Deals with other commands here }  

by Dave Rose at July 22, 2014 10:07 AM



Problem with the Python REPL in emacs

Let's say I have the following python code:


from b import test def test_a(): test() 


def test(): print "b" 

Now I open and press C-c C-c. It opens the python REPL and I can run test_a() inside. If I go to the file, replace print "b" by print "bb", and press C-c C-c, I can run test_a() in the REPL and I get "bb". For now it's all fine.

Now if I go back to, and press C-c C-c, I get "b" while running test_a() in the REPL. The REPL gets the old version of the file b.y.

Has someone already experimented that problem?

submitted by eskatrem
[link] [6 comments]

July 22, 2014 09:32 AM



Applications of Harrow's algorithm for solving linear equations

In Harrow's algorithm for solving a system of linear equations the output is a quantum state rather than explicit information. Has anyone been able to apply knowledge of this quantum state to solve a problems which would classically use the solutions of the linear equations?

by Sam Burville at July 22, 2014 09:00 AM



What's the essential similarities and differences between Scala and Gosu (related to Java)?

I am studying Scala right now, and have been doing so for the last month. Today, (via Twitter) I stumbled upon another language that seems to be quite similar, Gosu.

What are the essential similarities and differences between Scala and Gosu with respect to their most influential parent, Java? Are there any web-sites that compare and contrast these two languages?

by chaotic3quilibrium at July 22, 2014 08:52 AM

reading cookie value in play framewrok 2.2 scala template

I am trying to read a cookie value inside the play framework template (not inside a controller). I am trying the following which is not working:

@ val cookieVal = request.cookies.get('PLAY_SESSION').value

Any suggestions to fix this will be greatly appreciated. The reason why I am trying this is to change how the page gets rendered based on a cookie value.

by ali haider at July 22, 2014 08:51 AM

Spark Streaming Stateful Network Word Count

This is the example code that came with Spark. I copied the code here and this is the link to it: However, when I was trying to run the program using command "bin/run-example org.apache.spark.examples.streaming.StatefulNetworkWordCount localhost 9999", I was given the following error:

14/07/20 11:52:57 ERROR ActorSystemImpl: Uncaught fatal error from thread [] shutting down ActorSystem [spark] java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; at org.apache.spark.streaming.scheduler.JobScheduler.getPendingTimes(JobScheduler.scala:114) at org.apache.spark.streaming.Checkpoint.(Checkpoint.scala:43) at org.apache.spark.streaming.scheduler.JobGenerator.doCheckpoint(JobGenerator.scala:259) at$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:167) at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1$$anonfun$receive$1.applyOrElse(JobGenerator.scala:76) at at at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec( at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask( at scala.concurrent.forkjoin.ForkJoinPool.runWorker( at Exception in thread "Thread-37" org.apache.spark.SparkException: Job cancelled because SparkContext was shut down at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:639) at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:638) at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:638) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1215) at$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201) at$class.terminate(FaultHandling.scala:163) at at$1(ActorCell.scala:431) at at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:240) at at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec( at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask( at scala.concurrent.forkjoin.ForkJoinPool.runWorker( at 14/07/20 11:53:00 ERROR Executor: Exception in task ID 0 java.lang.IllegalStateException: cannot create children while terminating or terminated at$class.makeChild(Children.scala:184) at$class.attachChild(Children.scala:42) at at at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.(ReceiverSupervisorImpl.scala:67) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:263) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1080) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1080) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) at at org.apache.spark.executor.Executor$ at java.util.concurrent.ThreadPoolExecutor.runWorker( at java.util.concurrent.ThreadPoolExecutor$ at 14/07/20 11:53:06 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[,5,main] org.apache.spark.SparkException: Error sending message to BlockManagerMaster [message = HeartBeat(BlockManagerId(,, 47668, 0))] at at at$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:113) at$$anonfun$initialize$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(BlockManager.scala:158) at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:790) at$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:158) at$$anon$ at$$anon$3$$anon$ at$ at$$anonfun$close$1.apply(Scheduler.scala:281) at$$anonfun$close$1.apply(Scheduler.scala:280) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at at at$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582) at$$anonfun$_start$1.apply(ActorSystem.scala:582) at$$anonfun$_start$1.apply(ActorSystem.scala:582) at$$anon$ at$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750) at$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753) at$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746) at$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746) at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15) at$ at$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593) at$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593) at at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$ at at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec( at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask( at scala.concurrent.forkjoin.ForkJoinPool.runWorker( at Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://spark/user/BlockManagerMaster#1887396223]] had already been terminated. at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) at


object StatefulNetworkWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: StatefulNetworkWordCount <hostname> <port>")


    val updateFunc = (values: Seq[Int], state: Option[Int]) => {
      val currentCount = values.foldLeft(0)(_ + _)

      val previousCount = state.getOrElse(0)

      Some(currentCount + previousCount)

    val sparkConf = new SparkConf().setAppName("StatefulNetworkWordCount")
    // Create the context with a 1 second batch size
    val ssc = new StreamingContext(sparkConf, Seconds(1))

    // Create a NetworkInputDStream on target ip:port and count the
    // words in input stream of \n delimited test (eg. generated by 'nc')
    val lines = ssc.socketTextStream(args(0), args(1).toInt)
    val words = lines.flatMap(_.split(" "))
    val wordDstream = => (x, 1))

    // Update the cumulative count using updateStateByKey
    // This will give a Dstream made of state (which is the cumulative count of the words)
    val stateDstream = wordDstream.updateStateByKey[Int](updateFunc)

I wonder if it is because, it is trying to set up the checkpoint at my local file system by doing commands "ssc.checkpoint(".")", while the file is not a file compatible with hadoop? (the file must be compatible with hadoop in order to set the checkpoint) If it is, how could I fix it? Thanks!

by user2895478 at July 22, 2014 08:47 AM


TCP & UDP time taken/retransmission calculation

What i know is that, TCP provides reliable point-to point communication, whereas UDP doesn't establish a connection before sending data, it just sends.

For this question, assuming that for UDP-IPv4 communication the overhead is 66bytes, and TCP-IPv4 overhead is 78 bytes.

And for the calculation in IPv6, i came out with two option.

Option 1:

Version- 4bits Traffic Class- 8bits Flow label- 20 bits Payload Length- 16 bits Next header- 8 bits Hop Limit- 8 bits Source Address- 128 bits Destination Address- 128 bits

Total = 320 bits = 40 bytes

TCP over IPv6: 20 bytes (TCP) + 40 bytes (IPv6) = 60 bytes

Option 2: By using the overhead of IPv4 (78 bytes) + TCP (20 bytes) = 98 Bytes

The reason why i have option 2 is because IPv6 is 20 bytes larger than IPv4.

So i'm not sure which option is valid. Can anyone enlighten me?

by June at July 22, 2014 08:44 AM


Designing hardware and software for Nest thermostat [on hold]

For the Nest thermostat, what hardware is needed to build one of those devices (ie touchscreen)? I will try to be as specific as possible. For the components of the nest thermostat ( shown in this image : ) could someone explain what each part is called and some of the functions of each part?

I have updated this question to make it more specific.

by Jacob Smith at July 22, 2014 08:41 AM

Receiving "bad symbolic reference to org.json4s.JsonAST..." in Eclipse Scala-IDE using 2.11

I am attempting to use the Eclipse Scala-IDE for 2.11 (downloaded the prepackaged bundle from the web-site). I have been using the Scala Worksheet to work with a SaaS API returning JSON. I've been pushing through just using String methods. I decided to begin using json4s. I went to and obtained the following libraries:

  • json4s-core-2.11-3.2.10.jar
  • json4s-native-2.11-3.2.10.jar
  • paranamer-2.6.jar

I have added all three jars to the Project's Build Path. And they appear under the project's "Referenced Libraries".

I have the following code in a Scala Worksheet:

package org.public_domain

import org.json4s._
import org.json4s.native.JsonMethods._

object WorkSheet6 {
  println("Welcome to the Scala worksheet")
  parse(""" { "numbers" : [1, 2, 3, 4] } """)

I am receiving the following two compilation errors:

  • bad symbolic reference to org.json4s.JsonAST.JValue encountered in class file 'package.class'. Cannot access type JValue in value org.json4s.JsonAST. The current classpath may be missing a definition for org.json4s.JsonAST.JValue, or package.class may have been compiled against a version that's incompatible with the one found on the current classpath.
  • bad symbolic reference to org.json4s.JsonAST.JValue encountered in class file 'package.class'. Cannot access type JValue in value org.json4s.JsonAST. The current classpath may be missing a definition for org.json4s.JsonAST.JValue, or package.class may have been compiled against a version that's incompatible with the one found on the current classpath.

When I go look in the org.json4s package in the json4s-core-2.11-3.2.10.jar file, there is in fact no .class file indicating any sort of compiled object JsonAST.

This is a showstopper. Any help on this would be greatly appreciated.

by chaotic3quilibrium at July 22, 2014 08:33 AM

code completion for liftweb scala in intellij idea

I'm working with a Scala application using Intellij Idea 13.1.1 community edition. Although i am able to compile and run my application, but the IDE shows error in editor window as it cannot resolve the class. How can i remove that and enable code completion feature for Scala and Liftweb?

by Abinash Bastola at July 22, 2014 08:32 AM

How in shapeless do you say that a proof is the empty type (i.e. false)

I want to write a function that take a Nat as parameter, and return this nat ONLY if this nat isn't divisible by three.

for example :

def myFunction[N <: Nat](n :N)(implicit ev: /* what do I put here that say not divible by 3 ? */): N = n

To do that, I have to write something that say "N is not divisible by _3", or "Mod.Aux[N, _3,_0] is the empty type"

how can I do that in shapeless ?

by Molochdaa at July 22, 2014 08:23 AM

Waiting on multiple Akka FSM messages

I have an Akka FSM actor that runs the following pseudocode after receiving a message while in ReadyState

lookupA ! Wrapper(Lookup("A"))
lookupB ! Wrapper(Lookup("B"))
lookupC ! Wrapper(Lookup("C"))
goto(LookingUpDataState) using DataFound(a = None, b = None, c = None)

The actor then waits for responses which can be either FullResult[T] (extending ServiceResult[T]) or Empty (extending ServiceResult[Nothing]). Successful lookup results are used to populate the DataFound instance's fields and Empty lookup results result in a logged error message and the termination of the actor.

My question is this: how can I determine which lookup failed, so that I can log the failure or fallback to a default value? All I can think of is examining the sender's ActorRef (hacky) or adding a unique ID field to all messages (high overhead).

This is a simple problem to solve using Ask's and Futures. Does an idiomatic Akka solution exist?

by pkinsky at July 22, 2014 08:09 AM


In einem Experiment über Lügen und Bescheißen in ...

In einem Experiment über Lügen und Bescheißen in Berlin kam eine starke Korrelation zwischen Bescheißen und "im Sozialismus aufgewachsen" heraus. Die sollten würfeln und die Zahl oben oder auf der Rückseite nehmen, sich aber vor dem Würfeln festgelegt haben. Die Studie hat das aber nicht überprüft, ob die sich vorher festlegen. Und dann haben sie geguckt, ob die kleinen und großen Zahlen gleichverteilt sind. Waren sie nicht.
The authors found that, on average, those who had East German roots cheated twice as much as those who had grown up in West Germany under capitalism. They also looked at how much time people had spent in East Germany before the fall of the Berlin Wall. The longer the participants had been exposed to socialism, the greater the likelihood that they would claim improbable numbers of high rolls.
Wieso muss ich denn über dieses Experiment im Economist lesen und nicht in der deutschen Presse?

Update: Die Thüringer Allgemeine hatte was.

July 22, 2014 08:02 AM


How append, prepend, and generally insertAt work in RRB-tree

I read the paper about Relaxed Radix Balanced trees (RRB trees) and am trying to implement them. What I can't get is how insertion at an index should be performed step by step. Can anyone proficient in this data structure describe this procedure?

by Tvaroh at July 22, 2014 07:53 AM


ClrZmq returning messages always to first started client

We're creating a WPF app in which we execute python scripts from different Test Stations and show the output in its corresponding output panel, To run the scripts in parallel we are using Task but when we run the scripts in parallel from the stations, We are getting the output of other stations also into the station that is started first, we're using the following code,

private void ZmqStatusListener(string endPoint)
    using (Context context = new Context())
        StatusPort = string.Empty;
        TestResultPort = string.Empty;
        using (Socket server = context.Socket(SocketType.REP))
                if (isStatusContextActive == false || isPortChanged == true)
                    isStatusContextActive = true;
            catch (ZMQ.Exception ex)
                if (ex.Errno != 100)
                    string IPCPort = _globalParameters.GlbParam.GlbParamIpcStartPort;
                    if (IPCPort == string.Empty)
                        IPCPort = "0";
                    if (endPoint == EditorConstants.PortAddress.PortPrefix + IPCPort)
                        StatusPort = endPoint;
                        TestReultError = EditorConstants.CommonMessageTypes.TestReultError + ex.Message + EditorConstants.CommonMessageTypes.StackTraceMessage + ex.StackTrace;
                    isCancelledtask = true;
                    ScriptStatusDesc = new ScriptStatusDesc()
                        Status = "Failed",
                        statusDescription = "Failed"
            while (true)
                string message = server.Recv(Encoding.UTF8);
                server.Send(" ACK", Encoding.UTF8);
                // if (message == "Test Passed")

and for testing purpose we're breaking the while loop in this code based on a test message we kept in the python script, then we are able to get the output in the respective station correctly but this way we can only run in a synchronous fashion which we don't want as we require to run the test stations in parallel and the while loop should not break as it should be listening for the response.

by Mohammed Abdul Mateen at July 22, 2014 07:49 AM


Video lectures on type systems

For my job, I need to pick up a working understanding of the implementation of type systems (in particular, how to write typing rules based on a design document). I've been given a copy of Types and Programming Languages and that's great, but I'd really like to watch someone work through it.

In my school, the course that teaches this is called "Programming Languages," but searching for "video lectures programming languages" turns up lots of stuff about how to learn C++, which I'm not interested in. I couldn't find anything on MIT Open Courseware.

I'd really like video lectures that go through TAPL, but I'll settle for anything useful. Are there good resources out there?

by Patrick Collins at July 22, 2014 07:49 AM


Suggestions on a short piece of Haskell code?

I'm very new to Haskell and as a way of learning (first project) I am porting code from Tcl to Haskell. The project is a domain specific query language which is translated to SQL queries via a semantic layer. Now, with the operators of the query language restricted in the parser for the language, code similar to this actually seems more silly in Haskell than it did in Tcl :-)

type MatchOperator = [Char]

getOpJoiner :: MatchOperator -> String
getOpJoiner "!=" = " and "
getOpJoiner "!~" = " and "
getOpJoiner "!~~" = " and "
getOpJoiner _  = " or "  -- In reality, this can only be "=", "~", "~~" or "==" according to the parser

The code does what is should do (return the appropriate SQL logical joiner of statements depending on the query operator used), but I'm sure it can be made more pretty.

by Fredrik Karlsson at July 22, 2014 07:43 AM


The stable marriage algorithm with asymmetric arrays

I have a question about the stable marriage algorithm, for what I know it can only be used when I have arrays with the same number of elements for building the preference and the ranking matrices.

For example if we have 10 students that should be assigned to 10 dorms, then I can build the preference and ranking matrix of this data. The question that I have is what to do in the case that I have, for example, only 5 students to assign and for example 10 dorms. Can I still apply the stable marriage algorithm?

Maybe this question is a little bit foolish, but as I said I only saw this algorithm applied to quadratic arrays.

by Little at July 22, 2014 07:42 AM


Testing the validity of a factor model for stock returns

Consider the following m regression equation system:

$$r^i = X^i \beta^i + \epsilon^i \;\;\; \text{for} \;i=1,2,3,..,n$$

where $r^i$ is a $(T\times 1)$ vector of the T observations of the dependent variable, $X^i$ is a $(T\times k)$ matrix of independent variables, $\beta^i$ is a $(k\times1)$ vector of the regression coefficients and $\epsilon^i$ is the vector of errors for the $T$ observations of the $i^{th}$ regression.

My question is: in order to test the validity of this model for stock returns (i.e. the inclusion of those explanatory variables) using AIC or BIC criterion, should these criterion be computed on a time-series basis (i.e. for each stock), or on a cross-sectional basis (and then averaged over time)?

by Mariam at July 22, 2014 07:40 AM


How do i use play ws library in normal sbt project instead of play?

When i tried using Play WS library in a normal sbt project instead of play project I was bound to use play.api.Play.current and got java.lang.RuntimeException: "There is no started application" when tried to run the application.

by pamu at July 22, 2014 07:30 AM

Play! framework: define a variable in template?

I'm passing to a template an Event object and what I need to do is checking @event.getSeverity value. if the value is positive, I want to color a specific <div> in green. if the value is negative I want to color a specific <div> in red.

I couldn't find a way to define a variable. is it possible? it should be I think.
anyhow, what's the simplest way accomplishing this?


by socksocket at July 22, 2014 07:25 AM

how can i get all row key from column family in cassandra cli?

fetch all rowkey from column family in cassandra cli using hector

val query=HFactory.createRangesliceQuery(keyspace,le,se,se)


val query=HFactory.createRangesliceQuery(keyspace,le,se,se)

by Rohit Sharma at July 22, 2014 07:21 AM

scala getClass and type erasure?

I am a little puzzled by the getClass method when used in generic type case:

def a[K](key: K) = {

Shouldn't this always been AnyRef or Any or something? Since type K info is not available in runtime?

but a(3) will output Int , and a("fdasf") will output String.

Alright, this is a silly question, I get confused by the K and key's type information.

by Raymond at July 22, 2014 07:06 AM


Can I have a "dependent coproduct type"?

I'm reading through the HoTT book and I have a (probably very naive) question about the stuff in the chapter one.

The chapter introduces the function type $$ f:A\to B $$ and then generalizes it by making $B$ dependent on $x:A$ $$B:A\to\mathcal{U},\qquad g:\prod_{x:A}B(x)$$ and that is called the dependent function type.

Moving on, the chapter then introduces the product type $$ f:A\times B$$ and then generalizes it by making $B$ dependent on $x:A$ $$B:A\to\mathcal{U},\qquad g:\sum_{x:A}B(x)$$ and that is called the dependent pair type.

I can definitely see a pattern here.

Moving on, the chapter then introduces the coproduct type $$ f:A+B$$ and ... combobreaker ... there is no discussion of dependent version of this type.

Is there some fundamental restriction on that or it is just irrelevant for the topic of the book? In any case can someone help me with intuition on why function and product types? What makes those two so special that they get to be generalized to dependent types and then used to build up everything else?

by Kostya at July 22, 2014 07:03 AM

Satisfy edges' constraints when updating node in directed acyclical graph

I have a directed acyclical graph. Each node represents an event with start and end dates and each edge represents a constraint between to events with 2 properties:

  • max interval between previous event end and next event start
  • min interval between previous event end and next event start

When an event's date is updated, all edge constraints should be respected and every other event's date should be recalculated if those constraints are violated. Problem is that there might be multiple conflicting constraints and I'm struggling to find best way to traverse the graph updating events without breaking previously satisfied constraints.

I have no formal CS education and my knowledge in this area is quite limited. Is there an existing algorithm to solve my problem?

by package at July 22, 2014 07:02 AM


Which interest rate model for which product

Given the multitude of existing interest rate models (ranging from simple to very complex) it would be interesting to know when the additional complexity actually makes sense.

The models I have in mind:

  • Simple one factor (e.g. HW)
  • Two factor models
  • Two or one factor models with stochastic volatility
  • LIBOR-Market, HJM, SABR and SABR-LMM model

Are there any rules of thumb to decide which model to use for which product? (Perhaps there is some book dealing with that topic that I am not aware of)

Edit 22.02.2014:

While trying to answer the question myself I found the follwing very interesting paper on the emperical comparison of interest rate models. Here the authors mainly compare how well the different models can fit market data and hit the relevant market pries after being calibrated.

Thus a follow up Question: (that is also related to the question on model validation)

Does it suffice to hit the market pries spot on after calibration for a model to 
qualify for being used in pricing for an instrument ? Or are there other aspects to be
considered? (computational speed, statistical fittness, robustness of the hedges) 

Let's assume my model fits the market data really well - backtesting however shows that the hedges it provides don't work that well. Also the model might not be able to statistially fit the path of the underlying. E.g. mean reversion can be observed in some markets but not in others. One could argue that risk neutrality does not necessarily entail meaningful real-world scenarios etc.

I have good theoretical grasp of the models but have mainly used them for risk management (thus generating paths and analysing what happens to a portfolio or the balance sheet of an enterprise)

by Probilitator at July 22, 2014 06:58 AM


Temporary key binding

Sometime when I do very specific task, I would like to create temporary key-binding so that, during that task, I can easily do repeated functions executed. Once that task is done, restore back old key binding. This keys can be key-chord as well. Is there package available in that use case ?

submitted by sudurais
[link] [5 comments]

July 22, 2014 06:56 AM

Fred Wilson


Yesterday our portfolio company Flurry announced it was being acquired by Yahoo!

I thought I’d provide a bit of history since this was an interesting investment for us.

Back when Apple was launching its app platform in the winter of 2008, we met with Greg Yardley who had teamed up with Jesse Rohland to build an analytics service for app developers. We had known Greg from his work with Seth Goldstein at Root and we were fans. And it seemed to be a smart idea to give developers the ability to see what people were doing in their mobile apps. So we provided seed financing to Greg and Jesse along with our friends at First Round.

Pinch launched the first iOS analytics service and got rapid adoption. But they ran into some challenges, the two primary ones were monetization and getting onto Android and Blackberry (which was relevant back then). And that’s where Flurry entered the picture.

Flurry was a pivot into the same business as Pinch was in. They were already on Android and Blackberry but were far behind Pinch on iOS. They were led by a hard charging CEO named Simon Khalaf who had big ideas for monetization. It was a match made in heaven. So the two companies merged and Flurry became the surviving company.

Flurry continues to lead the mobile app analytics business. According to Simon’s blog post yesterday, there are 170,000 developers with 542,000 mobile apps using the Flurry service.

And now Flurry becomes a Yahoo! branded offering. There is no question that the Flurry data and its advertising products (powered by Flurry’s data) will be a great fit for Yahoo!’s mobile ambitions.

So we have a happy ending to a startup story with a few twists and turns. This is an example of where 1+1 equaled a lot more than two. I’ve been involved in a number of “startup mergers”. Some work. Some don’t. This one worked beautifully.

by Fred Wilson at July 22, 2014 06:50 AM


Scala path-dependent type: testing for type error [duplicate]

This question already has an answer here:

I've written some Scala to work with modular arithmetic using path dependent types. The parent class represents a group modulo some integer, and the inner class represents elements of this group.

class ModularInt( modulus : Int ) {
  case class Value( value: Int ) {
    def plus( a: Value ) : Value = {
      val rem: Int = (value + a.toInt) % modulus
      if( rem < 0 )
        Value(modulus + rem)

    def toInt = value

This seems to work correctly (e.g. modulo 7: 6 + 2 = 1).

A nice side effect of using path-dependent types is that numbers modulo different integers will throw a type error if you try to operate them together. However, I would like to insert a test that will confirm this fact.

import org.scalatest.FunSuite
import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner

class ModularIntSuite extends FunSuite {
  test("a mod n + b mod m => error") {
    val n: Int = 3
    val m: Int = 4
    val modulen = new ModularInt( n )
    val modulem = new ModularInt( m )

    val a = new modulen.Value(0)
    val b = new modulem.Value(0)

    assert(a plus b) // catch error here

However, as a plus b throws a type error at compile time, the test won't even run.

Is there anyway to insert a test for a compile error? Or does the fact that compile errors are tests of correctness themselves mean that this is a redundant check (i.e. what I'm trying to do makes no sense)?


by tensorproduct at July 22, 2014 06:49 AM


LibreSSL 2.0.3 Released

Bob Beck (beck@) has announced the release of LibreSSL 2.0.3:

We have released an update, LibreSSL 2.0.3 - which should
be arriving in the LibreSSL directory of an OpenBSD mirror near
you very soon. 
This release includes a number of portability fixes based on the
the feedback we have received from the community. It also includes
some improvements to the fork detection support. 
As noted before, we welcome feedback from the broader community.                

July 22, 2014 06:40 AM


Use of geometric mean for average return of several indices

Can anyone give any reference for using the geometric mean to average the returns from several indices? Note, this question is not about the usual use of geometric mean to obtain the average return from a single time series. It is about averaging several indices in a single time step, so for example :-

January 2014
index 1 return = 3%
index 2 return = 5%
index 3 return = -2%


returns = {3, 5, -2};
meanreturn = (GeometricMean[returns/100. + 1] - 1)*100

Answer: 1.95711 %


My current thought as to why the geometric mean might be used to average returns in a single time period is that it produces a lower result than the arithmetic mean, so for generally positive returns with a leptokurtic bias (shown red c/w blue normal dist.) the geometric mean would damp out the contribution of the higher returns. This seems a bit of a kluge though; any references welcome.

enter image description here

Averages generated from randomly generated distributed returns around a value of 1%

by Chris Degnen at July 22, 2014 06:33 AM


Custom JodaTime serializer using Play Framework's JSON library?

How do I implement a custom JodaTime's DateTime serializer/deserializer for JSON? I'm inclined to use the Play Framework's JSON library (2.1.1). There is a default DateTime serializer, but it uses dt.getMillis instead of .toString which would return an ISO compliant String.

Writing Reads[T] amd Writes[T] for case classes seems fairly straightforward, but I can't figure out how to do the same for DateTime.

by Dominykas Mostauskis at July 22, 2014 06:10 AM


Infinite autocorrelation - Unit root?

I have a time series of gold prices, on which I want to build an ARIMA model. The series is autocorrelated and if I can difference as often as I want, it always is.

First: data: d1gold Dickey-Fuller = -18.5829, Lag order = 19, p-value = 0.01 alternative hypothesis: stationary

Second: data: d2gold Dickey-Fuller = -32.6297, Lag order = 19, p-value = 0.01 alternative hypothesis: stationary .. and so on.

What can I do to fit the data in an ARIMA model? Data:

Best Regards Erik

by user9358 at July 22, 2014 06:04 AM




SVCJ (SVJJ) Duffie et. al Model implementation in Matlab

I'm attempting to implement aforementioned SVCJ model by Duffie et al in MATLAB. so far without success. It's supposed to price vanilla (european) calls . parameters provided, the expected price is: ~6.0411. The function throws the error:

Warning: Reached the limit on the maximum number of intervals in use. Approximate bound on error is 1.1e+03. The integral may not exist, or it may be difficult to approximate numerically to the requested accuracy.

Using quadv instead of integral didn't yield the expected result, but instead just returned NaNs.

    function price = SVCJDuffie2000Test(S0, K, V0, r, mus, muv, lambda, kappav, ...
        thetav, sigmav, sigmas, rho, rhoj, q, t, T)
        % this function should in theory calculate the analytical solution for
        % the SVCJ model by Duffie et al
        % T=0.25
        % K = 100;
        % S0 = 85;
        % kappav=4;
        % lambda=4;
        % rhoj=-0.5;
        % rho=-0.5;
        % thetav=0.04;
        % r=0.05;
        % sigmas=0.06;
        % mu =  0.2380
        % muv=0.02;
        % muJ = -0.04; 
        % sigmav=0.1;
        % V0 = 0.25;
        % mus = log((1+mu)*(1-rhoj*muv))-0.5*(sigmas^2);
        % SVCJDuffie2000Test( S0, K, V0, r, mus, muv, lambda, kappav,  thetav, sigmav, sigmas, rho, rhoj, 0,0,T )

         % S0, V0, r, mu, muv, lambda, kappav, 
         % thetav, sigmav, sigmas, rho, rhoj, 
         y          = log(S0);
         X0         = y;
         % nu, initial vola, nubar long run mean vola
         nu         = V0;
         c          = K;
         nubar      = thetav;
         sigmav     = sigmav;
         sigmacy    = sigmas;
         mucy       = mus;
         mucv       = muv;
         rhoj       = rhoj;
         rhobar     = rho;
         kappav     = kappav;
         zetabar    = q;
         lambdac    = lambda;
         % specific to SVCJ model
         lambdabar  = lambda;
         r          = r;

         % not needed for SVJJ, ie SVCJ
         sigmay    = 0;
         lambday   = 0;
         lambdav   = 0;
         muv        = 0;
         muy        = 0;

         mubar = thetafunc(1,0) - 1

        function retval = alphabar(tau,u)
            thetainter = lambdabar^-1*(lambdac*fc(u,tau));
            retval = alpha0(tau, u) - lambdabar*tau*(1+mubar*u)+lambdabar*thetainter;

        function retval = betabar(tau,u)
            a = u*(1-u);
            b = sigmav*rhobar*u-kappav;
            gamma = sqrt(b^2+a*sigmav^2);
            retval = a*(1-exp(-gamma*tau)) / (2*gamma-(gamma+b)*(1-exp(-gamma*tau)));

        function retval = alpha0(tau, u)
           a = u*(1-u);
           b = sigmav*rhobar*u-kappav;
           gamma = sqrt(b^2+a*sigmav^2);
           retval = -r*tau+(r-zetabar)*u*tau-kappav*nubar*...
                (((gamma+b)/sigmav^2)*tau + ...

        function retval = fy(u,tau)
            retval = tau*exp(muy*u+1/2*sigmay^2*u^2);

        function retval = fv(u,tau)
            a = u*(1-u);
            b = sigmav*rhobar*u-kappav;
            gamma = sqrt(b^2+a*sigmav^2);
            c_ = 1 - rhoj*mucv*u;
            retval =  (gamma-b)/((gamma-b)+muv*a)*tau-(2*muv*a)/((gamma*c_)^2-(b*c_-muv*a)^2)...

        function retval = fc(u,tau)
            a = u*(1-u);
            b = sigmav*rhobar*u-kappav;
            c_ = 1 - rhoj*mucv*u;
            gamma = sqrt(b^2+a*sigmav^2);
            d_ = (gamma-b)/((gamma-b)*c_+mucv*a)*tau-(2*mucv*a)/((gamma*c_)^2-(b*c_-mucv*a)^2)...
            retval = exp(mucy*u+sigmacy^2*u^2/2)*d_;

        function retval = thetafunc(c1, c2)
            retval = lambdabar^-1*(lambdac*thetacfunc(c1,c2));

        function retval = thetayfunc(c)
            retval = exp(muy*c+1/2*sigmay^2*c^2);

        function retval = thetavfunc(c)
            retval = 1/(1-muv*c);

        function retval = thetacfunc(c1,c2)
            retval = exp(mucy*c1+1/2*sigmacy^2*c1^2) / ...

        function retval = psi(y_, u,  t, T)
            retval =  exp(alphabar(T-t,u)+u*y_+ betabar(T-t,u)*nu);

        % extended transform p.13
        function retval = psichi(nu_, u, y, t, T)
            retval =  exp(alphabar(T-t,u)+u*y+ betabar(T-t,u)*nu_);

        function retval = Gab(a,b, y, X0_, T)
            integratefunc = @(nu_) imag(psi(complex(a,nu_*b),X0_, 0, T) * exp(complex(0,-nu_*y))) / nu_;
            retval = psi(a,X0_,0,T)/2  - 1/pi*integral(integratefunc, 0, Inf, 'ArrayValued',true);
            %retval = 1;

        % depends on payoff function, see p. 6, 18
        %aT = -r*T;
        %bT = 0;
        %d = 1;

        % could also be:    
        aT = 0;
        bT = 1;
        d = -r*T;

        GbTplusdminusd = Gab(bT+d,-d,-log(c),X0,T)
        GbTminusd =Gab(bT,-d,-log(c),X0,T)

        price = exp(aT)*GbTplusdminusd-c*exp(aT)*GbTminusd;

Functions in the paper:

thetainer: p. 23

alpha0: p.22

alphabar: p.22

betabar: p.22

fy,fv,fc: p. 23

thetayfunc, thetacfunc, thetavfunc: p. 22

psi: p.21

Gab: p. 13

not a function, but depending on the derivative type:

GbTplusdminusd,GbTminusd, price: p. 18

lambdabar = lambda, according to: p. 24

by jcfrei at July 22, 2014 05:38 AM


Write a function with different return types in OCaml

How can I re-write this function in OCaml so that it allows the elements of the tuple to have different types

let nth i (x,y,z) = 
match i with
|_->raise (Invalid_argument "nth")

by suguman at July 22, 2014 05:36 AM


Video: Nikola Tesla at KPCC!


Here I am playing Tesla at Sunday’s event! My entrance is at 55:00 in the video.

Over the weekend I was pleased to participate in a Nikola Tesla event at KPCC, one of our local public radio stations in Los Angeles. Science reporter Sanden Totten (at left in the photo above) hosted a retrospective of Tesla’s life and career, and brought up scientists from the California Science Center and the Griffith Observatory to talk about the principles of electricity.

Near the end of the presentation, I got to play a time-traveling Nikola Tesla to do a brief Q&A with Sanden. If you ever get invited to dress up like a famous scientist and tell jokes, I highly recommend it.

Here is video of the event on KPCC’s site! Sanden plays Mayor LaGuardia’s posthumous remembrance of Tesla at about 54 minutes in, which precedes my entrance right at 55:00.

The event was a co-production with Sanden’s radio show/podcast Brains On, which is a science and education show for kids specifically. There were a lot of kids in the audience at this event and I hope they enjoyed it!

I have to thank Sanden for inviting me to participate, and also for providing me with the opportunity to take what should probably be my author photo for time immemorial.



by David Malki at July 22, 2014 05:32 AM


Efficient Shamir secret sharing reconstruction

Shamir's secret sharing scheme is a well known way to convert a secret into a polynomial and distribute points in this polynomial. Some of these points can then be regrouped to reconstruct the original polynomial and subsequently the secret.

The regrouping process can be summarized as finding $p(0)$ in a polynomial knowing only the degree and pairs $(x_i, y_i)$ such that $p(x_i) = y_i$.

My question is: how do I do this efficiently? All implementations I've seen so far are $O(n^2)$ because they recompute the polynomial using Lagrange, with a few optimizations. Are there known faster ways?

by BoppreH at July 22, 2014 05:15 AM



What techniques are there for asking about satisfiability of a first order of a sentence?

In reading about SAT in general and applications in first order logic, I've noticed that there are algorithms for answering the question of if a particular instance of a propositional sentence is satisfiable, or if a sentence in a decidable fragment of first order logic is satisfiable (such as Presburger arithmetic) but I haven't seen any algorithms that explicitly attempt to decide satisfiability of first order sentences in general. There are also techniques for answering if a first order sentence is valid, by answering if its negation is unsatisfiable using first order resolution, or tableau methods.

It looks to me like first order resolution doesn't terminate for many satisfiable first order sentences. It can tell if a sentence is unsatisfiable but not if a sentence is satisfiable (or just undecidable.) Are there algorithms for asking if a sentence is satisfiable in first order logic?

by dezakin at July 22, 2014 05:02 AM



Invoking a Future inside a receive method and stopping the actor after that

I'm creating a bunch of actors that do some work and then they are stopped. In some of these actors I'm invoking 3rd party APIs that return a Future.

MyActor extends Actor 

def receive = {

   case MsgA => {
    //this evaluates to a Future
    val p : Future = someFutureAPICall()
    //stop the actor 
    context stop self 



Now since the Future is non-blocking the actor will be stopped right after that (?), even if the Future has not completed. What is the expected behavior in this case ?

For example, if I've an onComplete on the Future, will that ever be executed even if the actor has stopped ?

MyActor extends Actor 

def receive = {

   case MsgA => {
    //this evaluates to a Future
    val p : Future = someFutureAPICall()

    p.onComplete {
      //will these expressions ever be evaluated ? 
      case Success(x) =>"Success")
      case Failure(f) =>"Failure") 
    //stop the actor 
    context stop self 



by Soumya Simanta at July 22, 2014 04:51 AM

Planet Theory

Shifts In Algorithm Design

How to find approximate page rank fast, among other things


Jennifer Chayes is the current director of a research lab in Cambridge—that is Cambridge Massachusetts—for a company called Microsoft. She is famous for her own work in many areas of theory, including phase transitions of complex systems. She is also famous for her ability to create and manage research groups, which is a rare and wonderful skill.

Today Ken and I wish to talk about how to be “shifty” in algorithm design. There is nothing underhanded, but it’s a different playing field from what we grew up with.

Ken first noted the paradigm shift in 1984 when hearing Leslie Valiant’s inaugural talk on “Probably Approximately Correct” learning at the Royal Society in London. Last year Leslie wrote a book with this title. That’s about learning, while here we want to discuss the effect on algorithm design.

We illustrate this for a paper (ArXiv version) authored by Chayes and Christian Borgs, Michael Brautbar, and Shang-Hua Teng. It is titled, “Multi-Scale Matrix Sampling and Sublinear-Time PageRank Computation.” Since PageRank is a vital application, they don’t want their algorithms to be galactic. They trade off by not having the algorithms “solve” the problems the way we used to.

In The Good Old Days

I, Dick, recall the “good old days of theory.” When I first started working in theory—a sort of double meaning—I could only use deterministic methods. I needed to get the exact answer, no approximations. I had to solve the problem that I was given—no changing the problem. Well sometimes I did that, but mostly I had to solve the problem that was presented to me.

In the good old days of theory, we got a problem, we worked on it, and sometimes we solved it. Nothing shifty, no changing the problem or modifying the goal. I actually like today better than the “good old days,” so I do not romanticize them.

One way to explain the notion of the good old days is to quote from a Monty Python skit about four Yorkshiremen talking about the good old days. We pick it up a few lines after one of them says, “I was happier then and I had nothin’. We used to live in this tiny old house with great big holes in the roof.” …

{\mathsf{FIRST \ YORKSHIREMAN}}: You were lucky. We lived for three months in a paper bag in a septic tank. We used to have to get up at six in the morning, clean the paper bag, eat a crust of stale bread, go to work down t’ mill, fourteen hours a day, week-in week-out, for sixpence a week, and when we got home our Dad would thrash us to sleep wi’ his belt.

{\mathsf{SECOND \ YORKSHIREMAN}}: Luxury. We used to have to get out of the lake at six o’clock in the morning, clean the lake, eat a handful of ‘ot gravel, work twenty hour day at mill for tuppence a month, come home, and Dad would thrash us to sleep with a broken bottle, if we were lucky!

{\mathsf{THIRD \ YORKSHIREMAN}}: Well, of course, we had it tough. We used to ‘ave to get up out of shoebox at twelve o’clock at night and lick road clean wit’ tongue. We had two bits of cold gravel, worked twenty-four hours a day at mill for sixpence every four years, and when we got home our Dad would slice us in two wit’ bread knife.

{\mathsf{FOURTH \ YORKSHIREMAN}}: Right. I had to get up in the morning at ten o’clock at night half an hour before I went to bed, drink a cup of sulphuric acid, work twenty-nine hours a day down mill, and pay mill owner for permission to come to work, and when we got home, our Dad and our mother would kill us and dance about on our graves singing Hallelujah.

{\mathsf{FIRST \ YORKSHIREMAN}}: And you try and tell the young people of today that ….. they won’t believe you.

{\mathsf{ALL}}: They won’t!


Those were the days. I did feel like I sometimes worked twenty-nine hours a day, I was paid by Yale, but so little that perhaps it felt like I paid them. Never had to drink a cup of sulphuric acid—but the coffee—oh you get the idea.

Now today, in the 21st century, we have a better way to attack problems. We change the problem, often to one that is more tractable and useful. In many situations solving the exact problem is not really what a practitioner needs. If computing X exactly requires too much time, then it is useless to compute it. A perfect example is the weather: computing tomorrow’s weather in a week’s time is clearly not very useful.

The brilliance of the current approach is that we can change the problem. There are at least two major ways to do this:

{\bullet } Change the answer required. Allow approximation, or allow a partial answer. Do not insist on an exact answer.

{ \bullet } Change the algorithmic method. Allow algorithms that can be wrong, or allow algorithms that use randomness. Do not insist that the algorithm is a perfect deterministic one.

This is exactly what Chayes and her co-authors have done. So let’s take a look at what they do in their paper.


In their paper they study PageRank, which is the definition and algorithm made famous by Google. It gives a way to rank webpages in response to a query that supplements criteria from the query itself. An old query-specific criterion was the number of matches to a keyword in the query. Rather than rank solely by this count, PageRank emphasizes a general page score. The score is sometimes interpreted as a measure of “popularity” or “authority,” leading to the following circular-seeming definitions:

{\bullet} A webpage is popular if it has a healthy number of links from popular pages.

{\bullet} A webpage is authoritative if it is well cited, especially by other authoritative pages.

What the PageRank score actually denotes mathematically is the likelihood that a person randomly traversing links will arrive at any particular page. This includes a frequency {\alpha} with which the person will stop clicking, do something healthier like ride a bicycle, and start again on a “random” webpage.

The situation can be modeled by the classic random walk on a directed graph. We have a graph {G} on {N} nodes and an {N \times N} matrix {M} that is row-stochastic, meaning the entries in each row are non-negative and sum to {1}. Given that the web-walker is at node {i}, the entry {M[i,j]} is the probability of going next to node {j}. If node {i} has out-degree {b}, then

\displaystyle  M[i,j] = \frac{\alpha}{N} + \begin{cases} (1 - \alpha)\frac{1}{b} & \text{if~} i \text{~links to~} j\\ 0 & \text{otherwise.}\end{cases}

We can tweak this e.g. by modeling the user hitting the “Back” button on the browser, or jumping to another browser tab, or using a search engine. We could also set {\alpha} higher in case page {i} has few or no outgoing links. We still get an {M}, and since the use of {\alpha} effectively makes the graph strongly connected and averts certain pathologies, we get a beautiful conclusion from random-walk theory: There is a unique stationary distribution {P}, which is the unique left-eigenvector for the largest eigenvalue, which as normalized above is {1}:

\displaystyle  P M = P.

Then the PageRank of node {i} is {P(i)}. It is remarkable that this simple, salient idea from the good old days works so well. A further fact from the theory (and use of {\alpha}) is that if you start at any node, in the long run you will find yourself on page {i} with frequency {P(i)}. Here is Wikipedia’s graphical example:


The issue is: how to compute {P(i)}? In the good old days this was a trivial problem—just use linear algebra. But now the issue is that {N} is really big, let alone {N \times N} being unspeakably big. The {N} is too big even to get an approximation via the “further fact,” that is by simulating a random walk on the whole graph, and classical sparse-matrix methods might only help a little. This is where Chayes and company change the game: let us care about computing {P(i)} only for some {i}‘s, and even then, let us be content with fairly rough approximation.

The Shifted Problems

The approximation to PageRank is called SignificantPageRank. The paper gives a randomized algorithm that solves the following problem.

Let us be given a graph. Then, given a target threshold {\Delta} and an approximation factor {c > 1}, we are asked to output a set {S} of nodes such that with high probability, {S} contains all nodes of PageRank at least {\Delta}, and no node of PageRank smaller than {\frac{1}{c}\Delta}.

This is a perfect example of the shift. The algorithm is random, and the problem is to find not the nodes with a given PageRank, but those that are not too far away.

The nifty point is that the algorithm can tolerate fuzzing the matrix {M}, in a manner called SARA for “sparse and approximate row access”:

Given {i} and {\epsilon}, return a set {S} of {O(1/\epsilon)} columns {j} and values {p_j} such that for all {j}:

  • {j \in S \implies |p_j - M[i,j]| \leq \epsilon}, and

  • {j \notin S \implies M[i,j] \leq \epsilon}.

It is important to use this for different values of {\epsilon}. The cost of a query {(i,\epsilon) \rightarrow S} is {\Theta(|S|) = O(1/\epsilon)}.

If we picture “{N}” as “exponential” and take {\epsilon = 1/\mathsf{poly}(n)} where {n = \log N}, then this becomes an approximative version of {M} being succinct, which we just talked about. In this scaling of {\epsilon} we are effectively limiting to a {\mathsf{poly}(n)} local portion {S} of the graph around node {i}. Since we also have {\frac{\alpha}{N} \ll \epsilon}, under SARA entries outside {S} would become effectively zero, so that the chance of “teleporting” outside {S} on the whole would be regarded as negligible. In fact the paper also researches the case where each Web user always starts afresh at a “home node” {u} in that portion, making {M[i,u] = \alpha} just for that user. Then the {\alpha}-related probability is not negligible, and the resulting user-dependent estimate is called PersonalizedPageRank.

The problem they need to solve for SignificantPageRank then becomes “SignificantColumnSums”:

Given {M} and {\Delta,c} as above, find a set {S} of columns such that for all columns {j}:

  • {\sum_i M[i,j] \geq \Delta \implies j \in S};

  • {\sum_i M[i,j] \leq \frac{1}{c}\Delta \implies j \notin S}.

An even simpler problem which they use as a stepping-stone is “VectorSum”:

Given a length-{N} vector {P} with entries in {[0,1]}, and {1 \leq \Delta \leq N} and {c > 1}:

  • output yes if {{}\text{~}\sum_i P(i) \geq \Delta};

  • output no if {{}\text{~}\sum_i P(i) \leq \frac{\Delta}{c}}, don’t-care otherwise.

The goal is always to avoid looking at all {N} nodes or entries, but only an {N^{1 - \gamma}} or so portion of them, where {\gamma > 0}. Thus the problem shift is necessitated by {N} being huge. This isn’t my good-old-days idea of solving a problem, but can be called “{(\Delta,c)}-solving” it.

Multi-Level Sampling and Results

Ken and I are interested because these are similar to problems we have been encountering in our quest for more cases where quantum algorithms can be simulated in classical randomized polynomial time. Thus any new ideas are appreciated, and what catches our eye is a multi-level approximation scheme that exploits the requirement of SARA to work for different {\epsilon}. The situations are different, but we hope to adapt them.

The situation for VectorSum is that a probe {(i,\epsilon) \rightarrow P(i)} still costs order-of {1/\epsilon}, and returns {0} unless {P(i) \geq \epsilon}. A simple-minded use of a set {S} of random probes for the same {\epsilon} would yield the estimate

\displaystyle  \frac{N}{|S|} \sum_{i \in S: P(i) \geq \epsilon} P(i).

The resulting error has order {N\epsilon}, so we need {\epsilon \approx \frac{\Delta}{N}} which is rather demanding. Indeed the total cost would have order {\frac{|S|N}{\Delta}}, where {S} needs to be so large as to kill any hope of making the cost {N^{1 - \gamma}} or even {o(N)}. In the pivotal case where {\sum_i P(i) = \Delta}, we would need {|S| \approx \frac{N}{\Delta}}, incurring cost on the order of {(\frac{N}{\Delta})^2}.

However, they show that by using a different precision {\epsilon_t \approx \frac{t}{|S|}} for each random probe, they can get acceptable error with a reasonably small number of probes. The case {t = 1} where we have {\epsilon \approx \frac{1}{|S|} = \frac{\Delta}{N}} occurs only once, so its {\frac{N}{\Delta}} cost can be tolerated. Other probes have smaller cost, and while their precisions are looser, the aggregate precision on the estimate becomes good enough for the following result:

Theorem 1 Given {P,c,\Delta} as above and {\delta > 0}, VectorSum can be {(\Delta,c)}-solved with probability at least {1 - \delta} and cost

\displaystyle  O\left(\frac{N}{\Delta}\left(\frac{1}{c-1}\right)^2 \log\left(\frac{N}{\Delta(c-1)}\right)\log\left(\frac{2}{\delta}\right)\right).

Well in the good old days before LaTeX we wouldn’t even have been easily able to write such a formula with a typewriter, let alone prove it. But it is certainly better than {(\frac{N}{\Delta})^2}, and allows taking {\Delta = N^{\gamma}} to meet the goal of runtime {\tilde{O}(N^{1 - \gamma})}. As usual, for the details on SignificantColumnSums and the application problems, see the paper.

Open Problems

Do you miss the good old days? Or do you like the current approaches? What shifts, what new types of changing the goals, might we see in the future? For clearly today will one day be the “good old days.”

[changed qualifier on "certain pathologies", may as well footnote here that the "Back" button creates a Markov Chain with memory.]

by Pip at July 22, 2014 04:50 AM



Do I need to setup my ZFS (using FreeBSD 10) for foreign-language files (e.g. filenames in Japanese/Thai/etc language)?

I am in the process of setting up a ZFS file server. I am planning to store all sorts of files in it. I have several hundred music tracks that are in Japanese/Korean native character.

I want the file server to retain this native character naming. Do I need to setup my ZFS pool with utf8only and/or normalization options?


copy japanese songs into file server (via mounted shared folder in Windows). If I view the files on my file server in Windows Explorer, they must be displayed in their native characters.

Thanks a lot.

by mrjayviper at July 22, 2014 04:40 AM


How is the price of a bond actually determined?

How the price of a bond is actually determined? Is it the supply-demand that determines the price first and then the YTM is calculated on the back of this for that bond. Or is it that the changes to interest rate curve comes first and then the bond is priced using the typical discounting method and that becomes the price in stock market?

by Papal at July 22, 2014 04:39 AM


Regular expression (ab U a)* to NFA with two states (Sipser)?

In the 3rd edition of Sipser's Introduction to the Theory of Computation (example 1.56, p.68), there is a step-by-step procedure to transform (ab U a)* into a NFA. And then the text ends with: "In this example, the procedure gives an NFA with eight states, but the smallest equivalent NFA has only two states. Can you find it?" Nope. I can't. After a good deal of head scratching, I've convinced myself that it's not doable. But being a novice, I'm probably wrong. Can anyone help? Thanks!

by Garp at July 22, 2014 04:31 AM




unable to run oowriter as web user

I have a web server set up in my laptop. I was building a web application which i am accessing in the local network. I have a php file which contains following line:

$command = "/usr/bin/oowriter --invisible --convert-to pdf /var/www/hackathon/document/gaurav.doc -outdir /var/www/hackathon/pdf/" exec($command,$output,$return_var); print_r($output);

So, the problem is when i run in terminal above php runs perfect.But when accessed via web browser, output is just empty and no conver is done as intended. apache error log shows following lines:

[Java framework] Error in function createSettingsDocument (elements.cxx). javaldx failed! Warning: failed to read path from javaldx

i have tried solutions . but it didnt work.

i am having OpenJdk 7.

Anybody having any idea how to make this work??

by Gaurav Singh at July 22, 2014 03:57 AM



enter image description here

how there are two different conversion results for the same bases. Am i doing something wrong ! Please help !

thanks in advance.

by Byteology at July 22, 2014 03:40 AM






Is there a way to flatten nested monads of different types?

I am not sure how to describe this problem, so I'll just show the type signatures.

I have an instance of the following:

val x:Future[F[Future[F[B]]]] = ???

And I want an instance of:

val y:Future[F[B]] = ???

F is a Monad, so I have the following methods:

def pure[A](a:A):F[A] = ???
def flatMap[A, B](fa:F[A], f:A => F[B]):F[B] = ???
def map[A, B](fa:F[A], f:A => B):F[B] = flatMap(fa, (a:A) => pure(f(a)))

I think the following should work, but it does not feel right:

x.flatMap { fWithFuture =>
  val p = Promise[F[B]]
  flatMap(fWithFuture, (futureF: Future[F[B]]) => {

Is there a concept I'm missing?

A bit of background information. I am trying to define a function like this:

def flatMap[A, B](fa:Future[F[A]], f: A => Future[F[B]]):Future[F[B]] = ???

Maybe this is conceptually a weird thing. Any tips on useful abstractions are welcome.

by EECOLOR at July 22, 2014 03:05 AM



Why's Clojure failing from JNLP now?

I've got a Java app that's been in use for awhile and uses embedded Clojure scripts for some of the calculations. Old versions of it still run fine with the current JRE, but if I go to build a new version (only changes to the code are in unrelated Java code) it fails to launch properly. It also runs perfectly fine locally. Only JNLP is affected. access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
    at Source)
    at Source)
    at java.lang.SecurityManager.checkPermission(Unknown Source)
    at java.lang.Class.checkMemberAccess(Unknown Source)
    at java.lang.Class.getDeclaredConstructors(Unknown Source)
    at clojure.core$generate_proxy.invoke(core_proxy.clj:137)
    at clojure.core$get_proxy_class.doInvoke(core_proxy.clj:261)
    at clojure.lang.RestFn.applyTo(
    at clojure.core$apply.invoke(core.clj:617)
    at clojure.core$proxy.doInvoke(core_proxy.clj:335)
    at clojure.lang.RestFn.invoke(
    at clojure.lang.Var.invoke(
    at clojure.lang.AFn.applyToHelper(
    at clojure.lang.Var.applyTo(
    at clojure.lang.Compiler.macroexpand1(
    at clojure.lang.Compiler.analyzeSeq(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler$BodyExpr$Parser.parse(
    at clojure.lang.Compiler$LetExpr$Parser.parse(
    at clojure.lang.Compiler.analyzeSeq(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.analyzeSeq(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler$BodyExpr$Parser.parse(
    at clojure.lang.Compiler$FnMethod.parse(
    at clojure.lang.Compiler$FnExpr.parse(
    at clojure.lang.Compiler.analyzeSeq(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.analyzeSeq(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.access$100(
    at clojure.lang.Compiler$DefExpr$Parser.parse(
    at clojure.lang.Compiler.analyzeSeq(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.analyze(
    at clojure.lang.Compiler.eval(
    at clojure.lang.Compiler.load(
    at clojure.lang.RT.loadResourceScript(
    at clojure.lang.RT.loadResourceScript(
    at clojure.lang.RT.load(
    at clojure.lang.RT.load(
    at clojure.core$load$fn__1451.invoke(core.clj:5530)
    at clojure.core$load.doInvoke(core.clj:5529)
    at clojure.lang.RestFn.invoke(
    at clojure.core$eval1658.invoke(core.clj:6154)
    at clojure.lang.Compiler.eval(
    at clojure.lang.Compiler.load(
    at clojure.lang.RT.loadResourceScript(
    at clojure.lang.RT.loadResourceScript(
    at clojure.lang.RT.load(
    at clojure.lang.RT.load(
    at clojure.lang.RT.doInit(
    at clojure.lang.RT.<clinit>(
    at myapp(
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.sun.javaws.Launcher.executeApplication(Unknown Source)
    at com.sun.javaws.Launcher.executeMainClass(Unknown Source)
    at com.sun.javaws.Launcher.doLaunchApp(Unknown Source)
    at Source)
    at Source)

The line that it's failing on is this:

RT.loadResourceScript("myapp/clojure_scripts.clj"); // Initialize Clojure script processor with our script

We're currently using a self-signed cert for this app (I've been working with Comodo for the last week trying to get a trusted cert to see if that helps, but I've still not received it and I need this app working ASAP). What's odd to me is that old versions (also self-signed the exact same way) work. I just can't build a new version. I've rolled back my configs so that only actual java code changes should be different, but still no luck. I'm missing something somewhere...

by Brian Knoblauch at July 22, 2014 02:48 AM

ccw clojure set up troubles

I have been working with eclipse + counterclockwise for clojure development, using only the default template for non-web based work. However, I would now like to connect my environment to my localhost apache server. I began a new project, and changed the template from default to clojure-app and it did give me a web based framework in the project.clj file, as follows:

(defproject guestbook "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url ""
:dependencies [[org.clojure/clojure "1.6.0"]
             [compojure "1.1.6"]
             [hiccup "1.0.5"]
             [ring-server "0.3.1"]]
:plugins [[lein-ring "0.8.10"]]
:ring {:handler guestbook.handler/app
     :init guestbook.handler/init
     :destroy guestbook.handler/destroy}
:aot :all
{:open-browser? true, :stacktraces? false, :auto-reload? false}}
{:dependencies [[ring-mock "0.1.5"] [ring/ring-devel "1.2.1"]]}})

Next, I try to set up ring through lein ring server in the terminal. But I get this error: "'ring' is not a task. See 'lein help'." I am not sure how to proceed, or exactly what I can do to get the localhost server up.

by nmac at July 22, 2014 02:15 AM


Call by need compared to pass by function

SML uses pass‑by‑value, Haskell uses call‑by‑need. Unless I'm wrong (the purpose of this question), one can do call‑by‑need with SML, passing a function instead of a value; a function to be later evaluated when needed.

Adapting a classical example:

(* This will print `0` *)

fun pass_by_function (cond, a1, a2) =
   if cond then a1
   else a2

val e1 = fn () => 1 * 0
val e2 = fn () => 1 div 0

val _ = print (Int.toString (pass_by_function (true, e1, e2) ()))

(* This will fail with a divide error *)

fun pass_by_value (cond, a1, a2) =
   if cond then a1
   else a2

val _ = print (Int.toString (pass_by_value (true, 1 * 0, 1 div 0)))

It's a big matter for Haskell to be call‑by‑need. I'm just wondering if there is something else or if it's just, say “syntactic sugar”, compared to the way it needs to be done with SML.

Is this just a different way to write the same or is there a semantic difference I'm not aware of? Passing a function, is it indeed always the same as call‑by‑need? If not, when and how does it differs?

by Hibou57 at July 22, 2014 02:11 AM


How to log all requests with unfiltered

I'm using unfiltered to provide restful API, and have defined several intents. Now I have a new requirement, I have to log all the request url into a file, but I can't find a good solution for it.

I've read the document of unfiltered, not found something like "filter/interceptor" in SpringMVC. Is there any way to do it?

by Freewind at July 22, 2014 01:53 AM

arXiv Logic in Computer Science

Differentially Private MIMO Filtering for Event Streams and Spatio-Temporal Monitoring. (arXiv:1407.5553v1 [cs.SY])

Many large-scale systems such as intelligent transportation systems, smart grids or smart buildings collect data about the activities of their users to optimize their operations. In a typical scenario, signals originate from many sensors capturing events involving these users, and several statistics of interest need to be continuously published in real-time. In addition, in order to encourage user participation, privacy issues need to be taken into consideration. This paper considers the problem of providing differential privacy guarantees for such multi-input multi-output systems operating continuously. We show in particular how to construct various extensions of the zero-forcing equalization mechanism, which we previously proposed for single-input single-output systems. We also describe an application to privately monitoring and forecasting occupancy in a building equipped with a dense network of motion detection sensors, which is useful for example to control its HVAC system.

by <a href="">Jerome Le Ny</a>, <a href="">Meisam Mohammady</a> at July 22, 2014 01:30 AM

Tractable Model for Rate in Self-Backhauled Millimeter Wave Cellular Networks. (arXiv:1407.5537v1 [cs.IT])

Millimeter wave (mmW) cellular systems will require high gain directional antennas and dense base station (BS) deployments to overcome high near field path loss and poor diffraction. As a desirable side effect, high gain antennas provide interference isolation, providing an opportunity to incorporate self-backhauling--BSs backhauling among themselves in a mesh architecture without significant loss in throughput--to enable the requisite large BS densities. The use of directional antennas and resource sharing between access and backhaul links leads to coverage and rate trends that differ significantly from conventional microwave ($\mu$W) cellular systems. In this paper, we propose a general and tractable mmW cellular model capturing these key trends and characterize the associated rate distribution. The developed model and analysis is validated using actual building locations from dense urban settings and empirically-derived path loss models. The analysis shows that in sharp contrast to the interference limited nature of $\mu$W cellular networks, the spectral efficiency of mmW networks (besides total rate) also increases with BS density particularly at the cell edge. Increasing the system bandwidth, although boosting median and peak rates, does not significantly influence the cell edge rate. With self-backhauling, different combinations of the wired backhaul fraction (i.e. the faction of BSs with a wired connection) and BS density are shown to guarantee the same median rate (QoS).

by <a href="">Sarabjot Singh</a>, <a href="">Mandar N. Kulkarni</a>, <a href="">Amitava Ghosh</a>, <a href="">Jeffrey G. Andrews</a> at July 22, 2014 01:30 AM

Process-Oriented Parallel Programming with an Application to Data-Intensive Computing. (arXiv:1407.5524v1 [cs.PL])

We introduce process-oriented programming as a natural extension of object-oriented programming for parallel computing. It is based on the observation that every class of an object-oriented language can be instantiated as a process, accessible via a remote pointer. The introduction of process pointers requires no syntax extension, identifies processes with programming objects, and enables processes to exchange information simply by executing remote methods. Process-oriented programming is a high-level language alternative to multithreading, MPI and many other languages, environments and tools currently used for parallel computations. It implements natural object-based parallelism using only minimal syntax extension of existing languages, such as C++ and Python, and has therefore the potential to lead to widespread adoption of parallel programming. We implemented a prototype system for running processes using C++ with MPI and used it to compute a large three-dimensional Fourier transform on a computer cluster built of commodity hardware components. Three-dimensional Fourier transform is a prototype of a data-intensive application with a complex data-access pattern. The process-oriented code is only a few hundred lines long, and attains very high data throughput by achieving massive parallelism and maximizing hardware utilization.

by <a href="">Edward Givelberg</a> at July 22, 2014 01:30 AM

L(2,1)-labelling of Circular-arc Graph. (arXiv:1407.5488v1 [cs.DM])

An L(2,1)-labelling of a graph $G=(V, E)$ is $\lambda_{2,1}(G)$ a function $f$ from the vertex set V (G) to the set of non-negative integers such that adjacent vertices get numbers at least two apart, and vertices at distance two get distinct numbers. The L(2,1)-labelling number denoted by $\lambda_{2,1}(G)$ of $G$ is the minimum range of labels over all such labelling. In this article, it is shown that, for a circular-arc graph $G$, the upper bound of $\lambda_{2,1}(G)$ is $\Delta+3\omega$, where $\Delta$ and $\omega$ represents the maximum degree of the vertices and size of maximum clique respectively.

by <a href="">Satyabrata Paul</a>, <a href="">Madhumangal Pal</a>, <a href="">Anita Pal</a> at July 22, 2014 01:30 AM

Joint Channel Selection and Power Control in Infrastructureless Wireless Networks: A Multi-Player Multi-Armed Bandit Framework. (arXiv:1407.5447v1 [cs.GT])

This paper deals with the problem of efficient resource allocation in dynamic infrastructureless wireless networks. Assuming a reactive interference-limited scenario, each transmitter is allowed to select one frequency channel (from a common pool) together with a power level at each transmission trial; hence, for all transmitters, not only the fading gain, but also the number of interfering transmissions and their transmit powers are varying over time. Due to the absence of a central controller and time-varying network characteristics, it is highly inefficient for transmitters to acquire global channel and network knowledge. Therefore a reasonable assumption is that transmitters have no knowledge of fading gains, interference, and network topology. Each transmitting node selfishly aims at maximizing its average reward (or minimizing its average cost), which is a function of the action of that specific transmitter as well as those of all other transmitters. This scenario is modeled as a multi-player multi-armed adversarial bandit game, in which multiple players receive an a priori unknown reward with an arbitrarily time-varying distribution by sequentially pulling an arm, selected from a known and finite set of arms. Since players do not know the arm with the highest average reward in advance, they attempt to minimize their so-called regret, determined by the set of players' actions, while attempting to achieve equilibrium in some sense. To this end, we design in this paper two joint power level and channel selection strategies. We prove that the gap between the average reward achieved by our approaches and that based on the best fixed strategy converges to zero asymptotically. Moreover, the empirical joint frequencies of the game converge to the set of correlated equilibria. We further characterize this set for two special cases of our designed game.

by <a href="">Setareh Maghsudi</a>, <a href="">Slawomir Stanczak</a> at July 22, 2014 01:30 AM

Composing security protocols: from confidentiality to privacy. (arXiv:1407.5444v1 [cs.CR])

Security protocols are used in many of our daily-life applications, and our privacy largely depends on their design. Formal verification techniques have proved their usefulness to analyse these protocols, but they become so complex that modular techniques have to be developed. We propose several results to safely compose security protocols. We consider arbitrary primitives modeled using an equational theory, and a rich process algebra close to the applied pi calculus.

Relying on these composition results, we are able to derive some security properties on a protocol from the security analysis performed on each sub-protocol individually. We consider parallel composition and the case of key-exchange protocols. Our results apply to deal with confidentiality but also privacy-type properties (e.g. anonymity, unlinkability) expressed using a notion of equivalence. We illustrate the usefulness of our composition results on protocols from the 3G phone application.

by <a href="">Myrto Arapinis</a>, <a href="">Vincent Cheval</a>, <a href="">St&#xe9;phanie Delaune</a> at July 22, 2014 01:30 AM

Cooperative Game Theoretic Solution Concepts for top-$k$ Problems. (arXiv:1407.5442v1 [cs.SI])

The problem of finding the $k$ most critical nodes, referred to as the $top\text{-}k$ problem, is a very important one in several contexts such as information diffusion and preference aggregation in social networks, clustering of data points, etc. It has been observed in the literature that the value allotted to a node by most of the popular cooperative game theoretic solution concepts, acts as a good measure of appropriateness of that node (or a data point) to be included in the $top\text{-}k$ set, by itself. However, in general, nodes having the highest $k$ values are not the desirable $top\text{-}k$ nodes, because the appropriateness of a node to be a part of the $top\text{-}k$ set depends on other nodes in the set. As this is not explicitly captured by cooperative game theoretic solution concepts, it is necessary to post-process the obtained values in order to output the suitable $top\text{-}k$ nodes. In this paper, we propose several such post-processing methods and give reasoning behind each of them, and also propose a standalone algorithm that combines cooperative game theoretic solution concepts with the popular greedy hill-climbing algorithm.

by <a href="">Swapnil Dhamal</a>, <a href="">Akanksha Meghlan</a>, <a href="">Y. Narahari</a> at July 22, 2014 01:30 AM

Privacy-Preserving Important Passage Retrieval. (arXiv:1407.5416v1 [cs.IR])

State-of-the-art important passage retrieval methods obtain very good results, but do not take into account privacy issues. In this paper, we present a privacy preserving method that relies on creating secure representations of documents. Our approach allows for third parties to retrieve important passages from documents without learning anything regarding their content. We use a hashing scheme known as Secure Binary Embeddings to convert a key phrase and bag-of-words representation to bit strings in a way that allows the computation of approximate distances, instead of exact ones. Experiments show that our secure system yield similar results to its non-private counterpart on both clean text and noisy speech recognized text.

by <a href="">Luis Marujo</a>, <a href="">Jos&#xe9; Port&#xea;lo</a>, <a href="">David Martins de Matos</a>, <a href="">Jo&#xe3;o P. Neto</a>, <a href="">Anatole Gershman</a>, <a href="">Jaime Carbonell</a>, <a href="">Isabel Trancoso</a>, <a href="">Bhiksha Raj</a> at July 22, 2014 01:30 AM

Gateless Treasure: How to Get Sensitive Information from Unprotected External Storage on Android Phones. (arXiv:1407.5410v1 [cs.CR])

The flexibility of Android is mainly based on the cross application (app for short) access mechanism. Aside from providing convenience for both app developers and users, such a mechanism also brings the data in external storage which originally regarded harmless into serious privacy breaches. In this paper, we studied various popular apps and developed three different attacks leveraging the public information freely available to zero-permission apps. The first attack can smuggle out the sensitive data which reveals the victim's profile information directly, such as real name, phone number, email address, social networking accounts, etc. With the help of image comparing techniques and current social networking services, we can find out the owners of the phone numbers acquired from a victim's WhatsApp with a high probability. Moreover, the location of the victim is also not spared in the disaster. Comparing to the previous location inferring attacks, our approach is more general since the victim's identity is also acquired by our attack. These three attacks reveal the privacy risks of external storage are much more serious than people previously thought and need to be addressed urgently. In fact, all these threats are caused by the file system used by external storage which calls into question the reasonability of the assumptions on the cross app access mechanism. To the end, we propose a preliminary mitigation approach to achieve a delicate balance between utility and privacy of the data stored in external storage.

by <a href="">Xiangyu Liu</a>, <a href="">Wenrui Diao</a>, <a href="">Zhe Zhou</a>, <a href="">Zhou Li</a>, <a href="">Kehuan Zhang</a> at July 22, 2014 01:30 AM

Model based design of super schedulers managing catastrophic scenario in hard real time systems. (arXiv:1407.5404v1 [cs.DC])

The conventional design of real-time approaches depends heavily on the normal performance of systems and it often becomes incapacitated in dealing with catastrophic scenarios effectively. There are several investigations carried out to effectively tackle large scale catastrophe of a plant and how real-time systems must reorganize itself to respond optimally to changing scenarios to reduce catastrophe and aid human intervention. The study presented here is in this direction and the model accommodates catastrophe generated tasks while it tries to minimize the total number of deadline miss, by dynamically scheduling the unusual pattern of tasks. The problem is NP hard. We prove the methods for an optimal scheduling algorithm. We also derive a model to maintain the stability of the processes. Moreover, we study the problem of minimizing the number of processors required for scheduling with a set of periodic and sporadic hard real time tasks with primary/backup mechanism to achieve fault tolerance. EDF scheduling algorithms are used on each processor to manage scenario changes. Finally we present a simulation of super scheduler with small, medium and large real time tasks pattern for catastrophe management.

by <a href="">A. Christy Persya</a>, <a href="">T.R. Gopalakrishnan Nair</a> at July 22, 2014 01:30 AM

Low-Effort Specification Debugging and Analysis. (arXiv:1407.5399v1 [cs.SE])

Reactive synthesis deals with the automated construction of implementations of reactive systems from their specifications. To make the approach feasible in practice, systems engineers need effective and efficient means of debugging these specifications.

In this paper, we provide techniques for report-based specification debugging, wherein salient properties of a specification are analyzed, and the result presented to the user in the form of a report. This provides a low-effort way to debug specifications, complementing high-effort techniques including the simulation of synthesized implementations.

We demonstrate the usefulness of our report-based specification debugging toolkit by providing examples in the context of generalized reactivity(1) synthesis.

by <a href="">R&#xfc;diger Ehlers</a> (University of Bremen and DFKI), <a href="">Vasumathi Raman</a> (California Institute of Technology) at July 22, 2014 01:30 AM

Are There Good Mistakes? A Theoretical Analysis of CEGIS. (arXiv:1407.5397v1 [cs.LO])

Counterexample-guided inductive synthesis CEGIS is used to synthesize programs from a candidate space of programs. The technique is guaranteed to terminate and synthesize the correct program if the space of candidate programs is finite. But the technique may or may not terminate with the correct program if the candidate space of programs is infinite. In this paper, we perform a theoretical analysis of counterexample-guided inductive synthesis technique. We investigate whether the set of candidate spaces for which the correct program can be synthesized using CEGIS depends on the counterexamples used in inductive synthesis, that is, whether there are good mistakes which would increase the synthesis power. We investigate whether the use of minimal counterexamples instead of arbitrary counterexamples expands the set of candidate spaces of programs for which inductive synthesis can successfully synthesize a correct program. We consider two kinds of counterexamples: minimal counterexamples and history bounded counterexamples. The history bounded counterexample used in any iteration of CEGIS is bounded by the examples used in previous iterations of inductive synthesis. We examine the relative change in power of inductive synthesis in both cases. We show that the synthesis technique using minimal counterexamples MinCEGIS has the same synthesis power as CEGIS but the synthesis technique using history bounded counterexamples HCEGIS has different power than that of CEGIS, but none dominates the other.

by <a href="">Susmit Jha</a> (Strategic CAD Labs, Intel), <a href="">Sanjit A. Seshia</a> (EECS, UC Berkeley) at July 22, 2014 01:30 AM

How to Handle Assumptions in Synthesis. (arXiv:1407.5395v1 [cs.LO])

The increased interest in reactive synthesis over the last decade has led to many improved solutions but also to many new questions. In this paper, we discuss the question of how to deal with assumptions on environment behavior. We present four goals that we think should be met and review several different possibilities that have been proposed. We argue that each of them falls short in at least one aspect.

by <a href="">Roderick Bloem</a> (Graz University of Technology), <a href="">R&#xfc;diger Ehlers</a> (University of Bremen / DFKI), <a href="">Swen Jacobs</a> (Graz University of Technology), <a href="">Robert K&#xf6;nighofer</a> (Graz University of Technology) at July 22, 2014 01:30 AM

Program Synthesis and Linear Operator Semantics. (arXiv:1407.5393v1 [cs.PL])

For deterministic and probabilistic programs we investigate the problem of program synthesis and program optimisation (with respect to non-functional properties) in the general setting of global optimisation. This approach is based on the representation of the semantics of programs and program fragments in terms of linear operators, i.e. as matrices. We exploit in particular the fact that we can automatically generate the representation of the semantics of elementary blocks. These can then can be used in order to compositionally assemble the semantics of a whole program, i.e. the generator of the corresponding Discrete Time Markov Chain (DTMC). We also utilise a generalised version of Abstract Interpretation suitable for this linear algebraic or functional analytical framework in order to formulate semantical constraints (invariants) and optimisation objectives (for example performance requirements).

by <a href="">Herbert Wiklicky</a> (Imperial College London) at July 22, 2014 01:30 AM

Synthesis of a simple self-stabilizing system. (arXiv:1407.5392v1 [cs.LO])

With the increasing importance of distributed systems as a computing paradigm, a systematic approach to their design is needed. Although the area of formal verification has made enormous advances towards this goal, the resulting functionalities are limited to detecting problems in a particular design. By means of a classical example, we illustrate a simple template-based approach to computer-aided design of distributed systems based on leveraging the well-known technique of bounded model checking to the synthesis setting.

by <a href="">Adri&#xe0; Gasc&#xf3;n</a> (SRI International), <a href="">Ashish Tiwari</a> (SRI International) at July 22, 2014 01:30 AM

Nontransitive Ranking to Enhance Routing Decision in MANETS. (arXiv:1407.5385v1 [cs.NI])

An ad hoc network is an infrastructureless network in which nodes perform terminal as well as routing functions. A routing protocol is the only substitute to complete the communications in the absence of an access point. In spite of that mobile nodes or so called routers uses some mechanism for calculating the best route when it has multiple routes for the same destination. On the basis of one or more metrics routes are ranked from best to worst. But, in an ad hoc network many factors can affect this decision, such as the delay, load, route lifetime etc. Thus measuring and finding routes on the basis of crisp mathematical model for all these attributes is complicated. That why, the fuzzy approach for best route determination is required for MANET because some of the metrics are fuzzy or vague and the classical ranking of routes and transitivity in the ranking does not hold. The proposed Nontransitive Route Ranking subjective comparison of one route with others and performs nontransitive ranking to rank routes from best to worst. The pairwise comparisons of each route with others give more accurate and fair comparison. The proposed ranking is easier than classical ranking in which metrics have assigned some value and these values are combined to obtain the ranking. Experimental result shows the efficiency of the proposed model. Keywords: Fuzzy, Rank, Nontransitive, Route, Ranking, Relativity

by <a href="">Md. Amir Khusru Akhtar</a>, <a href="">Arshad Usmani</a>, <a href="">G. Sahoo</a> at July 22, 2014 01:30 AM

The Intractability of Dynamic Mechanism Design. (arXiv:1407.5373v1 [cs.GT])

We introduce a dynamic mechanism design problem in which the designer wants to offer for sale an item to an agent, and another item to the same agent at some point in the future. The agent's joint distribution of valuations for the two items is known, and the agent knows the valuation for the current item (but not for the one in the future). The designer seeks to maximize expected revenue, and the auction must be deterministic, truthful, and ex post individually rational. The optimum mechanism involves a protocol whereby the seller elicits the buyer's current valuation, and based on the bid makes two take-it-or-leave-it offers, one for now and one for the future. We show that finding the optimum mechanism --- arguably the simplest meaningful dynamic mechanism design problem imaginable --- is NP-hard. We also prove several positive results, among them a polynomial linear programming-based algorithm for the optimum randomized auction, and we show strong separations in revenue between non-adaptive, adaptive, and randomized auctions.

by <a href="">Christos Papadimitriou</a>, <a href="">George Pierrakos</a>, <a href="">Christos-Alexandros Psomas</a>, <a href="">Aviad Rubinstein</a> at July 22, 2014 01:30 AM

A Comparative Analysis for Determining the Optimal Path using PSO and GA. (arXiv:1407.5327v1 [cs.NI])

Significant research has been carried out recently to find the optimal path in network routing. Among them, the evolutionary algorithm approach is an area where work is carried out extensively. We in this paper have used particle swarm optimization (PSO) and genetic algorithm (GA) for finding the optimal path and the concept of region based network is introduced along with the use of indirect encoding. We demonstrate the advantage of fitness value and hop count in both PSO and GA. A comparative study of PSO and genetic algorithm (GA) is carried out, and it was found that PSO converged to arrive at the optimal path much faster than GA.

by <a href="">Kavitha Sooda</a>, <a href="">T. R. Gopalakrishnan Nair</a> at July 22, 2014 01:30 AM

An Optimum Scheduling Approach for Creating Optimal Priority of Jobs with Business Values in Cloud Computing. (arXiv:1407.5320v1 [cs.DC])

Realizing an optimal task scheduling by taking into account the business importance of jobs has become a matter of interest in pay and use model of Cloud computing. Introduction of an appropriate model for an efficient task scheduling technique could derive benefit to the service providers as well as clients. In this paper, we have addressed two major challenges which has implications on the performance of the Cloud system. One of the major issues is handling technical aspects of distributing the tasks for targeted gains and the second issue is related to the handling of the business priority for concurrently resolving business complexity related to cloud consumers. A coordinated scheduling can be achieved by considering the weightage of both aspects viz. technical requirements and business requirements appropriately. It can be done in such a way that it meets the QoS requirements of technical domain as well as business domain. Along with the technical priority a business Bp is required in creating a resultant priority which could be given to stages of further processing, like task allocation and arbitration schemes. Here we consider a technical priority Tp that is governed by a semi-adaptive scheduling algorithm whereas the resultant priority is derived in which a Business Priority Bp layer encapsulates the Technical Priority Tp to achieve the overall priority of the incoming tasks. It results in a Hybrid priority creation, which is a combination of both technical priority Tp and business priority Bp. By taking into account the business priority of the jobs it is possible to achieve a higher service level satisfaction for the tasks which are submitted with their native technical priority. With this approach the waiting time of the tasks tends to get reduced and it gives a better service level satisfaction.

by <a href="">Vivek Sharma</a>, <a href="">T. R. Gopalakrishnan Nair</a> at July 22, 2014 01:30 AM

The Expressive Power of Epistemic $\mu$-Calculus. (arXiv:1407.5166v1 [cs.LO])

While the $\mu$-calculus notoriously subsumes Alternating-time Temporal Logic (ATL), we show that the epistemic $\mu$-calculus does not subsume ATL with imperfect information (ATL$_i$) for the synchronous perfect-recall semantics. To prove this we first establish that jumping parity tree automata (JTA), a recently introduced extension of alternating parity tree automata, are expressively equivalent to the epistemic $\mu$-calculus, and this for any knowledge semantics. Using this result we also show that, for bounded-memory semantics, the epistemic $\mu$-calculus is not more expressive than the standard $\mu$-calculus, and that its satisfiability problem is EXPTIME-complete.

by <a href="">C&#x103;t&#x103;lin Dima</a>, <a href="">Bastien Maubert</a>, <a href="">Sophie Pinchinat</a> at July 22, 2014 01:30 AM

Cooperative oligopoly games with boundedly rational firms. (arXiv:1202.3885v2 [cs.GT] UPDATED)

We analyze cooperative Cournot games with boundedly rational firms. Due to cogni- tive constraints, the members of a coalition cannot accurately predict the coalitional structure of the non-members. Thus, they compute their value using simple heuris- tics. In particular, they assign various non-equilibrium probability distributions over the outsiders' set of partitions. We construct the characteristic function of a coalition in such an environment and we analyze the core of the corresponding games. We show that the core is non-empty provided the number of firms in the market is sufficiently large. Moreover, we show that if two distributions over the set of partitions are related via first-order dominance, then the core of the game under the dominated distribution is a subset of the core under the dominant distribution.

by <a href="">Paraskevas V. Lekeas</a>, <a href="">Giorgos Stamatopoulos</a> at July 22, 2014 01:30 AM


Why is cmis returning an error on datalist item creation in alfresco?

I have the following piece of code

    try {
          Map("cmis:objectTypeId" -> "D:dl:contact",
          "cmis:name" -> (data.emailAddress + "_Newsletter_sub"),
          "dl:contactEmail" -> data.emailAddress)
          case e: CmisRuntimeException => Logger.error("\n\n ====> silently ignoring this exception because the item is created\n"+e.getMessage+"\n\n");

which works fine and creates the data list item. but it throws an exception with the message
Execution exception[[CmisRuntimeException: Newly created object is not an item! New id: Object Id:
I'd like to know if this a normal/known issue with this api method and why it's happening. Thanks

by Dark Star1 at July 22, 2014 01:20 AM

DragonFly BSD Digest


Scala -- transform and map data

The raw data file I will be reading is a tab-delimited and one of the fields is a timestamp:

timestamp  userId  keyword
1405377264  A      google
1405378945  B      yahoo
1405377264  C      facebook

I got a case class defined as:

case class Event(date: String, userId: Int, keyword: String)

How do I go about transforming the timestamp to Date format and then map to the Event case class? I have the logic to convert the timestamp to Date:

import java.text.SimpleDateFormat
import java.util.Date

val df = new SimpleDateFormat("yyyy-MM-dd")
val dt = new Date(timestamp*1000L)
val date = df.format(dt) 

What is the right way to convert the raw data and map it to the case class?


by user2727704 at July 22, 2014 01:06 AM


Planet Clojure

July 2014 London Clojure Dojo at ThoughtWorks

Tuesday, July 29, 2014 from 7:00 PM to 9:30 PM (BST)

ThoughtWorks London Office
173 High Holborn
WC1V London
United Kingdom

Hosted By:
London Clojurians

Register for this event now at :

Event Details:

London Clojure Dojo at ThoughtWorks

The goal of the session is to help people learn to start working with Clojure through practical exercises, but we hope that more experienced developers will also come along to help form a bit of a London Clojure community. The dojo is a great place for new and experienced clojure coders to learn more. If you want to know how to run your own dojo or get an idea of what dojos are like you can read more here.


We hope to break up into groups for the dojo. So if you have a laptop with a working clojure environment please bring it along.


We’ll be discussing the meetup on the london-clojurians mailing list


Clojure is a JVM language that has syntactically similarities to Lisp, full integration with Java and its libraries and focuses on providing a solution to the issue of single machine concurrency.


Its small core makes it surprisingly easy for Java developers to pick up and it provides a powerful set of concurrency strategies and data structures designed to make immutable data easy to work with. If you went to Rich Hickey’s LJC talk about creating Clojure you’ll already know this, if not it’s well worth watching the Rich Hickey “Clojure for Java Programmers” video or Stuart Halloway “Radical Simplicity” video .

by London Clojurian Events at July 22, 2014 01:00 AM


Can some body help me to understand this Recommendations Engine

Came across this recommendation engine document. Can't understand it.

K-means clustering , Apriori enter image description here

by Sam Agosti at July 22, 2014 12:54 AM


What features does Pedestal have compared to other Clojure web frameworks?

Recently Relevance announced Pedestal, a new web framework for Clojure.

How does this framework compare to other Clojure web frameworks such as Compojure, Webjure, Conjure, Moustache and so on, in terms of features, ease of application development, developer tools, etc.

See Comparison of Clojure web frameworks

by noahlz at July 22, 2014 12:50 AM

Planet Theory

Classification of Passes in Football Matches using Spatiotemporal Data

Authors: Michael Horton, Joachim Gudmundsson, Sanjay Chawla, Joël Estephan
Download: PDF
Abstract: A knowledgeable observer of a game of football (soccer) can make a subjective evaluation of the quality of passes made between players during the game. We investigate the problem of producing an automated system to make the same evaluation of passes. We present a model that constructs numerical predictor variables from spatiotemporal match data using feature functions based on methods from computational geometry, and then learns a classification function from labelled examples of the predictor variables. Furthermore, the learned classifiers are analysed to determine if there is a relationship between the complexity of the algorithm that computed the predictor variable and the importance of the variable to the classifier. Experimental results show that we are able to produce a classifier with 85.8% accuracy on classifying passes as Good, OK or Bad, and that the predictor variables computed using complex methods from computational geometry are of moderate importance to the learned classifiers. Finally, we show that the inter-rater agreement on pass classification between the machine classifier and a human observer is of similar magnitude to the agreement between two observers.

July 22, 2014 12:41 AM

Symblicit algorithms for optimal strategy synthesis in monotonic Markov decision processes

Authors: Aaron Bohy, Véronique Bruyère, Jean-François Raskin
Download: PDF
Abstract: When treating Markov decision processes (MDPs) with large state spaces, using explicit representations quickly becomes unfeasible. Lately, Wimmer et al. have proposed a so-called symblicit algorithm for the synthesis of optimal strategies in MDPs, in the quantitative setting of expected mean-payoff. This algorithm, based on the strategy iteration algorithm of Howard and Veinott, efficiently combines symbolic and explicit data structures, and uses binary decision diagrams as symbolic representation. The aim of this paper is to show that the new data structure of pseudo-antichains (an extension of antichains) provides another interesting alternative, especially for the class of monotonic MDPs. We design efficient pseudo-antichain based symblicit algorithms (with open source implementations) for two quantitative settings: the expected mean-payoff and the stochastic shortest path. For two practical applications coming from automated planning and LTL synthesis, we report promising experimental results w.r.t. both the run time and the memory consumption.

July 22, 2014 12:41 AM

Complexity of Grundy coloring and its variants

Authors: Edouard Bonnet, Florent Foucaud, Eun Jung Kim, Florian Sikora
Download: PDF
Abstract: The Grundy number of a graph is the maximum number of colors used by the greedy coloring algorithm over all vertex orderings. In this paper, we study the computational complexity of Grundy Coloring, the problem of determining whether a given graph has Grundy number at least $k$. We show that Grundy Coloring can be solved in time $O^*(2^n)$ on graphs of order $n$. While the problem is known to be solvable in time $f(k,w)n$ for graphs of treewidth $w$, we prove that under the Exponential Time Hypothesis, it cannot be computed in time $O^*(c^w)$, for every constant $c$. We also consider two previously studied variants of Grundy Coloring, namely Weak Grundy Coloring and Connected Grundy Coloring. We show that Weak Grundy Coloring is fixed-parameter tractable with respect to the weak Grundy number. In stark contrast, it turns out that checking whether a given graph has connected Grundy number at least $k$ is NP-complete already for $k=7$.

July 22, 2014 12:41 AM

On the Algorithmic Lov\'{a}sz Local Lemma

Authors: Ioannis Giotis, Lefteris Kirousis, Kostas I. Psaromiligkos, Dimitrios M. Thilikos
Download: PDF
Abstract: The algorithm for Lov\'{a}sz Local Lemma by Moser and Tardos gives a constructive way to prove the existence of combinatorial objects that satisfy a system of constraints. We present an alternative and simpler analysis of the algorithm that does not involve counting witness-trees.

July 22, 2014 12:41 AM

Design and Analysis of RS Sort

Authors: Harsh Ranjan, Sumit Agarwal
Download: PDF
Abstract: This paper introduces a new comparison base stable sorting algorithm, named RS sort. RS Sort involves only the comparison of pair of elements in an array which ultimately sorts the array and does not involve the comparison of each element with every other element. RS sort tries to build upon the relationship established between the elements in each pass. Suppose there is an array containing three elements a1, a2, a3 and if a relationship exist such that a1<a2 and a2<a3 then it can be established that a1<a3 and so there is no need to compare a1 and a3. Sorting is a fundamental operation in computer science. RS sort is analyzed both theoretically and empirically. We have performed its Empirical analysis and compared its performance with the well-known quick sort for various input types.

July 22, 2014 12:41 AM

How the Experts Algorithm Can Help Solve LPs Online

Authors: Anupam Gupta, Marco Molinaro
Download: PDF
Abstract: We consider the problem of solving packing/covering LPs online, when the columns of the constraint matrix are presented in random order. This problem has received much attention: the main open question is to figure out how large the right-hand sides of the LPs have to be (compared to the entries on the left-hand side of the constraint) to get $(1+\epsilon)$-approximations online? It is known that the RHS has to be $\Omega(\epsilon^{-2} \log m)$ times the left-hand sides, where $m$ is the number of constraints.

In this paper we give a primal-dual algorithm to achieve this bound for all packing LPs, and also for a class of mixed packing/covering LPs. Our algorithms construct dual solutions using a regret-minimizing online learning algorithm in a black-box fashion, and use them to construct primal solutions. The adversarial guarantee that holds for the constructed duals help us to take care of most of the correlations that arise in the algorithm; the remaining correlations are handled via martingale concentration and maximal inequalities. These ideas lead to conceptually simple and modular algorithms, which we hope will be useful in other contexts.

July 22, 2014 12:41 AM

Efficient Algorithms for the Closest Pair Problem and Applications

Authors: Sanguthevar Rajasekaran, Sudipta Pathak
Download: PDF
Abstract: The closest pair problem (CPP) is one of the well studied and fundamental problems in computing. Given a set of points in a metric space, the problem is to identify the pair of closest points. Another closely related problem is the fixed radius nearest neighbors problem (FRNNP). Given a set of points and a radius $R$, the problem is, for every input point $p$, to identify all the other input points that are within a distance of $R$ from $p$. A naive deterministic algorithm can solve these problems in quadratic time. CPP as well as FRNNP play a vital role in computational biology, computational finance, share market analysis, weather prediction, entomology, electro cardiograph, N-body simulations, molecular simulations, etc. As a result, any improvements made in solving CPP and FRNNP will have immediate implications for the solution of numerous problems in these domains. We live in an era of big data and processing these data take large amounts of time. Speeding up data processing algorithms is thus much more essential now than ever before. In this paper we present algorithms for CPP and FRNNP that improve (in theory and/or practice) the best-known algorithms reported in the literature for CPP and FRNNP. These algorithms also improve the best-known algorithms for related applications including time series motif mining and the two locus problem in Genome Wide Association Studies (GWAS).

July 22, 2014 12:41 AM

Hellinger volume and number-on-the-forehead communication complexity

Authors: Troy Lee, Nikos Leonardos, Michael Saks, Fengming Wang
Download: PDF
Abstract: Information-theoretic methods have proven to be a very powerful tool in communication complexity, in particular giving an elegant proof of the linear lower bound for the two-party disjointness function, and tight lower bounds on disjointness in the multi-party number-in-the-hand (NIH) model. In this paper, we study the applicability of information theoretic methods to the multi-party number-on-the-forehead model (NOF), where determining the complexity of disjointness remains an important open problem.

There are two basic parts to the NIH disjointness lower bound: a direct sum theorem and a lower bound on the one-bit AND function using a beautiful connection between Hellinger distance and protocols revealed by Bar-Yossef, Jayram, Kumar and Sivakumar [BYJKS04]. Inspired by this connection, we introduce the notion of Hellinger volume. We show that it lower bounds the information cost of multi-party NOF protocols and provide a small toolbox that allows one to manipulate several Hellinger volume terms and lower bound a Hellinger volume when the distributions involved satisfy certain conditions. In doing so, we prove a new upper bound on the difference between the arithmetic mean and the geometric mean in terms of relative entropy. We then apply these new tools to obtain a lower bound on the informational complexity of the AND_k function in the NOF setting. Finally, we discuss the difficulties of proving a direct sum theorem for information cost in the NOF model.

July 22, 2014 12:40 AM

Lower Bounds on the Oracle Complexity of Nonsmooth Convex Optimization via Information Theory

Authors: Gábor Braun, Cristóbal Guzmán, Sebastian Pokutta
Download: PDF
Abstract: We present an information-theoretic approach to lower bound the oracle complexity of nonsmooth black box convex optimization, unifying previous lower bounding techniques by identifying a combinatorial problem, namely string guessing, as a single source of hardness. As a measure of complexity we use distributional oracle complexity, which subsumes randomized oracle complexity as well as worst-case oracle complexity. We obtain strong lower bounds on distributional oracle complexity for the box $[-1,1]^n$, as well as for the $L^p$-ball for $p \geq 1$ (for both low-scale and large-scale regimes), matching worst-case lower bounds, and hence we close the gap between distributional complexity, and in particular, randomized complexity, and worst-case complexity. Furthermore, the bounds remain essentially the same for high-probability and bounded-error oracle complexity, and even for combination of the two, i.e., bounded-error high-probability oracle complexity. This considerably extends the applicability of known bounds.

July 22, 2014 12:40 AM

A Sane Proof that COLk \le COL3

Authors: William Gasarch
Download: PDF
Abstract: Let COLk be the set of all k-colorable graphs. It is easy to show that if a<b then COLa \le COLb (poly time reduction). Using the Cook-Levin theorem it is easy to show that if 3 \le a< b then COLb \le COLa. However this proof is insane in that it translates a graph to a formula and then the formula to a graph. We give a simple proof that COLk \le COL3.

July 22, 2014 12:40 AM


SessionStore in ring

From ring's source, SessionStore requires read-session to return an empty map if the key is not found:

(read-session [store key] "Read a session map from the store. If the key is not found, an empty map is returned.")

But in the memory namespace in MemoryStore it inplements read-session deferring the atom which returns nil if the key is not found.

Is there a reason not to implement the full spec of the protocol, or is it a missing feature?

by loki at July 22, 2014 12:34 AM



Lambda the Ultimate Forum

wither formal methods?

I sometimes get a bee in my bonnet to look for tools to do model driven design and the like. So, I found a list of verification and synthesis tools. The mind boggles. For little people such as myself, I wish there were a table somewhere that listed the range of features and then showed what each system did/not support. I want stuff that would help with application development: user interface and state machine (e.g. audio playback engine + ui controlling it) type modeling and code generation (and round tripping). Anybody know of such guidance for the laymen?

July 22, 2014 12:09 AM

Portland Pattern Repository

HN Daily

July 21, 2014

Planet Theory

Using finite automata to draw graphs

The diagram below describes a finite state machine that takes as input a description of an indifference graph, and produces as output a 1-planar drawing of it (that is, a drawing with each edge crossed at most once).

Indifference graphs are the graphs that can be constructed by the following process. Initialize an active set of vertices to be the empty set, and then perform a sequence of steps of two types: either add a new vertex to the active set and make it adjacent to all previous active vertices, or inactivate a vertex (removing it from the active set but not from the graph). Thus, they can be represented by a sequence of binary values that specify whether the next step is to add or inactivate a vertex. These values are the input to the finite state machine.

In a paper at WADS last year with Bannister and Cabello, we showed how to test 1-planarity for several special classes of graphs, but not for indifference graphs. Some of our algorithms involved proving the existence of a finite set of forbidden configurations, and that idea works here, too: an indifference graph turns out to be 1-planar if and only if, for every K6 subgraph, the first three vertices of the subgraph (in the activation order) have no later neighbors outside the subgraph, and the last three vertices have no other earlier neighbors. K6 is 1-planar, but it has essentially only one drawing (modulo permutation of the vertices), and any example of this configuration would have a seventh vertex connected to four of the K6 vertices, something that's not possible in a 1-planar drawing.

At one level, the state diagram above can be viewed as a diagram for detecting this forbidden configuration. Every right-going transition is one that adds an active vertex, and every left-going transition is one that removes an active vertex. If a transition of either type does not exist in the diagram, it means that a step of that type will lead to an inescapable failure state. But the only missing transitions are the ones that would create a six-vertex active set by a sequence of transitions that does not end in three consecutive right arrows (creating a K6 in which one of the last three vertices has an earlier neighbor) or the ones that would exit a six-vertex active set by a sequence of transitions that does not begin with three consecutive left arrows (creating a K6 in which one of the first three vertices has a later neighbor). So, this automaton recognizes only the graphs that have no forbidden configuration.

On another level, the drawings within each state of the diagram show how to use this finite state machine to construct a drawing. Each state is labeled with a drawing of its active vertices, possibly with a yellow region that represents earlier inactive parts of the drawing that can no longer be modified. The numbers on the vertices give the order in which they were activated. For each left transition, it is always possible to remove the oldest active vertex from the drawing and replace the parts of the drawing surrounding it by a yellow region to create a drawing that matches the new state. Similarly, for each right transition, it is always possible to add one more active vertex to the drawing, connect it to the other active vertices, and then simplify some parts of the drawing to yellow regions, again creating a drawing that matches the new state. Therefore, every graph that can be recognized by this state diagram has a 1-planar drawing.

Since the machine described by the diagram finds a drawing for a given indifference graph if and only if the graph has no forbidden configurations, it follows that these forbidden configurations are the only ones we need to describe the 1-planar graphs and that this machine correctly finds a 1-planar drawing for every indifference graph that has one. This same technique doesn't always generalize: A result from my WADS paper that it's NP-complete to test 1-planarity for graphs of bounded bandwidth shows that, even when a class of graphs can be represented by strings of symbols from a finite alphabet it's not always going to be possible to find a finite state machine to test 1-planarity. But it would be interesting to find more graph classes for which the same simple technique works.

July 21, 2014 11:54 PM


Scala ListViewSelectionChange event outputs unexpected range

I'm currently trying to use a scala.swing.ListView in a Scala application.

I basically want to get the selected item of a ListView, as soon as the selection changes.

Here a sample which can be pasted into the Scala prompt.

import scala.swing._

class MainView extends MainFrame {

  import ListView._
  val selector = new ListView(List[Int]()) {
    renderer = Renderer(x => x.toString)
    selection.reactions += {
      case event.ListSelectionChanged(source, range, _) =>
        if(!range.isEmpty) {
          println("selection " + range)
    selection.intervalMode = IntervalMode.Single

  selector.listData = 1 +: selector.listData
  selector.listData = 2 +: selector.listData
  selector.listData = 3 +: selector.listData
  selector.listData = 4 +: selector.listData
  selector.listData = 5 +: selector.listData

  contents = new GridBagPanel(){
    layout(new ScrollPane(selector))  = new Constraints() {
      gridx = 0
      gridy = 0
      weighty = 1
      fill = GridBagPanel.Fill.Vertical
  visible = true

var view = new MainView()

If I would now select the first element in the list, I would expect Range to contain only 0, if I then select the second element, I would expect only 1 and so on.

However, if I select the first and then the second element, Range contains (0, 1). If I select the fifth and then the third element, range contains (2, 3, 4).

So, range contains always the range between the last and the current selected item, in ascending order.

Sadly, the Scala documentation does not state how this API works.

Does anyone know what I'm doing wrong here? Might this be the wrong event for reacting on ListView selection changes?

by Emiswelt at July 21, 2014 11:11 PM



CS & Maths

Hey guys, I'm gonna be doing a joint honours cs and maths bsc in the UK soon and was just wondering if I've bitten off more than I can chew here.

First off I have very little programming experience but schools I applied to seemed to stress that you didn't need any although I have been trying to learn python over the summer on codecademy and have some LateX and HTML background (I know it doesn't count)...

On the math side of things I'm not THAT worried as I just got a 5 on Calc AB and according to my uni syllabus, the first semester maths is basically review for me along with some calc 2, vectors/matrices and ofc discrete maths.

My main motivation for doing cs AND math is that I really enjoyed calc and would like to finish it off along with the fact that the degree is pretty flexible.

Anyone think I'll survive if I put the work in? If a degree like this is attractive to employers? Any input, opinions or advice would be much appreciated! Thanks!!

submitted by pboy07
[link] [7 comments]

July 21, 2014 10:48 PM



Spark ClassNotFoundException running the master

I have downloaded and built Spark 0.80 using sbt/sbt assembly. It was successful. However when running ./bin/ the following error is seen in the log file

Spark Command: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -cp :/shared/spark-0.8.0-incubating-bin-hadoop1/conf:/shared/spark-0.8.0-incubating-bin-hadoop1/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop1.0.4.jar
/shared/spark-0.8.0-incubating-bin-hadoop1/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar -Djava.library.path= -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip mellyrn.local --port 7077 --webui-port 8080

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/master/Master
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.master.Master
        at Method)
        at java.lang.ClassLoader.loadClass(
        at sun.misc.Launcher$AppClassLoader.loadClass(
        at java.lang.ClassLoader.loadClass(

Update: after doing sbt clean (per suggestion below) it is running: see screenshot. enter image description here

by javadba at July 21, 2014 10:39 PM

Planet FreeBSD

Application awareness of receive side scaling (RSS) on FreeBSD

Part of testing this receive side scaling work is designing a set of APIs that allow for some kind of affinity awareness. It's not easy - the general case is difficult and highly varying. But something has to be tested! So, where should it begin?

The main tricky part of this is the difference between incoming, outgoing and listening sockets.

For incoming traffic, the NIC has already calculated the RSS hash value and there's already a map between RSS hash and destination CPU. Well, destination queue to be much more precise; then there's a CPU for that queue.

For outgoing traffic, the thread(s) in question can be scheduled on any CPU core and as you have more cores, it's increasingly unlikely to be the right one. In FreeBSD, the default is to direct dispatch transmit related socket and protocol work in the thread that started it, save a handful of places like TCP timers. Once the driver if_transmit() method is called to transmit a frame it can check the mbuf to see what the flowid is and map that to a destination transmit queue. Before RSS, that's typically done to keep packets vaguely in some semblance of in-order behaviour - ie, for a given traffic flow between two endpoints (say, IP, or TCP, or UDP) the packets should be transmitted in-order. It wasn't really done for CPU affinity reasons.

Before RSS, there was no real consistency with how drivers hashed traffic upon receive, nor any rules on how it should select an outbound transmit queue for a given buffer. Most multi-queue drivers got it "mostly right". They definitely didn't try to make any CPU affinity choices - it was all done to preserve the in-order behaviour of traffic flows.

For an incoming socket, all the information about the destination CPU can be calculated from the RSS hash provided during frame reception. So, for TCP, the RSS hash for the received ACK during the three way handshake goes into the inpcb entry. For UDP it's not so simple (and the inpcb doesn't get a hash entry for UDP - I'll explain why below.)

For an outgoing socket, all the information about the eventual destination CPU isn't necessarily available. If the application knows the source/destination IP and source/destination port then it (or the kernel) can calculate the RSS hash that the hardware would calculate upon frame reception and use that to populate the inpcb. However this isn't typically known - frequently the source IP and port won't be explicitly defined and it'll be up to the kernel to choose them for the application. So, during socket creation, the destination CPU can't be known.

So to make it simple (and to make it simple for me to ensure the driver and protocol stack parts are working right) my focus has been on incoming sockets and incoming packets, rather than trying to handle outgoing sockets. I can handle outbound sockets easily enough - I just need to do a software hash calculation once all of the required information is available (ie, the source IP and port is selected) and populate the inpcb with that particular value. But I decided to not have to try and debug that at the same time as I debugged the driver side and the protocol stack side, so it's a "later" task.

For TCP, traffic for a given connection will use the same source/destination IP and source/destination port values. So for a given socket, it'll always hash to the same value. However, for UDP, it's quite possible to get UDP traffic from a variety of different source IP/ports and respond from a variety of different source/IP ports. This means that the RSS hash value that we can store in the inpcb isn't at all guaranteed to be the same for all subsequent socket writes.

Ok, so given all of that above information, how exactly is this supposed to work?

Well, the slightly more interesting and pressing problem is how to break out incoming requests/packets to multiple receive threads. In traditional UNIX socket setups, there are a couple of common design patterns for farming off incoming requests to multiple worker threads:

  • There's one thread that just does accept() (for TCP) or recv() (for UDP) and it then farms off new connections to userland worker threads; or
  • There are multiple userland worker threads which all wait on a single socket for accept() or recv() - and hope that the OS will only wake up one thread to hand work to.
It turns out that the OS may wake up one thread at a time for accept() or recv() but then userland threads will sit in a loop trying to accept connections / packets - and then you tend to find they get called a lot only to find another worker thread that was running stole the workload. Oops.

I decided this wasn't really acceptable for the RSS work. I needed a way to redirect traffic to a thread that's also pinned to the same CPU as the receive RSS bucket. I decided the cheapest way would be to allow multiple PCB entries for the same socket details (eg, multiple TCP sockets listening on *:80). Since the PCBGROUPS code in this instance has one PCB hash per RSS bucket, all I had to do was to teach the stack that wildcard listen PCB entries (eg, *:80) could also exist in each PCB hash bucket and to use those in preference to the global PCB hash.

The idea behind this decision is pretty simple - Robert Watson already did all this great work in setting up and debugging PCBGROUPS and then made the RSS work leverage that. All I'd have to do is to have one userland thread in each RSS bucket and have the listen socket for that thread be in the RSS bucket. Then any incoming packet would first check the PCBGROUP that matched the RSS bucket indicated by the RSS hash from the hardware - and it'd find the "right" PCB entry in the "right" PCBGROUP PCB has table for the "right" RSS bucket.

That's what I did for both TCP and UDP.

So the programming model is thus:

  • First, query the RSS sysctl (net.inet.rss) for the RSS configuration - this gives the number of RSS buckets and the RSS bucket -> CPU mapping.
  • Then create one worker thread per RSS bucket..
  • .. and pin each thread to the indicated CPU.
  • Next, each worker thread creates one listen socket..
  • .. sets the IP_BINDANY or IP6_BINDANY option to indicate that there'll be multiple RSS entries bound to the given listen details (eg, binding to *:80);
  • .. then IP_RSS_LISTEN_BUCKET to set which RSS bucket the incoming socket should live in;
  • Then for UDP - call bind()
  • Or for TCP - call bind(), then call listen()
Each worker thread will then receive TCP connections / UDP frames that are local to that CPU. Writing data out the TCP socket will also stay local to that CPU. Writing UDP frames out doesn't - and I'm about to cover that.

Yes, it's annoying because now you're not just able to choose an IO model that's convenient for your application / coding style. Oops.

Ok, so what's up with UDP?

The problem with UDP is that outbound responses may be to an arbitrary destination setup and thus may actually be considered "local" to another CPU. Most common services don't do this - they'll send the UDP response to the same remote IP and port that it was sent from.

My plan for UDP (and TCP in some instances, see below!) is four-fold:

  • When receiving UDP frames, optionally mark them with RSS hash and flowid information.
  • When transmitting UDP frames, allow userspace to inform the kernel about a pre-calculated RSS hash / flow information.
  • For the fully-connected setup (ie, where a single socket is connect() ed to a given UDP remote IP:port and frame exchange only occurs between the fixed IP and port details) - cache the RSS flow information in the inpcb;
  • .. and for all other situations (if it's not connected, if there's no hint from userland, if it's going to a destination that isn't in the inpcb) - just do a software hash calculation on the outgoing details.
I mostly have the the first two UDP options implemented (ie, where userland caches the information to re-use when transmitting the response) and I'll commit them to FreeBSD soon. The other two options are the "correct" way to do the default methods but it'll take some time to get right.

Ok, so does it work?

I don't have graphs. Mostly because I'm slack. I'll do up some before I present this - likely at BSDCan 2015.

My testing has been done with Intel 1G and 10G NICs on desktop Ivy Bridge 4-core hardware. So yes, server class hardware will behave better.

For incoming TCP workloads (eg a webserver) then yes, there's no lock contention between CPUs in the NIC driver or network stack any longer. The main lock contention between CPUs is the VM and allocator paths. If you're doing disk IO then that'll also show up.

For incoming UDP workloads, I've seen it scale linearly on 10G NICs (ixgbe(4)) from one to four cores. This is with no-defragmentation, 510 byte sized datagrams.

Ie, 1 core reception (ie, all flows to one core) was ~ 250,000 pps into userland with just straight UDP reception and no flow/hash information via recvmsg(); 135,000 pps into userland with UDP reception and flow/hash information via recvmsg().

4 core reception was ~ 1.1 million pps into userland, roughly ~ 255,000 pps per core. There's no contention between CPU cores at all.

Unfortunately what I was sending was markedly different. The driver quite happily received 1.1 million frames on one queue and up to 2.1 million when all four queues were busy. So there's definitely room for improvement.

Now, there is lock contention - it's just not between CPU cores. Now that I'm getting past the between-core contention, we see the within-core contention.

For TCP HTTP request reception and bulk response transmission, most of the contention I'm currently seeing is between the driver transmit paths. So, the following occurs:

  • TCP stack writes some data out;
  • NIC if_transmit() method is called;
  • It tries to grab the queue lock and succeeds;
It then appends the frame to the buf_ring and schedules a transmit out the NIC. This bit is fine.

But then whilst the transmit lock is held, because the driver is taking frames from the buf_ring to push into the NIC TX DMA queue
  • The NIC queue interrupt fires, scheduling the software interrupt thread;
  • This pre-empts the existing running transmit thread;
  • The NIC code tries to grab the transmit lock to handle completed transmissions;
  • .. and it fails, because the code it preempted holds the transmit lock already.
So there's some context switching and thrashing going on there which needs to be addressed.

Ok, what about UDP? It turns out there's some lock contention with the socket receive buffer.

The soreceive_dgram() routine grabs the socket receive buffer (SOCKBUF_LOCK()) to see if there's anything to return. If not, and if it can sleep, it'll call sbwait() that will release the lock and msleep() waiting for the protocol stack to indicate that something has been received. However, since we're receiving packets at such a very high rate, it seems that the receive protocol path contends with the socket buffer lock that is held by the userland code trying to receive a datagram. It pre-empts the user thread, tries to grab the lock and fails - and then goes to sleep until the userland code finishes with the lock. soreceive_dgram() doesn't hold the lock for very long - but I do see upwards of a million context switches a second.

To wrap up - I'm pleased with how things are going. I've found and fixed some issues with the igb(4) and ixgbe(4) drivers that were partly my fault and the traffic is now quite happily and correctly being processed in parallel. There are issues with scaling within a core that are now being exposed and I'm glad to say I'm going to ignore them for now and focus on wrapping up what I've started.

There's a bunch more to talk about and I'm going to do it in follow-up posts.
  • what I'm going to do about UDP transmit in more detail;
  • what about creating outbound connections and how applications can be structured to handle this;
  • handling IP fragments and rehashing packets to be mostly in-order - and what happens when we can't guarantee ordering with the hardware hashing UDP frames to a 4-tuple;
  • CPU hash rebalancing - what if a specific bucket gets too much CPU load for some reason;
  • randomly creating a toeplitz RSS hash key at bootup and how that should be verified;
  • multi-socket CPU and IO domain awareness;
  • .. and whatever else I'm going to stumble across whilst I'm slowly fleshing this stuff out.
I hope to get the UDP transmit side of things completed in the next couple of weeks so I can teach memcached about TCP and UDP RSS. After that, who knows!

by adrian at July 21, 2014 10:37 PM


Non-Approximate Dynamic All-Pairs Shortest Path algorithm for Undirected, Unweighted Graphs?

I am looking for an algorithm involving adding unweighted edges to an empty, undirected graph (with vertices) and then for each, updating the table of shortest paths.

An example is if we have vertices 1 to 5. We can see that initially, all of the paths are infinite except for the diagonal entries, i.e.:

  1 2 3 4 5
1 0 i i i i
2 i 0 i i i
3 i i 0 i i
4 i i i 0 i
5 i i i i 0

where i means "infinite". But if we add an edge (1, 2) then the 1-2nd and 2-1st entries of the table are updated:

  1 2 3 4 5
1 0 1 i i i
2 1 0 i i i
3 i i 0 i i
4 i i i 0 i
5 i i i i 0

Now, if we add the edge (2, 3), then obviously the 2-3rd and 3-2nd entires are updated, but also are 1-3rd and 3-1st (which has length 2, since the path is 1->2->3 and 3->2->1):

  1 2 3 4 5
1 0 1 2 i i
2 1 0 1 i i
3 2 1 0 i i
4 i i i 0 i
5 i i i i 0

Now, the more edges and vertices we add, the more complex the problem becomes, since we need to update other entries in the matrix if they are connected. The one fact that I know about this kind of graph is that it is symmetric along the diagonal.

I've been looking at this problem for a few hours and I do not think there is a generic algorithm that will solve the problem for a general unweighted, undirected graph. Are there resources on the problem?

by Ryan at July 21, 2014 10:33 PM


References on formalization

What are good references in the field of formalization, as applied to mathematics and computer science? I've found [1] to be quite complete, covering topics such as modal logic and some applications. Are there other references (not only books) following this line?

[1] Huth & Ryan, "Logic in Computer Science: Modelling and Reasoning about Systems"

by GmeSalazar at July 21, 2014 10:29 PM


Difference between hypervisor and exokernel

An hypervisor (1st type) is a software that creates and run virtual machines, managing guest's operative systems's requests to the hardware.
An exokernel is an operative system kernel, that let's programs access directly to the hardware or, with the support of specific libraries that implements abstactions, run different types of executables for that architecture.

Both the two are an abstraction layer between hardware and application software.
Both the two let you run on the same machine different executables/software/operative systems/whatever

So what's the difference between the two?

by Mignus at July 21, 2014 10:26 PM


Programmatically obtain ephemeral port with Akka

If I configure an ActorSystem in Akka to use an emphemeral tcp port (more specifically, I configure the http port to 0), is there a way to programmatically obtain this port once the ActorSystem is started?

Any attempts to create an actor using actorOf, and then printing out the actor path shows a locally-referenced actor. Additionally, I tried registering a RemoteLifeCycleEvent listener on the event stream, but this can only be done after the server has been started, and hence misses RemoteServerStarted event.

by hickin at July 21, 2014 10:23 PM

Spring Data Neo4j in Play 2.x app using Scala

I am currently working on a web service backed by Neo4j (v2.1.2, hosted on GrapheneDB) and Play (v2.3) using Scala (2.11.1) (hosted on Heroku).

Until now AnormCypher connects the Play app with the Neo4J instance and all things are working as expected.

However using AnormCypher doesn't feel quite right. I have to write a lot boiler plate code to convert the results I get from AnormCypher to the Scala classes which are used for the business logic. I would prefer using something similar to Spring Data Neo4j.

The code in the examples looks very clean to me and I want to try it in project.

All these technologies are new to me so I need some help:

  • How can I use Spring Data Neo4j in the play app (build.sbt, plugin.sbt, ...)?
  • Which difficulties do I have to expect regarding the fact that I am not using the whole Spring stack?
  • Which drawbacks does it have to integrate a Java based framework in a Scala app ?

by Robin de Bois at July 21, 2014 10:21 PM



How to display classpath used for "sbt run"?

How can the classpath used for run task be displayed under SBT 0.13?

I have found some info here!msg/simple-build-tool/0rhVRPnjyZU/DOYAd14gh1wJ:

I was dumping my classpaths as a way to troubleshoot my build recently and maybe this task can help you too:

lazy val printClasspath = task { this.runClasspath.getPaths.foreach(println); None }

runClasspath is a PathFinder [1] instance and you can interrogate it even further. I guess building a ':'-separated list of those paths will be easy.

I don't understand where and how I should use this tip. Please advise.

by Oleg Golovanov at July 21, 2014 10:01 PM


Perfect hashing family variation - injectivity on $r$ disjoint sets

We denote by $[t]$ the set $\{1,2,\ldots,t\}$.

A $(n,k)$-perfect hashing family is a set of functions $H=\{h_i:[n]\to[k]\}$ such that for every set $S\subset [n], |S|\leq k$, there exists some $h_S \in H$ such that $h_S$ is injective on $S$.

There are known (deterministic) builds of such family of size $O(e^{k+log^2k}log n)$, while there is a $\Omega(e^k \cdot \frac{log n}{\sqrt k})$ lower bound , which is almost tight.

We define an extension of such families as follows:

A family of functions $\mathcal F=\{f_i:[n]\to [k]\}$ is a $(n,r,k)$-perfect hashing family if for every set of sets, $\mathcal S=\{S_1,S_2,\ldots,S_r\subseteq [n]\}$, such that $$|\mathcal S|=r,\ \forall i:|S_i|=k,\ \forall j\neq i:S_i\cap S_j = \emptyset$$

There exists a function $f\in \mathcal F$ which is injective on all $S_i\in \mathcal S$.

That is, every set of $r$ disjoint sets of indices of size $k$ there exists a single function which is injective on all of them.

What is the smallest $(n,r,k)$-perfect hashing family we can build?

Specifially, I'm mainly interested in the $r=2$ case, but this might be useful to others for general $r$ values.

by R B at July 21, 2014 09:59 PM


Historic Value at Risk - Ratios vs. Differences

Quick Summary on Historic VaR
Let $S_0,...,S_n$ be the daily values of some stock (where $S_0$ is the current value). Then for $i=1,\ldots,n$ we let $$\hat r_i:=S_{i-1}/S_i \quad \text{and}\quad \hat S_i := S_0\cdot \hat r_i$$ Now we can estimate e.g. the 95% 1-day-VaR by looking at the $(0.95n)$-th smallest number amongst all the scenarios $\hat S_1,...,\hat S_n$ and then subtracting $S_0$.

The above approach works fine when we look at stocks since $S_i>0$ for all $i$. But what if we consider an interest rate that is potentially close to zero, and even worse, may go negative (which is the case for the German short-term treasury bills at the moment), then we have that the $\hat r_i$ become very large and potentially negative, which renders the scenarios $\hat S_i$ completely useless.

One solution might be to look at the differences, i.e. $\overline r_i := S_{i-1}-S_i$ and $\overline S_i = S_0+\overline r_i$. But this completely ignores the order of magnitude of an asset. So my questions are:

1) Does anybody have an idea as to what approach is usually used in practice

2) Are differences a sensible approach at all?

3) Are there potentially other methods to avoid this problem?

by Tom at July 21, 2014 09:58 PM


Examples of C# 5.0 async/await vs. Akka actors vs. basic fork/join?

Both C# and Scala have adopted frameworks for simplifying doing asynchronous/parallel computation, but in different ways. The latest C# (5.0, still in beta) has decided on an async/await framework (using continuation-passing under the hood, but in an easier-to-use way), while Scala instead uses the concept of "actors", and has recently taken the actors implementation in Akka and incorporated it into the base library.

Here's a task to consider: We receive a series of requests to do various operations -- e.g. from user input, requests to a server, etc. Some operations are fast, but some take awhile. For the slow ones, we'd like to asynchronously do the operation (in another thread) and process it when the thread is done, while still being free to process new requests.

A simple synchronous loop might be (pseudo-code):

while (1) {
  val request = waitForAnything(user_request, server_request)
  val result = do_request(request)
  if (result needs to be sent back)

In a basic fork/join framework, you might do something like this (pseudo-code):

val threads: Set[Thread]

while (1) {
  val request = waitForAnything(user_request, server_request, termination of thread)
  if (request is thread_terminate) {
    val result = request.thread_result
    if (result needs to be sent back)
  } else if (request is slow) {
    val thread = new Thread(() => do_request(request))
  } else {
    val result = do_request(request)
    if (result needs to be sent back)

How would this look expressed using async/await and using actors, and more generally what are the advantages/disadvantages of these approach?

by Urban Vagabond at July 21, 2014 09:42 PM


Explanation of Tag, Index, and Offset in Direct Mapping Cache

I'm going through an exercise trying to store address references into a direct mapped cache with 128 blocks and a block size of 32 bytes. The address are 20000, 20004, 20008, and 20016 in base 10.

When I start to store them, I realize that the offset is the only value that changes. All of the examples I find determine hits and misses based on the target and index values, without mention of offset. What is the purpose of the offset? If the tag and index are in the cache, is it a hit regardless of the offset? Or is the offset also looked at?

by dddd at July 21, 2014 09:39 PM



Creating a tree from a text file in json - Clojure

I have a text file consisting of a json value in each line. My file is as follows:


In the json given above, we the family attribute indicates a hierarchy, we would have "root" as the root,"b" as child of the "root","c" as child of "b" and and "d" would be child of "b" as well.

The idea is to iterate through the file and add the node being read at its proper place in a tree. One way would be to read these entries into a "map" and then use this map for any tree-manipulations. For any complex tree manipulations, I am not sure how this would work. It is quite possible that based on a certain requirement, I might have to detach a child from an existing parent and attach it to another parent. Apparently Clojure zippers are supposed to help in this regard but I am getting a little confused by how the hierarchy of nodes work in zippers.

It will be great if somebody could point me in the right direction here.

by sc_ray at July 21, 2014 09:22 PM

How can I get Clojure :pre & :post to report their failing value?

(defn string-to-string [s1] 
  {:pre  [(string? s1)]
   :post [(string? %)]}

I like :pre and :post conditions, they allow me to figure out when I have put "square pegs in round holes" more quickly. Perhaps it is wrong, but I like using them as a sort of poor mans type checker. This isn't philosophy though, this is a simple question.

It seems in the above code that I should easily be able to determine that s1 is a function argument in the :pre condition. Similarily, % in the :post condition is always the function return value.

What I would like is to print the value of s1 or % when either of these respective conditions fail within the AssertionError. So I get something like

(string-to-string 23)

AssertionError Assert failed: (string? s1) 
(pr-str s1) => 23 

With the AssertionError containing a single line for every variable that was identified as being from the function argument list and that was referenced in the failing test. I would also like something similar when the return value of the function fails the :post condition.

This would make it trivial to quickly spot how I misused a function when trying to diagnose from the AssertionError. It would at least let me know if the value is nil or an actual value (which is the most common error I make).

I have some ideas that this could be done with a macro, but I was wondering if there was any safe and global way to basically just redefine what (defn and (fn and friends do so that :pre and :post would also print the value(s) that lead to the test failing.

by Stephen Cagle at July 21, 2014 09:18 PM



What are the differences between Scala middleware choices?

Note: Unfortunately this question was closed, but I'm trying to maintain it for if someone else comes along with the same question.

I've been looking for a good solution to developing a service in Scala that will sit between mobile devices and existing web services.

The current list of viable options are:

There are probably more options out there. How does one decide which one to use? What are the traits (excuse the pun ;-) of a good Scala middleware choice. On the one side, I would like to go for Akka, because it is part of the TypeSafe Scala stack, but on the other, something like Finagle has a rich set of libraries and makes plumbing so easy. Spray looks nice and simple to use.

Any advice, insights or experience would be greatly appreciated. I'm sure someone out there must have some experience with some of these that they won't mind sharing.


I would love for this question to be reopened. A good answer to this question will help new Scalateers to avoid related pitfalls.


These are my own experiences since asking this question:

Finagle - I used Finagle for a project and it's rock solid.

Spray - In my latest project I'm using Spray and I'm extremely happy. The latest releases are built on Akka 2 and you can run it directly with the Spray-can library which removes the need for a web server. Spray is a set of libraries, rather than a framework and is very modular. The Learn about Spray: REST on Akka video gives a great overview, and this blog at Cakesolutions shows a really nice development approach and architecture.


Life moves pretty fast. If you don't stop and look around once in a while, you could miss it. - Ferris Bueller

These days the choice has become simpler. In my humble opinion Spray has won the battle. It is being integrated into Akka to become the next Akka HTTP. I have been using Spray now on multiple projects and can honestly say that it's fantastic and best supported software I have ever encountered.

This does not answer the initial question, but at least gives some indication on why Spray seems like the best choice in most cases. It is extremely flexible, non-blocking and very stable. It has both client-side and server-side libraries and a great testkit. Also, have a look at these stats to get an idea on performance: Web Framework Benchmarks

by JacobusR at July 21, 2014 09:03 PM


How to get Geometric Brownian Motion's closed-form solution in Black-Scholes model?

The Black Scholes model assumes the following dynamics for the underlying, well known as the Geometric Brownian Motion: $$dS_t=S_t(\mu dt+\sigma dW_t)$$

Then the solution is given: $$S_t=S_0\,e^{\left(\mu-\frac{\sigma^2}{2}\right)t+\sigma W_t}$$

It can be shown by Ito Lemma on function $f(t,W_t)=\ln S_t$ that this solution is correct as it leads to above dynamics.

But how do we solve the above SDE originally to find this solution?

Guessing the above solution to apply Ito seems unlikely to me.

by emcor at July 21, 2014 09:01 PM


Portland Pattern Repository


Scala class with covariance in derivable type of parameter

I want this typeclass:

abstract class Model[U](val query: TableQuery[ModelTable[U]]) {
    // ...

But with covariance in the type of query.

The idea is that I have functions that take an instance of U, and need access to an instance of a corresponding TableQuery[ModelTable[U]]. Adding a type parameter is nasty, because I can no longer write functions using context bounds:

def f[U : Mode](u: U) = // ...

and writing out the implicit parameters everytime is quite hassle. And as for now I cannot see why this would be impossible. Given a concrete type for U, the type constraint for query should be trivial to derive for the compiler.

Maybe more concrete: why can this not compile:

// error: Unbound Wildcard Type
abstract class Model[U](val query: _ <: TableQuery[ModelTable[U]]) {
    // ...

by A.J.Rouvoet at July 21, 2014 08:44 PM

Changing multiple sbt settings in a given scope in build.sbt

I want to group settings by the context in which they apply in my build.sbt. Instead of

foo in ThisBuild := bar

baz in ThisBuild := bar

bar in Compile := 5

quux in Compile := 7

I'd like something that roughly looks like (don't care much about the specific syntax/API, just the conceptual grouping and lack of repetition)

in(ThisBuild) {
  foo := bar
  baz := bar

in(Compile) {
  bar  := 5
  quux := 7

Is such a thing possible/clean/idiomatic?

by copumpkin at July 21, 2014 08:36 PM



Preferred way to create a Scala list

There are several ways to construct an immutable list in Scala (see contrived example code below). You can use a mutable ListBuffer, create a var list and modify it, use a tail recursive method, and probably others that I don't know about.

Instinctively, I use the ListBuffer, but I don't have a good reason for doing so. Is there a preferred or idiomatic method for creating a list, or are there situations that are best for one method over another?

import scala.collection.mutable.ListBuffer

// THESE are all the same as: 0 to 3 toList.
def listTestA() ={
    var list:List[Int] = Nil

    for(i <- 0 to 3) 
        list = list ::: List(i)

def listTestB() ={
    val list = new ListBuffer[Int]()

    for (i <- 0 to 3) 
        list += i

def listTestC() ={
    def _add(l:List[Int], i:Int):List[Int] = i match {
        case 3 => l ::: List(3)
        case _ => _add(l ::: List(i), i +1)
    _add(Nil, 0)

by agilefall at July 21, 2014 08:27 PM


The Great Whitespace Debate

I filed this as ‘rant’ and ‘satire’ because it’s not serious, but I absolutely love the writing here.


by steveklabnik at July 21, 2014 08:18 PM


Lein test failing with No such var: leiningen.util.injected/add-hook

I get this error in a new project, without touching tests or test related configuration

Exception in thread "main" java.lang.RuntimeException: No such var: leiningen.util.injected/add-hook, compiling:(NO_SOURCE_PATH:1) at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler$InvokeExpr.parse( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler$BodyExpr$Parser.parse( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler$IfExpr$Parser.parse( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler$BodyExpr$Parser.parse( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler$BodyExpr$Parser.parse( at clojure.lang.Compiler$TryExpr$Parser.parse( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler$BodyExpr$Parser.parse( at clojure.lang.Compiler$FnMethod.parse( at clojure.lang.Compiler$FnExpr.parse( at clojure.lang.Compiler.analyzeSeq( at clojure.lang.Compiler.analyze( at clojure.lang.Compiler.eval( at clojure.lang.Compiler.eval( at clojure.lang.Compiler.eval( at clojure.core$eval.invoke(core.clj:2795) at clojure.main$eval_opt.invoke(main.clj:296) at clojure.main$initialize.invoke(main.clj:315) at clojure.main$null_opt.invoke(main.clj:348) at clojure.main$main.doInvoke(main.clj:426) at clojure.lang.RestFn.invoke( at clojure.lang.Var.invoke( at clojure.lang.AFn.applyToHelper( at clojure.lang.Var.applyTo( at clojure.main.main( Caused by: java.lang.RuntimeException: No such var: leiningen.util.injected/add-hook at clojure.lang.Util.runtimeException( at clojure.lang.Compiler.resolveIn( at clojure.lang.Compiler.resolve( at clojure.lang.Compiler.analyzeSymbol( at clojure.lang.Compiler.analyze( ... 42 more

by Federico Tomassetti at July 21, 2014 08:15 PM



Advice for Real World Application of Encryption Strategy?

Person A has a small piece of data they want to keep secure, even from themselves for 3 months.

Person B wants to ensure person A can secure their data for that amount of time.

Person C wants person A's data. Person C is not above stealing from person A if they can.

Person A trusts person B but knows person B could be compromised by person C.


My initial attempt to solve this was by suggesting split key encryption:

The data is encrypted and A and B get a key (using something like Cloudflare's Red October).

B won't release his key to A until 3 months have passed.

Downsides are though:

What if B dies?

Because C can compromise B, if C steal's A's key it could all be over.


So I'm trying to find a better plan, any suggestions?

submitted by pikadrew
[link] [2 comments]

July 21, 2014 08:04 PM


Portland Pattern Repository




Video lectures on type systems [migrated]

For my job, I need to pick up a working understanding of the implementation of type systems (in particular, how to write typing rules based on a design document). I've been given a copy of Types and Programming Languages and that's great, but I'd really like to watch someone work through it.

In my school, the course that teaches this is called "Programming Languages," but searching for "video lectures programming languages" turns up lots of stuff about how to learn C++, which I'm not interested in. I couldn't find anything on MIT Open Courseware.

I'd really like video lectures that go through TAPL, but I'll settle for anything useful. Are there good resources out there?

by Patrick Collins at July 21, 2014 07:40 PM



Compare two Maps in Scala

Is there any pre-defined function that I can use to compare two Maps based on the key and give me the difference? Right now, I iterate Map1 and foreach key, I check if there is an element in Map2 and I pattern match to find the difference. Is there a much elegant way to do this?

by user3102968 at July 21, 2014 07:30 PM




Spark Tachyon: How to delete a file?

In Scala, as an experiment I create a sequence file on Tachyon using Spark and read it back in. I want to delete the file from Tachyon using the Spark script also.

val rdd =  sc.parallelize(Array(("a",2), ("b",3), ("c",1)))
val rdd2 = sc.sequenceFile[String,Int]("tachyon://")

I don't understand the Scala language very well and I cannot find a reference about file path manipulation. I did find a way of somehow using Java in Scala to do this, but I cannot get it to work using Tachyon.

new File("tachyon://").delete()

by bjjer at July 21, 2014 07:14 PM

Why is the _0 Nat in Shapeless a class instead of an object?

I'm trying to understand Shapeless, and I came across this:

 * Base trait for type level natural numbers.
 * @author Miles Sabin
trait Nat {
  type N <: Nat

 * Encoding of successor.
 * @author Miles Sabin
case class Succ[P <: Nat]() extends Nat {
  type N = Succ[P]

 * Encoding of zero.
 * @author Miles Sabin
class _0 extends Nat {
  type N = _0

_0 is a special and unique case, like Nil for a List. _0 has no predecessor. Why isn't it an object/case object (which is a singleton)? HLists seem to do this.

 * `HList` ADT base trait.
 * @author Miles Sabin
sealed trait HList

 * Non-empty `HList` element type.
 * @author Miles Sabin
final case class ::[+H, +T <: HList](head : H, tail : T) extends HList {
  override def toString = head+" :: "+tail.toString

 * Empty `HList` element type.
 * @author Miles Sabin
sealed trait HNil extends HList {
  def ::[H](h : H) = shapeless.::(h, this)
  override def toString = "HNil"

 * Empty `HList` value.
 * @author Miles Sabin
case object HNil extends HNil

by jordan3 at July 21, 2014 07:03 PM





Get only the Header using the WS library from play framework 2.2.1. (Scala)

I am stuck using the 2.2.1 version of the play framework and I am trying to check the Content-Length of a request for a file using the WS libraries without actually downloading the body of the request.

In the version 2.3.x of play there has been a new method added getStream() that lets you process the request yourself with a stream, but I am stuck on version 2.2.1 for now and cannot find a similar way.

So far I have tried this but it never seems to get a response:

import{ResponseHeaders, WS}
import play.api.libs.iteratee.Concurrent.joined

WS.url(url).get {
  (rsp: ResponseHeaders) =>
     val (wsConsumer, _) = joined[Array[Byte]]
     val size = rsp.headers.get("Content-Length").map(_.head).fold(0L)(_.toLong)"Size: $size")

by Wextux at July 21, 2014 06:52 PM

Having trouble with implicit conversion in scala

Having this code

case class Workspace(ident: Long, name: String)
case class Project(ident: Long, name: String)

implicit def workspaceJSON: JSONR[Workspace] = new JSONR[Workspace] {
  def read(json: JValue) =
    Workspace.applyJSON(field[Long]("id"), field[String]("name"))(json)

implicit def projectJSON: JSONR[Project] = new JSONR[Project] {
  def read(json: JValue) =
    Project.applyJSON(field[Long]("id"), field[String]("name"))(json)

def parseEnt[T: JSONR](json: JValue): Either[String, T] =
  fromJSON[T](json){ _.toString }

def fetchProjects(ws: Workspace): Either[String, Project] = {
  parseEnt(parse("some text"))

Which fails to compile on parseEnt(parse("some text")) with

ambiguous implicit values:  both method taskJSON in class Fetcher of type =>
Fetcher.this.JSONR[types.Task]  and method workspaceJSON in class Fetcher of type =>
Fetcher.this.JSONR[Fetcher.this.Workspace]  match expected type Fetcher.this.JSONR[T]

Is there a way to assure scala, that in this case I want type variable T to be a Project and choose projectJSON function to parse it? Or if I'm doing it wrong, then how do it in right way?

by cvb at July 21, 2014 06:44 PM

presentation compiler: type completion in method call with more than one argument

I load source into compiler with askReload, and then i try to call askTypeCompletion after . (dot). Completion in first case (method with 2 arguments) is not working, but completion in method call with one arg works as expected.

  val list = Seq(1,2)
  def add(x: Int, y: Int): Int = x + y
  def minusOne(x: Int) = x - 1

  add(list.<completion is not working)
  minusOne(list.<works fine>)

what's interesting is if i have code:

  implicit class OptionW[T](opt: Option[T]) {
    def cata[A](some: T => A, none: A) = getOrElse none

  Option("").cata(x => x.<not working>)

completion after dot is not working again, but if i type comma after dot and then try again to complete after dot, it works: Option("").cata(x => x.<works!>,) Is it some bug or expected behaviour?

by wedens at July 21, 2014 06:41 PM

Spark - scala: shuffle RDD / split RDD into two random parts randomly

How can I take a rdd array of spark, and split it into two rdds randomly so each rdd will include some part of data (lets say 97% and 3%).

I thought to shuffle the list and then shuffledList.take((0.97*rddList.count).toInt)

But how can I Shuffle the rdd?

Or is there a better way to split the list?

by webSpider at July 21, 2014 06:33 PM



How to append or prepend an element to a tuple in Scala

I have a tuple and want to add an element without loosing type safety. This is what I want to achieve:

val tuple = ("", 1, 1f) // (String, Int, Float)

val newTuple:(String, Int, Float, Double) = tuple :+ 1d

by EECOLOR at July 21, 2014 06:26 PM



Masters in Computer Science, non-computer undergrad degree (Criminal Justice)

I recently graduated with a 3.92 in criminal justice (BS). I am considering looking into a masters in computer science program since honestly my degree is fairly useless on its own. I know some schools offer bridge programs and such (I'm in the NY-NJ area) but don't really know where to begin or if this is even a good idea. Anyone have any advice? My reason for pursuing a computer science degree is that I have an interest in programming (have only recently started on my own) and have a love for computers in general. I also think that it will greatly help my career and life and doing this now while I'm young will help me greatly in the future. Thank you.

submitted by pattywagon
[link] [2 comments]

July 21, 2014 06:19 PM



Computer resource under-usage [on hold]

here is a list of the core components of my Dell computer:

Processor: dual core Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz RAM: 96 GB System: Windows 7 64-bit Hard drives: 2TB SATA Video: NVIDIA Quadro 600

I'm trying to make the most use out of these resources. I'm running a few high intensity applications and processes right now, but I see in my Task Manager that the system is only using around 10% (or less) of both the CPU and RAM. (See screenshot) Why is this? Can somebody help me understand where the bottleneck is on my system?

enter image description here



by user20291 at July 21, 2014 05:57 PM



Is it possible to have two or more context bound classes in Scala

How do I specify that I need e.g. a ClassTag AND an Ordering for T?

Example def sort[T: ClassTag <and> Ordering](future: Future[Seq[T]]): Future[Seq[T]]

by longliveenduro at July 21, 2014 05:52 PM



Reading web page with log in (clojure)

I need to download some contents of a web page daily and I plan on using enlive for that. Trouble is, that I need to log in in with a POST first and the authentification of the page that I am interested in is then done with the session's cookies. So I can't just use

(html/html-resource ( url))

I didn't find a way to do this in clojure. Otherwise doing the reading in Java would be fine as well. In the end it should work as a worker on heroku.


by majnu at July 21, 2014 05:42 PM


What are you working on this week?

It’s Monday, so it is time for our weekly “What are you working on?” thread. Please share links and tell us about your current project. Do you need feedback, proofreading, collaborators?

by zhemao at July 21, 2014 05:41 PM


Deterministic Buchi + its complement covers LTL?

It is well known that deterministic Buchi automata (DBA) are less expressive than non-deterministic Buchi automata (NBA), and in particular DBA are not enough to cover linear temporal logic (LTL). However, the argument often used is that DBA are not closed under complementation - that is there is a language $\mathcal D$ recognized by DBA, but $\mathcal D^c$ is not recognized by any DBA. As an example, $\mathcal D = (b^*a)^\omega$ and $\mathcal D^c = (a+b)^*a^\omega$.

I wonder though, whether for any $\omega$-regular language $\mathcal L$ it holds either $\mathcal L$ is recognized by some DBA, or $\mathcal L^c$ is recognized by some DBA. If not, what is an example of a language which is not recognizable by DBA and which complement is not recognizable either?

by Ilya at July 21, 2014 05:37 PM



g2k14: Ken Westerback on DHCP and dump(8)

Having missed Ljubljana 1, I looked forward to Ljubljana 2 with great expectations. I was not disappointed! Mitja ran a great hackathon with a nice site and an excellent city around it.

July 21, 2014 05:35 PM


Add plugins under a same project in sbt

I'm trying to build a Scala project aggregated by multiple projects, one of which is an sbt plugin. I would like to use this plugin in another subproject under the same parent project, but I don't quite understand how to do this.

My "build.sbt" in the project root is like this:

lazy val plugin = project
  .settings(common: _*)
  .settings(name := "My plugin",
    sbtPlugin := true)

lazy val examples = project
  .dependsOn(lib, plugin)
  .settings(common: _*)
  .settings(name := "Examples")

How to add the plugin as a plugin to project examples?

by lastland at July 21, 2014 05:29 PM

Why does Clojure's for macro only accept one body expression?

Clojure's for macro accepts two arguments: a sequence of binding forms, and a body expression. As such, if I want to do multiple things in the loop, I have to wrap multiple expressions in a do block to make them a single expression.


(doseq [a (range 3)]
  (prn 'a a)
  (inc a))


(for [a (range 3)]
  (prn 'a a)
  (inc a))

The doseq works as expected. The for complains:

clojure.lang.ArityException: Wrong number of args (3) passed to: core/for

My question is, why doesn't this work? Why didn't the designer(s) of Clojure allow multiple "body" expressions in the for loop, like they did in doseq and when? It's not as if there is any semantic ambiguity, right?

by Jeff Terrell at July 21, 2014 05:26 PM


plotting High Frequency finance data using quantmod [on hold]

Hi I have data in XTS format like this

 >  ITXxts 

 Time                  Price   Volume 
 2014-07-18 09:00:00    67.63   460
 2012-04-27 09:00:00    67.63   73
 2012-04-27 09:00:00    67.63   85
 2012-04-27 09:00:01    67.63   3
 2012-04-27 09:01:03    67.79   1
 2012-04-27 09:01:16    67.64   91
 2012-04-27 09:05:50    67.83   75

Now aggregated the series into 5 minute blocks

tsagg5min =aggregatets(ITXxts,on="minutes",k=5)

2014-07-18 09:05:00    67.45   43
2014-07-18 09:10:00    67.82   12
2014-07-18 09:15:00    67.91   341
2014-07-18 09:20:00    67.70   275


enter image description here

Now my question is how can I plot it in the OHLC format in such a way that It calculates the the Open High Low and Close of each 5 minute chunk in which the data was aggregated

by Rup Mitra at July 21, 2014 05:24 PM


Lower bounds for Polynomials computing the boolean functions

Expressing a boolean function $f$ $:\{ 0,1 \}^{n} \rightarrow \{0,1 \}$ using a polynomial $P(x_{1},...,x_{n})$, where $x_{1},...,x_{n}$ may be integer, finite fields, or other fields.

One of the most popular examples is Fourier Analysis e.g. .

Another topic is transforming small depth circuits to polynomials over integer with small degree and magnitudes of small weights. For example, transformation from ACC circuits to polynomials proved by Yao and Beigel and Tarui is notable.

My questions are:

Qestion A. Is there a list of results about lower bounds and remarkable techniques for polynomials computing boolean functions. I know the Williams's separation between NEXP and ACC.

Question B. Does the recent separation result about ACC $\circ $ THR circuits by Williams contain some interpretation relating with polynomials computing boolean funcitons ?

Question C. Is proving lowerbounds for the number of monomials or degrees or other formally defined compleixty measures relating wtih the algebraic compleixty theory and arithmetic circuit complexity established by Valiant with VP v. VNP problem??

I am reading the survey written by Shiplka, and have met a defference between boolean compleixty and arithmetic compleixty: semantic versus syntatic.

Here is a response to the J's question.

The survey has a remark about defference between polynomial and polynomial function.

Taking an example, $x^{2}-x$ is a polynomial which is NOT identicdally zero but is a polynomial function $GF(2)^{n}\rightarrow GF(2)$ identically zero over $GF(2)$.

In algebraic compleixty, Valiant's question is about compleixyt of expressioning polynomial (not polynomial funciton).

However, I mainly want to consider polynomial funcitons. Thanks.

by shen at July 21, 2014 05:23 PM


How to price a Swing Option?

I'm working in the commodity market and I've to price Swing Options with MATLAB, preferably with finite element.

Has anyone already priced these kind of derivatives?

I'm thinking about using the structure for the pricing of an American Option and then do it iteratively.

More details about Swing Options are included in this paper.

Note that swing options are really useful in commodity markets because you can exercise them more than once (like American options); obviously there are some constraints that limit you.

I've already tried to price them with Least Squares Monte Carlo method (using the algorithm presented by Longstaff and Schwartz).

Now I want to price them with finite element but I'm having some difficulties. In particular I'm pricing them without jumps, so I'm using an EDP discretized (and not a PIDE).

I'd like to know if anyone already implemented such a thing?

by alberto at July 21, 2014 05:21 PM


scala boolean variables need to change dynamicly

I have this method that I need to evaluate an expression based on these Boolean values:

var Ma, Co, Ar, Le, St, Ri: Boolean = false
def Logic: Boolean =
  (Ma  &  (!Ma  |  Co)  &  (!Ma  |  Ar)  &  (!Co  |  Ma)  &  
      (!Ar  |  Ma)  &  (!Ar  |  Le  |  St |  Ri))

In main() method I have:

def main(args: Array[String]) {   
  val ls: List[String] = List("Ma", "Co", "Ar")
  //now I need to change the value of Ma, Co and Ar variables to "true"??

Is there a general way that may help to change the value of only these Boolean variables to true that are found in this list?

by Heva at July 21, 2014 05:21 PM

How to pattern-match an `Option` in the underlying?

It says in Scala, pattern-matching will be converted to an invocation of unapply(...):Option[...].

Here I defined an object with unapply:

object Hello {
  def unapply(s:String): Option[String] = Some(s)

Then I can use it in pattern-matching:

"hello" match {
   case Hello(s) => println(s)

Actually(maybe) it will be converted to:

Hello.unapply("hello") match {
  case Some(s) => println(s)
  case _ =>

But the Some here is also a case class, so the Some.unapply will be called.

That should be infinite. Which is not possible.

So how do Scala pattern matching the Option in the underlying?

by Freewind at July 21, 2014 05:09 PM

Surprising behaviour related to records, protocols and compilation

I encountered a, to me, slightly surprising behaviour seemingly related to clojure records.

The setup is as follows:

  1. One namespace defines a record type:

    (ns defrecordissue.arecord)
    (defrecord ARecord [])
  2. Another namespace defines a protocol, and extends it to the record type defined in 1:

    (ns defrecordissue.aprotocol
      (:require [defrecordissue.arecord])
      (:import [defrecordissue.arecord ARecord]))
    (defprotocol AProtocol
      (afn [this]))
    (extend-protocol AProtocol
      (afn [this] 42))
  3. A third namespace constructs an instance of the record and invokes the protocol function on the record.

    (ns defrecordissue.aot1
      (:require [defrecordissue.aprotocol]
    (defrecordissue.aprotocol/afn (defrecordissue.arecord/->ARecord))

When the defrecordissue.aot1 namespace is compiled, in my case using lein compile defrecordissue.aot1, compilation fails with the following exception:

Exception in thread "main" java.lang.IllegalArgumentException: No implementation of method: :afn of protocol: #'defrecordissue.aprotocol/AProtocol found for class: defrecordissue.arecord.ARecord, compiling:(aot1.clj:5:1)
    at clojure.lang.Compiler$InvokeExpr.eval(
    at clojure.lang.Compiler.compile1(
    at clojure.lang.Compiler.compile(
    at clojure.lang.RT.compile(
    at clojure.lang.RT.load(
    at clojure.lang.RT.load(
    at clojure.core$load$fn__5018.invoke(core.clj:5530)
    at clojure.core$load.doInvoke(core.clj:5529)
    at clojure.lang.RestFn.invoke(
    at clojure.core$load_one.invoke(core.clj:5336)
    at clojure.core$compile$fn__5023.invoke(core.clj:5541)
    at clojure.core$compile.invoke(core.clj:5540)
    at user$eval7.invoke(NO_SOURCE_FILE:1)
    at clojure.lang.Compiler.eval(
    at clojure.lang.Compiler.eval(
    at clojure.lang.Compiler.eval(
    at clojure.core$eval.invoke(core.clj:2852)
    at clojure.main$eval_opt.invoke(main.clj:308)
    at clojure.main$initialize.invoke(main.clj:327)
    at clojure.main$null_opt.invoke(main.clj:362)
    at clojure.main$main.doInvoke(main.clj:440)
    at clojure.lang.RestFn.invoke(
    at clojure.lang.Var.invoke(
    at clojure.lang.AFn.applyToHelper(
    at clojure.lang.Var.applyTo(
    at clojure.main.main(
Caused by: java.lang.IllegalArgumentException: No implementation of method: :afn of protocol: #'defrecordissue.aprotocol/AProtocol found for class: defrecordissue.arecord.ARecord
    at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:541)
    at defrecordissue.aprotocol$fn__40$G__35__45.invoke(aprotocol.clj:5)
    at clojure.lang.AFn.applyToHelper(
    at clojure.lang.AFn.applyTo(
    at clojure.lang.Compiler$InvokeExpr.eval(
    ... 25 more 

If I change 3) to construct the record class directly, like so:

(ns defrecordissue.aot2
  (:require [defrecordissue.aprotocol]

(defrecordissue.aprotocol/afn (defrecordissue.arecord.ARecord.))

Compilation succeeds.

My suspicion is that this is somehow related to, but I don't understand exactly what is happening.

I should also add that without the lein clean, compilation succeeds the second time, since a class for the record is now available on the classpath. Therefore, I can get around this problem by AOT-compiling the namespace defining the record type.

I created a simple leiningen project on GitHub that illustrates the issue, see README for usage:

Why am I seeing this behaviour, and what is the correct way to avoid it?


I added a new branch to the GitHub repo better illustrating the core issue:

The problem occurs regardless of where (ie. in which namespace) the record instance is constructed.

by Ragge at July 21, 2014 05:04 PM

Encoding singleton objects as lazy vals


We are working on a static verifier for Scala programs (early work described in this Master's thesis), and the current focus lies on verifying Scala features that involve lazy evaluation. We are mainly interested in the semantics (behaviour) of the features, not in other (nevertheless important) aspects such as comprehensibility or conciseness.

To simplify things, we - for the time being - ignore special roles that singleton objects could have. For example, that some are companion objects (which is probably orthogonal w.r.t. to their lazy nature), or that some are package objects.

Properties of lazy vals and singleton objects

Lazy vals

Assume a lazy val

lazy val v = I

where I is the initialiser block, i.e., the code that determines the value of the lazy val. The initialiser block I is executed when the lazy val v is dereferenced for the first time.

Singleton objects

Assume a singleton object

object Foo {

  val v1 = I1
  var v2 = I2
  lazy val v3 = I3
  def m() {C2}

where C1 is the code making up the constructor of object Foo, where I1 to I3 are again initialiser blocks, and where C2 is the body of method m. When the object Foo is first used (dereferenced, or assigned to a variable/field), then C1, I1 and I2 are executed. I3 is only executed when Foo.v3 is dereferenced (since v3 is a lazy val) and C2 is executed whenever m is called.


Consider this version of Foo, where the singleton object has been encoded by a lazy val and an anonymous class:

// Should probably sit in a package object
lazy val Foo = new {

  val v1 = I1
  var v2 = I2
  lazy val v3 = I3
  def m() {C2}

Can anybody think of a reason for why the encoding of the singleton object Foo as a lazy val would show a different behaviour than the original singleton object? That is, are there (corner) cases where the encoded version would have a different semantics than the original code?

by Malte Schwerhoff at July 21, 2014 05:04 PM


Aktuelles Highlight aus dem Josef-S-Gerichtsverfahren: ...

Aktuelles Highlight aus dem Josef-S-Gerichtsverfahren: Schaut mal, wie der fiese Gewalttäter in den Gerichtssaal geführt wurde. Damit auch wirklich keiner der Schöffen oder Beobachter den Eindruck hat, dass es sich hier nicht um einen hochkriminellen Gewalttäter handelt.

July 21, 2014 05:02 PM

Portland Pattern Repository


References to learn more about graph laplacian.

I have vaguely heard of this connection between random matrix theory and graphs (the spectral gap of their laplacians) on compact Riemann surfaces.

  • Can someone give a pedagogic reference which helps learn this subject?

  • Also is there something called "free probability theory" which is somehow equivalent to random matrix theory?

  • Would you call the usual QFT of Hermitian matrices a random matrix theory?

by user6818 at July 21, 2014 04:59 PM


Ansible synchronize prompts passphrase even if already entered at the beginning

The synchronize module of Ansible (v1.6.5) prompts for the passphrase (Enter passphrase for key) even though I already entered it at the beginning of running the playbook.

Any idea why?

I run my playbook with the following options:

-u myuser --ask-sudo-pass --private-key=/path/to/id_rsa

Here is my synchronize task:

- name: synchronize source files in src location
  sudo: yes
  synchronize: src={{local_src}} dest={{project_dirs.src}} archive=yes delete=yes rsync_opts=["--compress"]
  when: synchronize_src_files

UPDATE with ssh-agent

Following the advice of Lekensteyn, I tried with ssh-agent. I do not have a prompt anymore but the task fails. What am I missing?

eval `ssh-agent -s`
ssh-add ~/.ssh/id_rsa

The error:

TASK: [rolebooks/project | synchronize source files in src location] **********
failed: [] => {"cmd": "rsync --delay-updates -FF --compress --delete-after --archive --rsh 'ssh -i /home/vagrant/.ssh/id_rsa -o StrictHostKeyChecking=no' --rsync-path=\"sudo rsync\" [--compress] --out-format='<<CHANGED>>%i %n%L' /projects/webapp mike@", "failed": true, "rc": 12}
msg: sudo: no tty present and no askpass program specified
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.0]

by YAmikep at July 21, 2014 04:59 PM


difference between multilayer perceptron and linear regression

What is the difference between multilayer perceptron and linear regression classifier. I am trying to learn a model with numerical attributes, and predict a numerical value.


by user20287 at July 21, 2014 04:58 PM

Understanding Property Testing with a toy example

I am newbie with this property testing and I am trying to understand it with a few examples. I first dealt with a toy example.

I did not understand the first step of the test in the following slide. Why is the first step $S= 2/\epsilon$?

enter image description here

Could someone please explain me in more detail (i.e. with a concrete example)? I would appreciate with that.

by Hakan at July 21, 2014 04:58 PM



Scala - define implicitNotFound in implicit conversion

I have a class Foobar[A, B](x: A) and an implicit conversion method to turn Foobar(x) => x if there is evidence that A <:< B. This currently works, so if I call Foobar(x).xyzzy where xyzzy is a method defined on x, as long as evidence exists that A <:< B the conversion works fine. My problem is that if no evidence that A <:< B exists, invoking Foobar(x).xyzzy errors with the message that xyzzy is not a member of Foobar. This is true, and I understand why this happens, but I want the error in this case to instead say Cannot prove A <:< B or something to that effect. I think implicitNotFound should help me do this, but I'm having trouble making it work.


import scala.language.implicitConversions
class Qwerty[A](val _value: A) {
  def toFoobar[B](func: A => B): Foobar[A,B] = new Foobar[A,B](_value, func)

object Qwerty {
  implicit def wrapQwerty[A](anything: A): Qwerty[A] = new Qwerty(anything)
  implicit def unwrapQwerty[A](qwerty: Qwerty[A]): A = qwerty._value
  implicit def unwrapFoobar[A, B >: A](foobar: Foobar[A, B]): B = foobar.func(foobar._value)

class Foobar[A, B](val _value: A, val func: A => B) { }

import Qwerty._

println(5.toFoobar(_ + 2) - 1) // works
println(5.toFoobar(_ + "bad type") - 1) // errors (correctly) but I want error message to say "type mismatch"

by Kvass at July 21, 2014 04:55 PM

puppet facter last run status

Is it possible for facter to show the last status of puppet runs?

typically puppet stores the result in YAML format in


any way to get these values as a facter?

by krisdigitx at July 21, 2014 04:55 PM

Best way to serialize a large amount of data using Scala 2.9

I want to serialize a Graph described as a LinkedHashMap using Scala version 2.9.3. and then deserialize/read it with another program. The amount of nodes and edges is in the range of millions and I'm planning to use Spark to parallelize the execution in the near future*. I've spent a lot of time reading about the different options but I'm still clueless about which one is the best. So, I need some advice about libraries and some links to examples.

*: That's why I use Scala 2.9.3.

by Mr K. at July 21, 2014 04:52 PM




Scala value toInt is not a member of Option[String]

I am new with Scala and I can't figure this out. I am sure it is a simple solution but as a newbie I can't find it.

I have this code

val resID = CartManager.getReservation(request)
   val id = if (resID.isDefined && resID.get.status == "FOUND") else -1

and it gives me an error:

value toInt is not a member of Option[String]

How can I get the id of the object Reservation if it is defined?

Thank you

by konkivo at July 21, 2014 04:34 PM

Scala: how can I generate numbers according to an expected distribution?

Sometimes Scala's random doesn't help when wanting to generate random numbers of smaller ranges or when you already know that certain numbers already have an associated probability attached to them. Obviously, scala.util.Random.nextInt won't get the whole job done.

How do I select numbers according to their weights?

by crockpotveggies at July 21, 2014 04:31 PM


Closed form solution for a single layer linear perceptron

Let f be a one-layer neural network which is linear (ie. no activation function). Let it have $p$ inputs and $q$ outputs. These are fully connected by weights $W$. We have $n$ inputs $x \in \mathbb{R}^d$ and $n$ outputs $y \in \mathbb{R}^{d'}$. We arrange these (column) vectors into matrices $Y$ and $X$, and the weights into a matrix, so that we can express the computation of the network on the whole dataset as $WX = Y$ (that is, if we could find the perfect $W$). Just to clarify, the network contains no hidden layer, just the input and output layers.

Training the network is a matter of choosing $W$ to minimize $e_W = \sum_{i=1}^n||y_i - Wx_i||^2$.

It seems to me that this problem should have a closed form solution that can be solved by a linear solver in some way. Unfortunately, my linear algebra is lacking. Is there a closed form solution? If not, where is the difficulty?

by Peter at July 21, 2014 04:28 PM


Difference of pattern matching for "::(a,b)" and "List(a,b)"

To use :: to pattern match a list:

val ::(a, tail) =  List(1,2,3)

// a: Int = 1
// tail: List[Int] = List(2, 3)

But with List:

val List(a, tail) = List(1,2,3)

// match error

I tried to figure out why the later one can't be matched, but without lucky.

I see in the abstract class SeqFactory, there is a unapplySeq method:

def unapplySeq[A](x: CC[A]): Some[CC[A]] = Some(x)

But when x is List[Int], the return value is still Some[List[Int]], I can't find where the magic is

by Freewind at July 21, 2014 04:27 PM


Portland Pattern Repository

High Scalability

StackOverflow Update: 560M Pageviews a Month, 25 Servers, and It's All About Performance

The folks at Stack Overflow remain incredibly open about what they are doing and why. So it’s time for another update. What has Stack Overflow been up to?

The network of sites that make up StackExchange, which includes StackOverflow, is now ranked 54th for traffic in the world; they have 110 sites and are growing at a rate of 3 or 4 a month; 4 million users; 40 million answers; and 560 million pageviews a month.

This is with just 25 servers. For everything. That’s high availability, load balancing, caching, databases, searching, and utility functions. All with a relative handful of employees. Now that’s quality engineering.

This update is based on The architecture of StackOverflow (video) by Marco Cecconi and What it takes to run Stack Overflow (post) by Nick Craver. In addition, I’ve merged in comments from various sources. No doubt some of the details are out of date as I meant to write this article long ago, but it should still be representative. 

Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there’s no compelling reason to change. Yet SO is pragmatic. They use Linux where it makes sense. There’s no purity push to make everything Linux or keep everything Microsoft. That wouldn’t be efficient. 

Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t  been a problem because they’ve been quite successful at sizing their system correctly.

So it appears Jeff Atwood’s quote: "Hardware is Cheap, Programmers are Expensive", still seems to be living lore at the company.

Marco Ceccon in his talk says when talking about architecture you need to answer this question first: what kind of problem is being solved?

First the easy part. What does StackExchange do? It takes topics, creates communities around them, and creates awesome question and answer sites. 

The second part relates to scale. As we’ll see next StackExchange is growing quite fast and handles a lot of traffic. How does it do that? Let’s take a look and see….


by Todd Hoff at July 21, 2014 04:00 PM


Efficient way to check if a graph node's children were visited during traversal [on hold]

I have a generic traversal algorithm for graphs:

Traversal (container) {
    while (container is not empty ){
        visit container.pop()
        push new nodes

and I want to make a time-efficient "child head count" for nodes:

Traversal (container) {
    while (container is not empty ){
        visit container.pop()
        push new nodes
        close parent nodes if last child

The best way that i could think of was to have pointers to "visited" booleans of the node's children and just "and" them up as such:

node1 -> [node2(true), node5(false), node7(true)]

node1.all_children_visited() = false

Is there a way to do this with lower than O(n) complexity?

by Jean-Luc Nacif Coelho at July 21, 2014 03:55 PM



jtimberman’s Code Blog 2014-07-21 15:50:56

This post will illustrate load_current_resource and a basic use of chef-shell.

The chef-shell is an irb-based REPL (read-eval-print-loop). Everything I do is Ruby code, just like in Chef recipes or other cookbook components. I’m going to use a package resource example, so need privileged access (sudo).

% sudo chef-shell

The chef-shell program loads its configuration, determines what session type, and displays a banner. In this case, we’re taking all the defaults, which means no special configuration, and a standalone session.

loading configuration: none (standalone session)
Session type: standalone

This is the chef-shell.
 Chef Version: 11.14.0.rc.2

run `help' for help, `exit' or ^D to quit.


To evaluate resources as we’d write them in a recipe, we need to switch to recipe mode.

chef > recipe_mode

I can do anything here that I can do in a recipe. I could paste in my own recipes. Here, I’m just going to add a package resource to manage the vim package. Note that this works like the “compile” phase of a chef-client run. The resource will be added to the Chef::ResourceCollection object. We’ll look at this in a little more detail shortly.

chef:recipe > package "vim"
 => <package[vim] @name: "vim" @noop: nil @before: nil @params: {} @provider: nil @allowed_actions: [:nothing, :install, :upgrade, :remove, :purge, :reconfig] @action: :install @updated: false @updated_by_last_action: false @supports: {} @ignore_failure: false @retries: 0 @retry_delay: 2 @source_line: "(irb#1):1:in `irb_binding'" @guard_interpreter: :default @elapsed_time: 0 @sensitive: false @candidate_version: nil @options: nil @package_name: "vim" @resource_name: :package @response_file: nil @response_file_variables: {} @source: nil @version: nil @timeout: 900 @cookbook_name: nil @recipe_name: nil>

I’m done adding resources/writing code to test, so I’ll initiate a Chef run with the run_chef method (this is a special method in chef-shell).

chef:recipe > run_chef
[2014-07-21T09:04:51-06:00] INFO: Processing package[vim] action install ((irb#1) line 1)
[2014-07-21T09:04:51-06:00] DEBUG: Chef::Version::Comparable does not know how to parse the platform version: jessie/sid
[2014-07-21T09:04:51-06:00] DEBUG: Chef::Version::Comparable does not know how to parse the platform version: jessie/sid
[2014-07-21T09:04:51-06:00] DEBUG: package[vim] checking package status for vim
  Installed: 2:7.4.335-1
  Candidate: 2:7.4.335-1
  Version table:
 *** 2:7.4.335-1 0
        500 testing/main amd64 Packages
        100 /var/lib/dpkg/status
[2014-07-21T09:04:51-06:00] DEBUG: package[vim] current version is 2:7.4.335-1
[2014-07-21T09:04:51-06:00] DEBUG: package[vim] candidate version is 2:7.4.335-1
[2014-07-21T09:04:51-06:00] DEBUG: package[vim] is already installed - nothing to do

Let’s take a look at what’s happening. Note that we have INFO and DEBUG output. By default, chef-shell runs with Chef::Log#level set to :debug. In a normal Chef Client run with :info output, we see the first line, but not the others. I’ll show each line, and then explain what Chef did.

[2014-07-21T09:04:51-06:00] INFO: Processing package[vim] action install ((irb#1) line 1)

There is a timestamp, the resource, package[vim], the action install Chef will take, and the location in the recipe where this was encountered. I didn’t specify one in the resource, that’s the default action for package resources. The irb#1 line 1 just means that it was the first line of the irb in recipe mode.

[2014-07-21T09:04:51-06:00] DEBUG: Chef::Version::Comparable does not know how to parse the platform version: jessie/sid
[2014-07-21T09:04:51-06:00] DEBUG: Chef::Version::Comparable does not know how to parse the platform version: jessie/sid

Chef chooses the default provider for each resource based on a mapping of platforms and their versions. It uses an internal class, Chef::Version::Comparable to do this. The system I’m using is a Debian “testing” system, which has the codename jessie, but it isn’t a specific release number. Chef knows that for all debian platforms to use the apt package provider, and that’ll do here.

[2014-07-21T09:04:51-06:00] DEBUG: package[vim] checking package status for vim
  Installed: 2:7.4.335-1
  Candidate: 2:7.4.335-1
  Version table:
 *** 2:7.4.335-1 0
        500 testing/main amd64 Packages
        100 /var/lib/dpkg/status
[2014-07-21T09:04:51-06:00] DEBUG: package[vim] current version is 2:7.4.335-1
[2014-07-21T09:04:51-06:00] DEBUG: package[vim] candidate version is 2:7.4.335-1

This output is the load_current_resource method implemented in the apt package provider.

The check_package_state method does all the heavy lifting. It runs apt-cache policy and parses the output looking for the version number. If we used the :update action, and the installed version wasn’t the same as the candidate version, Chef would install the candidate version.

Chef resources are convergent. They only get updated if they need to be. In this case, the vim package is installed already (our implicitly specified action), so we see the following line:

[2014-07-21T09:04:51-06:00] DEBUG: package[vim] is already installed - nothing to do

Nothing to do, Chef finishes its run.

Modifying Existing Resources

We can manipulate the state of the resources in the resource collection. This isn’t common in most recipes. It’s required for certain kinds of development patterns like “wrapper” cookbooks. As an example, I’m going to modify the resource object so I don’t have to log into the system again and run apt-get remove vim, to show the next section.

First, I’m going to create a local variable in the context of the recipe. This is just like any other variable in Ruby. For its value, I’m going to use the #resources() method to look up a resource in the resource collection.

chef:recipe > local_package_variable = resources("package[vim]")
 => <package[vim] @name: "vim" @noop: nil @before: nil @params: {} @provider: nil @allowed_actions: [:nothing, :install, :upgrade, :remove, :purge, :reconfig] @action: :install @updated: false @updated_by_last_action: false @supports: {} @ignore_failure: false @retries: 0 @retry_delay: 2 @source_line: "(irb#1):1:in `irb_binding'" @guard_interpreter: :default @elapsed_time: 0.029617095 @sensitive: false @candidate_version: nil @options: nil @package_name: "vim" @resource_name: :package @response_file: nil @response_file_variables: {} @source: nil @version: nil @timeout: 900 @cookbook_name: nil @recipe_name: nil>

The return value is the package resource object:

chef:recipe > local_package_variable.class
 => Chef::Resource::Package

(#class is a method on the Ruby Object class that returns the class of the object)

To remove the vim package, I use the #run_action method (available to all Chef::Resource subclasses), specifying the :remove action as a symbol:

chef:recipe > local_package_variable.run_action(:remove)
[2014-07-21T09:11:50-06:00] INFO: Processing package[vim] action remove ((irb#1) line 1)
[2014-07-21T09:11:52-06:00] INFO: package[vim] removed

There is no additional debug to display. Chef will run apt-get remove vim to converge the resource with this action.

Load Current Resource Redux

Now that the package has been removed from the system, what happens if we run Chef again? Well, Chef is convergent, and it takes idempotent actions on the system to ensure that the managed resources are in the desired state. That means it will install the vim package.

chef:recipe > run_chef
[2014-07-21T09:11:57-06:00] INFO: Processing package[vim] action install ((irb#1) line 1)

We’ll see some familiar messages here about the version, then:

[2014-07-21T09:11:57-06:00] DEBUG: package[vim] checking package status for vim
  Installed: (none)
  Candidate: 2:7.4.335-1
  Version table:
     2:7.4.335-1 0
        500 testing/main amd64 Packages
[2014-07-21T09:11:57-06:00] DEBUG: package[vim] current version is nil
[2014-07-21T09:11:57-06:00] DEBUG: package[vim] candidate version is 2:7.4.335-1

This is load_current_resource working as expected. As we can see from the apt-cache policy output, the package is not installed, and as the action to take is :install, Chef will do what we think:

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  g++-4.8 geoclue geoclue-hostip geoclue-localnet geoclue-manual
  geoclue-nominatim gstreamer0.10-plugins-ugly libass4 libblas3gf libcolord1
  libcolorhug1 libgeoclue0 libgnustep-base1.22 libgnutls28 libminiupnpc8
  libpoppler44 libqmi-glib0 libstdc++-4.8-dev python3-ply xulrunner-29
Use 'apt-get autoremove' to remove them.
Suggested packages:
  vim-doc vim-scripts
The following NEW packages will be installed:
0 upgraded, 1 newly installed, 0 to remove and 28 not upgraded.
Need to get 0 B/905 kB of archives.
After this operation, 2,088 kB of additional disk space will be used.
Selecting previously unselected package vim.
(Reading database ... 220338 files and directories currently installed.)
Preparing to unpack .../vim_2%3a7.4.335-1_amd64.deb ...
Unpacking vim (2:7.4.335-1) ...
Setting up vim (2:7.4.335-1) ...
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vim (vim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vimdiff (vimdiff) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rvim (rvim) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rview (rview) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vi (vi) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/view (view) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/ex (ex) in auto mode

This should be familiar to anyone that uses Debian/Ubuntu, it’s standard apt-get install output. Of course, this is a development system so I have some cruft, but we’ll ignore that ;).

If we run_chef again, we get the output we saw in the original example in this post:

[2014-07-21T09:50:06-06:00] DEBUG: package[vim] is already installed - nothing to do

by Joshua Timberman at July 21, 2014 03:50 PM

Planet Clojure

Ninety-Nine Haskell Problems [Euler/Clojure too]

Ninety-Nine Haskell Problems

From the webpage:

These are Haskell translations of Ninety-Nine Lisp Problems, which are themselves translations of Ninety-Nine Prolog Problems.

Also listed are:

Naming isn’t the only hard problem in computer science. The webpage points out that due to gaps and use of letters, there are 88 problems and not 99.

If you want something a bit more challenging, consider the Project Euler problems. No peeking but there is a wiki with some Clojure answers,


I first saw this in a tweet by Computer Science.

by Patrick Durusau at July 21, 2014 03:50 PM



Grails 1.3.9 and grails-melody: javamelody.jar not packaged when building war with OpenJDK 1.6

We are maintaining a webapp developed in Grails 1.3.9.

For monitoring performance, the app had grails-melody 1.21 plugin installed.

It seems that the plugin is not available in the repositories any more for grails 1.3.x. I downloaded it from google-code as suggested in the documentation.

Another post in stackoverflow suggests that zipped plugins can be put in lib and then referenced from BuildConfig.groovy.

plugins {
    runtime ":hibernate:1.3.9"
    build ":tomcat:1.3.9"
    compile ":dojo:"

    // Downloaded from
    // Installed from lib

    compile ":grails-melody:1.21"

I did that and this procedure worked fine when building war file with Oracle JDK 7 (on Ubuntu 14.04). I had to rename to so that it was found.

$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

During the build, melody's dependencies were loaded at the beginning of the build process, when grails file were copies as well:

Downloading: /home/matejk/devel/grails/grails-1.3.9/lib/servlet-api-2.5.jar ...
Download complete.
Downloading: /home/matejk/devel/grails/grails-1.3.9/lib/jsp-api-2.1.jar ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: /home/matejk/devel/grails/grails-1.3.9/lib/groovy-all-1.7.8.jar ...
Download complete.
Downloading: /home/matejk/devel/grails/grails-1.3.9/lib/commons-beanutils-1.8.0.jar ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: /home/matejk/devel/grails/grails-1.3.9/lib/aspectjweaver-1.6.8.jar ...

Resulting war file had javamelody, jrobin and itext jars in WEB-INF/lib.

However, the requirement is to build the app with JDK 1.6 on another machine (Jenkins) where clean checkout of sources is done for every build.

java -version
java version "1.6.0_31"
OpenJDK Runtime Environment (IcedTea6 1.13.3) (6b31-1.13.3-1ubuntu1)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

Download of jars happened later in the build process:

Executing hibernate-1.3.9 plugin post-install script ...
Plugin hibernate-1.3.9 installed
Installing zip /var/lib/jenkins/.ivy2/cache/org.grails.plugins/grails-melody/zips/ ...
    [mkdir] Created dir: /var/lib/jenkins/workspace/etermin-2.4/target/projects/etermin-2.4/plugins/grails-melody-1.21
    [unzip] Expanding: /var/lib/jenkins/.ivy2/cache/org.grails.plugins/grails-melody/zips/ into /var/lib/jenkins/workspace/etermin-2.4/target/projects/etermin-2.4/plugins/grails-melody-1.21
Installed plugin grails-melody-1.21 to location /var/lib/jenkins/workspace/etermin-2.4/target/projects/etermin-2.4/plugins/grails-melody-1.21. ...
Resolving plugin JAR dependencies ...
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.
Downloading: ...
Download complete.

However, when creating war with that JDK, javamelody, jrobin and itext jars are not packaged.

Consequently deployment and startup of webapp fails.

SEVERE: Error configuring application listener of class net.bull.javamelody.SessionListener
java.lang.ClassNotFoundException: net.bull.javamelody.SessionListener
        at org.apache.catalina.loader.WebappClassLoader.loadClass(
        at org.apache.catalina.loader.WebappClassLoader.loadClass(
        at org.apache.catalina.core.StandardContext.listenerStart(
  • Is the procedure to install plugin from local directory correct?
  • Any ideas how to resolve this problem to make jars to be packaged into war?



EDIT: After removing grails files in ~/.grails/1.3.9/ the behaviour with JDK 6 and JDK 7 is the same:

jar files are not packaged and deployment fails.

by matejk at July 21, 2014 03:47 PM


Min-max selection sort

Is there already modified version of selection sort that works like this pseudocode:

a // the array, index starts at 1 (not zero-based)
n = len(a)
for x in 1 to n
  big = x 
  small = x
  last = n-x+1
  for y in x+1 to last
    if a[big] < a[y] then big = y
    if a[small] > a[y] then small = y
  if big <> small 
    if small <> x then swap(a[small],a[x])
    if big <> last then swap(a[big],a[last])

if there is such an algorithm, what is the name of it?

by Kokizzu at July 21, 2014 03:46 PM


Matt Might

Understand and implement laziness with examples in Scala, JavaScript, Swift and Racket

At first glance, laziness seems to be about efficiency.

In computing, laziness means delaying the computation of a value until the value is necessary.

(And, if the value is never necessary, it is never computed.)

In reality, laziness is more about expressiveness: laziness allows the expression of programs that would otherwise not terminate.

In some languages (like Scala, Swift and Perl), laziness also makes it possible to add new constructs to a language without using macros.

Since most languages do not provide support for laziness, this article explores how to implement laziness in languages that support closures.

We’ll look at manual transformations from languages that have native support for laziness (Scala) into (1) languages that do not support laziness (JavaScript) and (2) languages that have partial support (Swift).

We’ll also use macro-generating macros in Racket to implement native and transparent support for multiple kinds of laziness in a strict language.

Click here to read the rest of the article

July 21, 2014 03:33 PM




Build scatter-gather in akka

I want to build actor chain with scatter-gather pattern

My flow looks like this:


When I creating actors I use following code:

val a5: ActorRef = actorSystem.actorOf(Props(new A5))
val a4: ActorRef = actorSystem.actorOf(Props(new A4(a5, otherArgs4))
val a3: ActorRef = actorSystem.actorOf(Props(new A3(a4, otherArgs3)))
val a2: ActorRef = actorSystem.actorOf(Props(new A2(a3)))
val a1: ActorRef = actorSystem.actorOf(Props(new A1(a2)))

Can I create actors chain in smarter and more configurable way?

by KrzyH at July 21, 2014 03:26 PM

Access @ApiOperation attributes in Swagger ApiAuthorizationFilter

I have a Jax-RS webservice, which I document using Swagger. Now I want to check the attributes from the @ApiOperation annotation of each method to do some filtering.

@ApiOperation(value = "a description", response = String.class, hidden=true)
public static Response get(){}

I created an ApiAuthorizationFilterImpl according to this example:

I overwrote the isOperationAllowed method:

public boolean isOperationAllowed(Operation operation, ApiDescription api, Map<String, List<String>> params, Map<String, String> cookies, Map<String, List<String>> headers) {

       System.out.println("params: "+params.keySet()); //<--params is empty

Now, when I am trying to access params, it is empty. On the opposite, I can print the parameters from operation.parameters() but I also can't access them, as I get a Scala list with parameters.

The filter in general is access when starting the Swagger-UI, so it seems to be correctly passed in my web.xml.

Is this a silly mistake from my side, or how will I be able to access the parameters?

As a workaround I also wouldn't mind to access parameters from @ApiModelProperty annotation.

by tareq at July 21, 2014 03:22 PM

In sbt can't add or override maven URL resolvers with params

I'm trying to override the default external-resolvers with maven resolvers of type http:/.....?key=value.

I tried to add to build.sbt the following with no luck of solving my problem:

val repo1="Repo1" at "http://.....?key=value"

val repo2=Resolver.url("Repo2", url("http://....."))( Patterns(Seq("[organisation]/[module](_[scalaVersion])(_[sbtVersion])/[revision]/[module](_[scalaVersion])(_[sbtVersion])-[revision](-[classifier]).pom?key=value"), Seq("[organisation]/[module](_[scalaVersion])(_[sbtVersion])/[revision]/[artifact]-[revision](-[classifier]).[ext]?key=value"), true) )

externalResolvers:=Seq(repo1, repo2)

repo1, of course, doesn't work because the the params are embedded inside the URL and not at the end after the question mark (e.g., http:/.....?key=value/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.pom).

While repo2 seems to have a problem analyzing the the returned pom file: problem while downloading module descriptor: https:/...../org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.pom?key=value: invalid md5: expected=<?xml computed=8cde60b7d723d15aeb5e59aaf033351f (821ms)

It seems to me that sbt is considering repo2 as a repo of type ivy and not as a maven one.

I tried the above with sbt 0.13.0 and 0.13.5.

Any idea how to solve this issue and support url with params?

by COMPUTERPUNC at July 21, 2014 03:19 PM

Create a service which returns a Future [on hold]

I have been dealing with Play! Framework recently. I noticed some API support asynchronous services. For example MongoDB has ReactiveMongo which supports asynchronous API. I am interested in creating my own asynchronous services, however could not find any reference online. How would you implement asynchronous services? Or is there a good materials I can study on?

Edit: My question is, what does it take a service to be non-blocking or asynchronous? I want to create a service which will not block the caller, but allow caller to wait on the future.

by Mayumi at July 21, 2014 03:18 PM



Portland Pattern Repository

Planet Scala

Scala for Java Developers

A few weeks ago I was asked to review a book called "Scala for Java Developers". Being preparing some Scala lessons to share at work it made lots of sense to agree. Finding a good way to introduce Java and .NET developers to Scala is hard. If someone is interested in learning a new language it's easy, just give them the Programming in Scala book. The challenge I'm up against is making a case for actually spending time learning it at all. What caught my eye a few years ago was all the expressiveness and succinctness I could achieve by making the leap. My expectations were to find a book that explained how to translate all the Java knowledge into Scala while learning the interesting new features one could discover making such a transition. The books was not exactly what I expected but that is not a bad thing.

The first chapter introduces the reader to the Activator and drops them into the REPL. It shows good examples of how easy it is to declare classes, play around with collections and express basic stuff. That is what I expected to find but then it picks up speed. It continues on the next chapter by creating a Java web service and exemplifying how to migrate it to Scala piece by piece. Then it goes on to IDEs, SBT, mentions many useful SBT plugins, worksheets and even a writing a REST client. Followed by a tour of all the testing frameworks available on Scala land. Moving to Play Framework and then some debugging.

Halfway through the book most of what regular developer faces on a day to day basis has been touched upon. Scala syntax is explained almost by accident, just as a necessity to show what can be done. It doesn't teach the language but rather provides examples of interesting and most likely new and simpler ways to do stuff a Java developer is used to. The rest of the book continues on at full speed with databases, webservices, xml, json, CRUDs and concurrency. I was surprised it even tackles Iteratees to illustrate reactive web applications like chat rooms. It even mentions Scala.js!

Overall it shows many libraries and tools used in the Scala ecosystem. It takes on many of the common problems most people are trying to solve at work. I think it can appeal to Spring, JavaEE or Rails users and similar. It provides a bird's eye view of the frameworks available while forcing the reader to actually try stuff out immediately in the REPL. Readers will get a feeling for what Scala enables and will surely go out looking for more.

My conclusion is if you are already into Scala and learning on your own or with other books then it is not for you. I recommend the book as a teaser for programmers who don't know Scala at all or barely and want to discover what it is about skipping to the fun part. Also, for someone like me who is trying to get other people interested it provides many ideas and examples that are simple to setup and play with. Having more Scala resources is good and this book fills a spot that needed to be filled.

by Tomás Lázaro ( at July 21, 2014 02:59 PM


Automatic decryption of GELI disks with encrypted keys and passphrases stored on a remote system.

I always liked the idea of more of a deadmans switch style of encryption security, so I created an rc script that automatically retrieves geli keys and passphrases from a remote system via ssh. You can put the key and the passphrase on different systems for even more security, and the key file and passphrase file can be encrypted themselves so they can be stored on an untrusted server.

You could use this to encrypt a disk on a VPS for example and have it retrieve the keys from your local server. So long as your local server is up your VPS can reboot unattended. Or the other way around, you could store the encryption keys on a VPS or even a shell host as long as you have scp access. Even busybox serves scp, so you could store a key or passphrase file on an embedded router or an android phone if you want.

Hopefully others find this useful, I think it strikes a nice balance of security and convenience.

submitted by FakingItEveryDay
[link] [4 comments]

July 21, 2014 02:56 PM



How is PnL calculated

In Fixed Income, I know that bonds PnL are evaluated depending on where the price lies on price/yield curve at the end of the day, compared to where it started from at beginning of the day. The portfolio of bonds will have a specific DV01, which will be used to compute the PnL.

Can someone tell me if this is right or is there something more? For equities it should be just a simple sum of stock prices at the end of day vs beginning of day? Is this right?

by Papal at July 21, 2014 02:45 PM



Default implementation for generic method

Suppose I have a trait Foo

Also I have a trait that uses trait Foo in type parameter

trait Bar[T <: Foo] {
 def bar: T = new Foo {} //doesn't compile

I want to provide default implementaition in trait Bar for bar method. I need this because there are several implementations that do not need to override bar method(they are just happy with Foo), but also there are some implemenations that do need to provide specific implemetaions of Foo in bar method.

Is this possible?

by maks at July 21, 2014 02:43 PM


Overcoming Bias

Tegmark’s Vast Math

I recently had a surprise chance to meet Max Tegmark, and so I first quickly read his enjoyable new book The Mathematical Universe. It covers many foundations of physics topics that he correctly says are unfairly neglected. Since I’ve collected many opinions on foundation of physics over decades, I can’t resist mentioning the many ways I agree and disagree with him.

Let me start with what Tegmark presents as his main point, which is that the total universe is BIG, almost as big as it could possibly be. There’s a vast universe out there that we can’t see, and will never see. That is, not only does space extent far beyond our cosmological horizon, but out there are places where physics sits in very different equilibria of fundamental physics (e.g., has a different number of useful dimensions), and nearby are the different “many worlds” of quantum mechanics.

Furthermore, and this is Tegmark’s most unique point, there are whole different places “out there” completely causally (and spatially) disconnected from our universe, which follow completely different fundamental physics. In fact, all such mathematically describable places really exist, in the sense that any self-aware creatures there actually feel. Tegmark seems to stop short, however, of David Lewis, who said that all self-consistent possible worlds really exist.

Tegmark’s strongest argument for his distinctive claim, I think, is that we might find that the basic math of our physics is rare in allowing for intelligent life. In that case, the fact of our existence should make us suspect that many places with physics based on other maths are out there somewhere:

Why do the five arrows in the bottom panel conspire to allow any habitable range of Higgs properties? This could well be a fluke: five random arrows would allow some range with 19% probability, so we need only invoke a small amount of luck. … However, it’s perfectly plausible that further physics research could uncover more striking fine-tuning of this discrete type with, say, ten or more arrows conspiring to allow a habitable range for some physical parameter or parameters. And if this happens, then we can argue … that this is evidence for the existence … of other universes where the laws of physics are different, giving quite different requirements for life!

I accept that this consideration would modestly push us in this direction, just as the existence of physics parameters fine-tuned for life modestly pushes us to believe that other spatial regions out there have different values of those parameters. I accept these because I accept the self-indication principle in indexical inference, which says that the info that you exist should increase your belief in possible worlds where you might have existed, in proportion to the number of slots where you could have existed.

However, it isn’t clear that Tegmark accepts the self-indication principle, since he also says that you should believe in many worlds quantum mechanics if you survive many rounds of quantum suicide, which is where you randomly suicide depending on a quantum event. The self-indication principle does not say that seeing this should increase your belief in many worlds, as there aren’t substantially more slots for you to occupy in scenarios where quantum suicide attempts fail. Since both many worlds and stochastic quantum mechanics predict exactly the same conditional probabilities of observations given quantum suicide attempts, such data can’t distinguish them. So I wouldn’t see surviving quantum suicide as further evidence for many worlds.

Tegmark also argues that we should reject a description of reality that has math plus extra “baggage.” Apparently, if there are many possible math descriptions of universes, then it would be extra “baggage” for the universe to point to only one math and say “that’s the math that really exists.” This is pretty close to the argument of David Lewis. But Tegmark also seems to accept that there is some “measure” over all these maths, and that the reason to take some actions over others is to favorably change that measure. Yet this measure also seems to also be an extra “baggage” over and beyond the math descriptions themselves. The main difference I see between adding a “this exists” pointer and adding a measure over the space of maths is that the later approach has more slots for me to exist, and so is favored by self-indication.

Tegmark does say that we struggle to find reasonable measures for many physics problems, and as a result we don’t actually know what inflation implies. He suspects this is due to our over-eager embrace of the concept of infinity. But Tegmark also follows Everett in using infinity to derive the Born probability rule in quantum mechanics. Everett noted that the relative measure of Born-rule-deviant worlds approached zero after an infinity of measurements, and thus the final state is “as if” those worlds didn’t exist. To get the same effect, Tegmark instead considers a superposition of an infinity of spatial regions out there containing the same exact observer. But this trick wouldn’t work with a finite set of such regions, and I have my doubts that it is fair to treat such regions is if they hadn’t decohered. (My approach to deriving the Born rule in many worlds stays with strictly finite everything.)

Tegmark also seems to think that a similar approach explains the second law of thermodynamics, but that doesn’t at all make sense to me. The main thing to explain there is why our best measure for predicting the past is pretty much the inverse of our best measure for predicting the future. I didn’t see Tegmark addressing that at all. (Btw, Sean Carroll recently mostly rejected the solution he wrote a book elaborating.)

My last disagreement with Tegmark is that he sees the simulation argument as self-destructing because since we know nothing about the universe that might simulate us, it could also be simulated by another universe, which is a “a reductio ad absurdum.” But I think it quite reasonable to put substantial weight there being real costs of computation in the base universe, which means they would tend to limit the ability of universes they simulate to spawn yet further universes. Which makes it less likely that we are a second level simulation, relative to a first level one.

Let me conclude with some of the many ways I agree with the book:

  • We reject or accept total theories, not places. If parallel universes are implied by our best theories, we should believe in them.
  • It is vitally important to figure out good physics measures, and good physics explanations for them.
  • I also prefer to reason about theories without infinities, when nearly as plausible on other grounds. Take a limit to infinity as the very last theory step, if possible.
  • Our brains aren’t doing much quantum computing.
  • Quantum mechanics seems random because we observers are splitting; the universe is deterministic.
  • Foundations of physics gets unfairly neglected. People who do it have to hide it to save their careers.
  • The world is hurting itself by not coordinating to reduce existential risk. We have a vast great potential future if we don’t kill ourselves.
  • We don’t need a new understanding of consciousness to solve foundations of physics problems. “Consciousness is the way information feels when being processed in certain complex ways.” Saying that our math “exists” is pretty much the same as saying that the stuff that fits our math is the stuff that “feels” when arranged in certain ways.

by Robin Hanson at July 21, 2014 02:15 PM


How to run Scala files on a web server

This could be either an incredibly easy or horrifically complicated question; but I've started writing code in Scala, and want to run said code on a web server.

There's a few questions that I need answering really, and I apologise for my complete lack of knowledge on the subject, web servers scare me (I'm a PHP developer so all I need to do there is upload to any linux apache server):

  • How to save and upload files (I know how to use ftp software, do I need to do anything from Eclipse?)
  • What to do to my server to run the files (at the moment I just have a linux apache server from fasthosts, is that enough?)

Any other advice would be so greatly appreciated; and the less jargon the better - the whole thing blags me a little.

UPDATE: To put the sort of work I'm doing into context, I'm making a rather large, social-media style site that'll have a very large amount of small, regular posts/updates. Also thanks for the answers so far!

Seems like i need to do all of the following:

  • Learn Scala (I can write in PHP and Java so should be okay)
  • Install Apache tomcat on my server somehow
  • Install Lift (and then either Maven or the SBT)
  • Upload the whole thing?

What do I do about databases?! I need something really serious, with MySQL be okay? Urgh this goes on and on...

by Alex at July 21, 2014 02:14 PM



Risk Parity portfolio construction

If I would like to construct a fully invested long only portfolio with two asset classes (Bonds $B$ and Stocks $S$) based on the concept of 'risk parity' the weights $W$ of my portfolio would be the following:

Then the weight of the bonds $W_B = \textrm{Vol}(S)/[\textrm{Vol(S)}+\textrm{Vol(B)}]$ and the weights of the stocks $W_S = 1 - W_B$.

Based on this I am going to overweigh the low volatility asset and underweight the high volatility asset. My question is: How to calculate the weights for a portfolio with multiple asset classes, 5 for example, so that each asset class will have the same volatility and contribute the same amount of risk into my portfolio. From historical data I can extract the volatility of each asset class and the correlation between them.

by Karamos at July 21, 2014 02:05 PM


Die Russen sagen, ein ukrainischer Su-25 sei zur Abschusszeit ...

Die Russen sagen, ein ukrainischer Su-25 sei zur Abschusszeit nahe an MH17 rangeflogen. Leider ist Su-25 kein Abfangjäger sondern ein Erdkampfflugzeug, das soll Bodentruppen unterstützen. Der kommt bis 7km hoch, die MH17 flog 10.

Aber abgesehen davon, voll die solide Theorie!1!!

Update: Die Pressekonferenz soll wohl ansonsten ganz lustig gewesen sein; die Russen haben gemeint, hey, die Amis haben ja umfangreiche Telemetriedaten von dieser Liste an Satelliten hier, die das hätten sehen müssen. Die sollen doch einfach mal die Daten auf den Tisch legen!1!!

Update: Ah, da steht jetzt mehr. Ja, man kann in eine Su-25 auch Luft-Luft-Raketen tun. RT schreibt, Su-25 käme bis 10 km hoch. Abgesehen davon hat die ukrainische Luftwaffe auch Abfangjäger wie den Su-27, die können dann bis 19km rauf.

Oh, und die Russen legen gleich mal Aufklärungsfotos der Buk-Stellungen der Ukrainer bei und meinen, die MH17 sei in deren Range gewesen.

July 21, 2014 02:02 PM


Implementing Receiver in SPARK

I have been trying to implement receiver for SPARK 0.9 . I have captured packets using Jnetpcap library and need to pass it to spark in Scala. Is it enough to write the capturing part of the packet in "def receive()" method?

Edit: Here is code from this link that captures packets using the Jnetpcap library:

import java.util.Date
import java.lang.StringBuilder
import org.jnetpcap.Pcap
import org.jnetpcap.packet.PcapPacket
import org.jnetpcap.packet.PcapPacketHandler

object PacketCapture1 {
  def main(args: Array[String]){
    val snaplen = 64 * 1024 // Capture all packets, no trucation
    val flags = Pcap.MODE_PROMISCUOUS // capture all packets
    val timeout = 10 * 1000
    //val errbuf = new StringBuilder()

    val jsb = new java.lang.StringBuilder()
    val errbuf = new StringBuilder(jsb);
    val pcap = Pcap.openLive("eth0", snaplen, flags, timeout, errbuf)
    if (pcap == null) {
      println("Error : " + errbuf.toString())
    val jpacketHandler = new PcapPacketHandler[String]() {

      def nextPacket(packet: PcapPacket, user: String) {
        println("Received packet at %s caplen=%4d len=%4d %s\n", new Date(packet.getCaptureHeader.timestampInMillis()),
          packet.getCaptureHeader.caplen(), packet.getCaptureHeader.wirelen(), user)
    pcap.loop(30, jpacketHandler, "jNetPcap works!")


How to implement spark receiver for the packets captured using this code?

by user3823859 at July 21, 2014 01:52 PM



Turing Machine that Determines whether an equal amount of 1's and 0's have been scanned [on hold]

Design a program for a Turing machine.

Create a decision procedure for the set of binary strings which contain the same number of 0's and 1's.

Please help! Thanks.

I am a beginner student and have no idea where to start. So sorry for how elementary this is.

by David Kent at July 21, 2014 01:46 PM

Lambda the Ultimate Forum

Depth-first search in APL

The latest video in Dyalog's library - Depth-First Tree-Search in APL - is now available (

The classic depth-first search algorithm is explored using APL, a simple and concise array notation with high-order functions. The presentation highlights APL's incremental development style, using simple steps and culminating in a purely functional solution for the N-Queens problem. The informative style of this presentation brings clarity to this advanced topic, making it accessible even to those who are still near the start of their APL journey.

Once you've seen the video, why not examine the code used in greater detail and try the expressions for yourself in the online tutorial at (Learn tab > Depth-first search)

July 21, 2014 01:42 PM


"failure in processEncodedAnnotation" using Jackson with ScalaModule on Android

I use Scala 2.11.0 with sbt and android-sdk-plugin together with Jackson as the Json Marshaller. When the app tries to deserialize from Json to my Scala case class the following exception occurs on the device:

W/dalvikvm﹕ WARNING: could not find annotation member bytes in Lscala/reflect/ScalaSignature;                                                                               
E/dalvikvm﹕ Failed creating annotation element (m=0x41a575c0 n=0x41a575e0 a=0x0 r=0x0)                                                                                      
W/System.err﹕ java.lang.RuntimeException: failure in processEncodedAnnotation                                                                                               
W/System.err﹕ java.lang.RuntimeException: failure in processEncodedAnnotation                                                                                               
W/System.err﹕ at java.lang.Class.getDeclaredAnnotations(Native Method)                                                                                                      
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.AnnotatedClass.resolveClassAnnotations(                                                  
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.AnnotatedClass.getAnnotation(                                                            
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.findAutoDetectVisibility(                   
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.<init>(                                                 
W/System.err﹕ at com.fasterxml.jackson.module.scala.introspect.ScalaPropertiesCollector.<init>(ScalaPropertiesCollector.scala:15)                                           
W/System.err﹕ at com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospector$.constructPropertyCollector(ScalaClassIntrospector.scala:24)                          
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.collectProperties(                                        
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.forDeserialization(                                        
W/System.err﹕ at com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.forDeserialization(                                        
W/System.err﹕ at com.fasterxml.jackson.databind.DeserializationConfig.introspect(                                                            
W/System.err﹕ at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(                                                     
W/System.err﹕ at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(                                                        
W/System.err﹕ at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(                                        
W/System.err﹕ at com.fasterxml.jackson.databind.deser.DeserializerCache.hasValueDeserializerFor(                                                 
W/System.err﹕ at com.fasterxml.jackson.databind.DeserializationContext.hasValueDeserializerFor(                                             
W/System.err﹕ at com.fasterxml.jackson.databind.ObjectMapper.canDeserialize(                                                                         
W/System.err﹕ at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.canRead(                          
W/System.err﹕ at org.springframework.web.client.RestTemplate$AcceptHeaderRequestCallback.doWithRequest(                                               
W/System.err﹕ at org.springframework.web.client.RestTemplate$HttpEntityRequestCallback.doWithRequest(                                                 
W/System.err﹕ at org.springframework.web.client.RestTemplate.doExecute(                                                                               
W/System.err﹕ at org.springframework.web.client.RestTemplate.execute(                                                                                 
W/System.err﹕ at                                                                                
W/System.err﹕ at$$anonfun$exchange$1.apply(ScalaRestTemplate.scala:22)                                                         
W/System.err﹕ at$$anonfun$exchange$1.apply(ScalaRestTemplate.scala:22)                                                         
W/System.err﹕ at scala.util.Try$.apply(Try.scala:161)                                                                                                                       
W/System.err﹕ at$$anonfun$genericRestCallForEntity$1.apply(ScalaRestTemplate.scala:31)                                         
W/System.err﹕ at$$anonfun$genericRestCallForEntity$1.apply(ScalaRestTemplate.scala:31)                                         
W/System.err﹕ at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)                                                                      
W/System.err﹕ at scala.concurrent.impl.Future$                                                                                
W/System.err﹕ at java.util.concurrent.ThreadPoolExecutor.runWorker(                                                                            
W/System.err﹕ at java.util.concurrent.ThreadPoolExecutor$                                                                            
W/System.err﹕ at                                                                                                                      

by longliveenduro at July 21, 2014 01:36 PM

[Exception: DB plugin is not registered.]

package models
import play.api._
import play.api.mvc._
import anorm._
import anorm.SqlParser._
import play.api.db.DB
import play.api.Play.current

import scala.language.postfixOps
case class User(name:String)

object User  {
def insert(name:String) : Boolean={
DB.withConnection { implicit connection =>
val result:Boolean= SQL("insert into user(name,username,password)    VALUES('"+name+"','"+name+"','"+name+"')").execute()
return result


"In above program an exception is occured named "[Exception: DB plugin is not registered.]". how to fix it"

by rahul at July 21, 2014 01:36 PM

Twilio Proxy: Howto create HTTPS Rest Spray Server

I found the shadow proxy twilio:
With this proxy I can forward an incoming request on the one hand to a true server and on the ohter hand to a shadow server.

The Twilio proxy is using Spray for REST to accomplish this task.

If everything is sent via HTTP, I could test it successfully.

But I like to send all the data via TLS.
My question is: How to create a HTTPS-Server with Spray?

The full code you can find here:
Here's the code excerpt from ProxyService.scala how the Twilio proxy creates the HTTP Proxy:

def buildNewRequest(request: HttpRequest, config: ShadowServerConfig): HttpRequest = {

val newUri = Try({
  val origUri = URI.create(request.uri)
  val queryParams = overrideValues(request.queryParams, config.queryOverride)
  new URI(origUri.getScheme, origUri.getUserInfo, origUri.getHost, origUri.getPort, origUri.getPath, marshal(FormData(queryParams)).right.get.asString, origUri.getFragment).toASCIIString

val newEntity = request.header[HttpHeaders.`Content-Type`] match {
  case Some(_) => {
      val form =[FormData].right.get
      val formParams = form.copy(fields = overrideValues(form.fields, config.formOverride))

  case None => request.entity

HttpRequest(request.method, newUri, request.headers, newEntity, request.protocol)

The TLS support is described here: I do not know how to send a HTTPS-Request object.

Bringing it in an example, it should look like this:

The client sends an request to the url of the shadow proxy: 
"https://localhost:8000" and the shadowProxy forwards it to  
a) the true-server "https://localhost:1234" 
b) and to the shadow-server "https://localhost:2345" 

Thanks for any help.

by Lamine at July 21, 2014 01:20 PM

Ansible Django Module Alternative Python Interpreter

I'm writing an ansible playbook for deplying a django app and as part of the process I'd like to run the staticcollect command. The issue I'm having is that the remote server has two python interpreters, one Python2.6 and one Python2.7, with Python2.6 being the default.

When I run the playbook it runs using the Python2.6 interpretor and I need it to run against the Python2.7 interpretor. Any idea on how this can be acheived?

My playbook is as follows:

- hosts: xxxxxxxxx
    hg_branch:  dmv2
    django_dir: /opt/app/xxxx
    conf_file:  /opt/app/conf/xx_uwsgi.ini
        STATIC_ROOT: /opt/app/serve/static
  remote_user: xxxxxx
  - name: Update the hg repo
    command: chdir=/opt/app/xxxxx  hg pull -u --rev {{hg_branch}}
  - name: Collect static resources
    environment: django_env
    django_manage: command=collectstatic app_path={{django_dir}}
  - name: Restart the django service
    command: touch {{conf_file}}
  - name: check nginx is running
    service: name=nginx state=started

by user1662366 at July 21, 2014 01:18 PM


AWS Week in Review - July 14, 2014

Let's take a quick look at what happened in AWS-land last week:

Monday, July 14
Tuesday July 15
Wednesday, July 16
Thursday, July 17
Friday, July 18

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

-- Jeff;

by Jeff Barr ( at July 21, 2014 01:13 PM


Translating "Why Functional Programming Matters" into Haskell

For cultural and intellectual enrichment, I've decided to learn a bit of Haskell. I've been reading Hughes' "Why Functional Programming Matters" and am trying to translate its code into true Haskell. I've attached some of my attempt below (for the numerical parts of the paper; the alpha-beta algorithm is even more interesting but I'd also have to write a game evaluator from scratch!).

At this point it's been more of an exercise in Haskell syntax than anything else. I've already done simple things like translate repeat into the native Haskell iterate, translate a few functions that used a lot of parentheses to function composition (making them more point-free in the process), etc.

But my code certainly works, and I wonder if it's sufficiently "Haskell-ish". Can any experts out there give me some hints?

-- 4.1 Newton-Raphson square roots
next n x = (x + n/x)/2.0

-- -- this is "iterate::(a->a)->a->[a]"
-- repeat f a  = a : iterate f (f a)

within eps (a:b:rest) = 
  if abs(a-b) <= eps 
    then b
    else within eps (b:rest)

sqroot a0 eps n = within eps (iterate (next n) a0)

relative eps (a:b:rest) = 
  if abs(a-b) <= eps*abs(b)
    then b
    else relative eps (b:rest)

relativesqrt a0 eps n = relative eps (iterate (next n) a0)

-- 4.2 numerical differentiation

easydiff f x h = (f (x+h) - f x) / h

differentiate h0 f x = map (easydiff f x) (iterate (/2) h0)

-- diff1a h0 eps f x = within eps (differentiate h0 f x)
diff1 h0 eps f = within eps . differentiate h0 f 

elimerror n (a:b:rest) = (b*(2**n)-a)/(2**n-1) : elimerror n (b:rest)

-- need fromIntegral to make a non-integer out of the Int which comes out of round
order (a:b:c:rest) =  fromIntegral (round (logBase 2 ((a-c)/(b-c)-1)))

improve s = elimerror (order s) s 

--diff2a h0 eps f x = within eps (improve (differentiate h0 f x))
diff2 h0 eps f = within eps . improve . differentiate h0 f 

--super s = map second (iterate improve s) -- how can we make this point-free?
super :: (RealFrac t, Floating t) => [t] -> [t] 
           -- w/o this it wants to be [double]->[double]
super = map second . iterate improve

-- second (a:b:rest) = b
second = head . tail

diff3 h0 eps f = within eps . super . differentiate h0 f

-- 4.3 integration

easyintegrate f a b = (f a + f b)*(b-a)/2

-- addpair becomes (uncurry (+))

integrate f a b = integ f a b (f a) (f b) 

integ f a b fa fb = 
  (fa+fb)*(b-a)/2 : map (uncurry (+)) (zip (integ f a m fa fm) (integ f m b fm fb))
  where m = (a+b)/2 
        fm = f m 

-- test: following should be about pi
approxpi eps = within eps (improve (integrate (\x -> 4/(1+x*x)) 0 1))
superpi eps = within eps (super (integrate (\x -> 4/(1+x*x)) 0 1))

-- is there any way to keep track of the number of iterations? state monad, but seems like a lot of work...\

by Andrew Jaffe at July 21, 2014 01:12 PM

Scala: combine elements in same position of two lists

I've two lists and I want to obtain another list combining, in a tuple, elements in same position.

An example:

val x = List(1,2,3)
val y = List(4,5,6)

result = List((1,4),(2,5),(3,6))

by diegoo at July 21, 2014 01:07 PM

Check for acceptance of type, rather than value, with isDefinedAt

I have a case where I want use isDefinedAt to check if a partial function accepts a type, rather than a specific value.

val test: PartialFunction[Any, Unit] = { 
  case y: Int => ???
  case ComplexThing(x, y, z) => ??? 

Here you could do something like test isDefinedAt 1 to check for acceptance of that value, however, what I really want to do is check for acceptance of all Ints (more specifically, in my case the type I want to check is awkward to initialize (it has a lot of dependencies), so I would really like to avoid creating an instance if possible - for the moment I'm just using nulls, which feels ugly). Unfortunately, there is no test.isDefinedAt[Int].

I'm not worried about it only accepting some instances of that type - I would just like to know if it's completely impossible that type is accepted.

by Lattyware at July 21, 2014 01:06 PM

Portland Pattern Repository


sslserver using 100% cpu

I've a qmail setup with sslserver being used to secure the pop connections. Normally it runs very well, however some instances of sslserver start taking 100% cpu and its child process qmail-pop3d process is seen as zombie. The process has to be killed to solve the prb. The sslserver version is 0.70. OS is CentOs 6.0 64 bit with 2.6.32-220 kernel. Any clues how I can debug this ?

by amolkul at July 21, 2014 12:53 PM


Installing pymatbridge on Windows

In version 0.3 of pymatbridge, an additional dependency was added for ZMQ. To install this version, you might therefore need to compile a messenger mex extension linking against a dll of zeromq. However, this turns out to be rather difficult to do in some cases.

Does anyone have examples of success stories installing pymatbridge from source on a machine running Windows?

by arokem at July 21, 2014 12:49 PM

Do you need to install Scala separately if you use sbt?

Reason I ask, is because it's possible to specify a Scala version in the build.sbt file (using scalaVersion setting), and once you do that, sbt will automatically download that Scala version to use with the project.

I also seem to remember that despite having Scala 2.11.1 on my system, sbt would compile and run with Scala 2.10 if no version was specified.

So the question is, do I need to install Scala separately if I got sbt installed?

by Electric Coffee at July 21, 2014 12:34 PM

Future with State monad

I would like to use State monad to implement caching for data provided from third party API. Let's imagine method getThirdPartyData(key: String) which firstly checks cache then if it's not present there should make request to API. First and most naive implementation which came to my mind was enclosing State type within Future -

Future[State[Cache, ThirdPartyData]]

But it's not correct because when request fails you will lose your cache (getThirdPartyData will return Failure).

Second option which came to my mind was to extend, or rather redefine State monad - s => Future[(s,a)], instead of s => (s,a), but I thought that it's quite popular problem so scalaz probably has some already defined way to solve this issue.

Any help greatly appreciated!

by Kamil Badyla at July 21, 2014 12:33 PM

Clojure - how to use my personal debug library in multiple projects?

I have a mascip.debug library with a few functions that I like to use everywhere. In each of my projects (only two at the moment - I'm a new Clojurian), I have this library in src/.

I would like to put it somewhere in my PATH, and then be able to use it in any project that I write. Is that silly? Possible? Or would you say I absolutely should start using Maven, as explained here, so I can declare the dependency in my ~/.lein/profiles.clj file?

by mascip at July 21, 2014 12:26 PM


Comparison of multicurve calibration methods

It seems that there are mayor softwares around offering a multicurve framework based on bootstrap. I find this puzzling nowadays, given the distinct advantages of best-fit optimization methods and the hurdles in extending bootstrap techniques to the multicurve setting (e.g. cyclic interdependencies among curves, nontrivial products, usage of nonliquid instruments, overall coherence, dates mismatches, TOY effect, pre-first-tenor forwards, joint curves+term structure dynamics calibration etc). Therefore I must be missing some major drawback of full calibration or overestimating issues for bootstrap. The literature is outdated and mostly partisan, just like those I asked to dismiss either choice altogether. Could someone please shed some more light on these two possibilities and respective drawbacks?

(Moreover, Henrard mantains that for best-fit calibration a Newton-Rhapson optimizer suffices, while in my opinion the landscape is not so well-behaved... any views on this?)

by Quartz at July 21, 2014 12:20 PM


How to disable http port in play framework?

When I run Play Framework using https port, application starts on two ports - default (9000) and https (443). How to disabled default 9000 port and run the Play application ONLY on https port?

I run application through the following command:

play -Dhttps.port=443 -Dhttps.keyStore=/path/to/keystore -Dhttps.keyStorePassword=password start

I get some logs:

[info] play - Application started (Prod)
[info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
[info] play - Listening for HTTPS on port /0:0:0:0:0:0:0:0:443

by Roman Krolikov at July 21, 2014 12:07 PM



Scala + Gradle + AndroidStudio 0.8.+

Is it still impossible to write Android applications with Scala/Gradle/AndroidStudio pack? I couldn't find anything relevant on the web.

Links would be great. Thanks.

by midnight at July 21, 2014 12:03 PM


Der Iran hat die Hälfte seines Bestands an hochangereichertem ...

Der Iran hat die Hälfte seines Bestands an hochangereichertem Uran unschädlich gemacht. Für Atomkraftwerke braucht man 5% Anreicherung, für Kernwaffen 90%, der Iran hatte Uran mit 20%.

Schade nur, dass Israel gerade eine Gaza-Offensive fährt, so fällt diese Meldung komplett unter den Tisch.

July 21, 2014 12:02 PM


Why is Scala's syntax for tuples so unusual?

In mathematics and computer science, a tuple is an ordered list of elements. In set theory, an (ordered) n-tuple is a sequence (or ordered list) of n elements, where n is a positive integer.

So, for example, in Python the 2nd item of a tuple would be accessed via t[1].

In Scala, access is only possible via strange names t._2.

So the question is, why can't I access data in tuples as Sequence or List if it is by definition? Is there some sort of idea or just yet not inspected?

by yura at July 21, 2014 11:54 AM

Eclipse: How do I quickly open a folder (in the package, project or resource explorer) with a keyboard shortcut?

I am used to quickly search and open files with CTRL+SHIFT+R - Open Resource. However, Open Resource does not match folders. I would need to do the same for folders within the project, or alternatively, for matching files with part of their project-relative path.

The use case is working with Ansible roles, where all files are named "main.yml" and are located in a hierarchical folder structure:

           |     +----main.yml 
           |     +----main.yml 
           |     +----main.yml 

And I need to quickly open the various main.yml files by searching for their path e.g. typing something like

CTRL + SHIFT + <<insert key here>> 



and obtain a short list including apache/vars/main.yml

Is there any feature or plugin providing such functionality? It would also be useful for opening maven pom.xml files and the like...

by Carlo Bonamico at July 21, 2014 11:47 AM

Scala : statements are not executed in order

I am learning scala I tried following program

package com.avdongre

object MyModule {
  def abs(n: Int): Int =
    if (n < 0) -n
    else n
  private def formatAbs(x: Int) = {
    val msg = "The absolute value of %d is %d"
    msg.format(x, abs(x))

  def factorial(n: Int): Int = {
    def go(n: Int, acc: Int): Int =
      if (n <= 0) acc
      else go(n - 1, n * acc)
    go(n, 1)
  def main(args: Array[String]): Unit =    


I get following output

The absolute value of -42 is 42

Why factorial is getting called first ?

by Avinash at July 21, 2014 11:46 AM

Java / Scala import not working due to incorrect build path?

I'm very confused about how you import packages and their classes. I have two scala projects with the following directory structures:

    * seemingly the same as project 1 *

The contents of some_file.scala:

package utils

class InterestingClass {
  override def toString(): String = "I should be interesting"

The contents of

package worksheet

import utils.InterestingClass

object learning {
  println("Welcome to the Scala worksheet")       //> Welcome to the Scala worksheet
  println(new InterestingClass())                 //> I should be interesting

When I created project1/src/main/scala/, in Eclipse I right clicked main.scala that was showing as a package underneath src and selected: Build Path > Use As Source Folder, and in Eclipse then "deleted" the src folder. project2 is actually a project I have downloaded as part of an online course so I guess there's some config in the .project file or the .settings folder that I need to tweak because in project1 auto compiles as expected and produces the two outputs including I should be interesting but in project2 it does not and the line import utils.InterestingClass shows the error not found: object utils.

I guess there's a subtle different in the build paths but I can't find it and the error message doesn't get me much further. Any tips on how to debug import errors? Looking at the build paths, they look correct (both have project1/src/main/scala or project2/src/main/scala).

by AJP at July 21, 2014 11:43 AM

Planet Emacsen

Irreal: Backup Fonts for Unicode Glyphs

Trey Harris posed a question on the Google+ Emacs Community that’s been bothering me for some time. I use the terrific Inconsolata as the default for monospaced fonts but many of the Unicode glyphs are missing. Mac OS X takes care of finding missing glyphs in native apps like or but with Emacs I just get a missing glyph. I’ve tried various fixes over the years but nothing worked very well.

Harris suggested using the unicode-fonts package to handle this. Happily, it’s an ELPA package so I just loaded it up and added the required lines to my init.el and those missing glyphs starting appearing in Emacs. Years of frustration gone just like that.

There are still a few missing characters but the README for unicode-fonts lists a few font packages that you can load to supply most characters you’re apt to need. Even without loading the extra fonts, I’ve got almost everything I need so I haven’t bothered with them. If you’ve been having a problem with missing Unicode glyphs, give unicode-fonts a try. It’s easy to load with ELPA and if it doesn’t work for you, it’s easy to uninstall. It’s a huge win for me.

by jcs at July 21, 2014 11:42 AM


Are all Chomsky-Type3 grammars LL(1)?

Referring to this Question, where an answer is stating that all Type 3 languages are LL(1), I'd like to know if all Type 3 grammars are possibly LL(1).

If not, why is it so? Are there maybe ambiguous grammars in the set of the Type 3 grammars?

by Theolodis at July 21, 2014 11:41 AM


Complexity of an algorithm for deciding 3-colorability of graph by the chromatic polynomial modulo $x-3$

As explained on MO computing the chromatic polynomial $P(G,x)$ modulo $x-3$ is enough for deciding 3-colorability.

For non adjacent vertices $u$ and $v$, $G+uv$ is the graph with the edge $uv$ added and the graph $G/uv$ is obtained by merging the two vertices.

According to wikipedia

$$P(G,k)= P(G+uv, k) + P(G/uv,k) \qquad (1) $$

The recursion for computing $P(G,x)$ modulo $x-3$ is siginificantly faster since induced $K_4$ stops computing further.

Consider the following naive algorithm for checking 3-colorability using $P(G,x) \pmod{x-3}$.

 function color3(G: connected graph)
 #returns 0 if G is not 3-colorable, otherwise reports it is and stops
 if G contains induced K_4 
     return 0
 if G is a clique on 3 or less vertices 
     report '3-colorable'. Stop.
 if we can add edge uv which induces K_4 in G 
     return color3(G/uv) #the induced K_4 contributes 0 to the sum
 #the above makes the problem strictly smaller
 #this might be greatly improved by trying to induce K_4-e
 [C] pick the lexicographic first non-adjacent u,v and return color3(G+uv)+color3(G/uv)

One call to color3 is polynomial, so the complexity depends on the number of double recursion in [C].

According to Wikipedia the worst case for computing (1) is $\phi^{n+m}$. 3-coloring 4-regular graphs is NP-complete, so for 4-regular it is $\phi^{3n}$.

Appears to me terminating by induced $K_4$ would greatly improve the running time.


Q1 What is the complexity of color3?

$O(c^n)$ for small $c$ will be of practical interest. $2^{o(n)}$ might indicate complexity collapse.

Q2 How to improve [C]?

There might be other improvements like other 4-chromatic subgraph or $F$ being in a graph class where coloring is polynomial.

Experimental results:

For several 4-regular graphs on 246 vertices which are the line graphs of 3-regular graphs color3 found 3-coloring in less than a minute in sage 6.2 implementation. For several smaller graphs which are not 3-colorable it correctly returned 0. For reduction SAT to 3-color the running time was fast too.

Sage implementation:

by joro at July 21, 2014 11:32 AM


Sending and receiving different types of messages with spray

I am using spray routing and spray client. I see that pipleline in spray client allows to define, by default, one and only one type of answer. I want to expose with spray routing a rest json service which replies either with a ServiceResponse type, either with a Error response type. I can define types and marshalling of ServiceResponse and Error on spray routing, and define a route which returns either a ServiceResponse or an Error.

I don't see how to call that from spray client, since the result would be a Future[ServiceResponse] or Future[Error]. The only solution I made work is to define an encapsulation : MetaResponse(Option(ServiceResponse), Option[Error]). In the spray client I use pattern matching to know in which case I am. But I find this solution ugly.

Is there a way to specify that a given client request can expect several return types?


by Joel at July 21, 2014 11:32 AM


Please try to break my login system...

Hi everyone!

As a test for a project I just came up with a login system which I'm hoping is secure without needing to use TLS/SSL. It works in a similar way to HTTP digest authentication, CRAM-MD5, etc., but only uses JS and Crypto-JS (for SHA512) on the client side.

Now before you ask, no this is not meant to be a replacement for TLS. Of course TLS is far better and this system is simply designed to stop packet sniffers and the like from stealing passwords in plain text. It doesn't stop session hijacking and it doesn't encrypt normal traffic, it just secures the login (hopefully).

I have included some example data that would have been captured by packet sniffing during a valid login for the "admin" user. My question is, can anybody use the form and the data given to find admin's password? Also if anybody finds any flaws in the system I'd love to hear about them.

Thanks all, I'll look forward to hearing your comments!


submitted by polaris64
[link] [2 comments]

July 21, 2014 11:23 AM


Functional way of joining two js object arrays based on common id

I am trying to acheive something similar to SQL table join, in the most elegant (functional) way, preferably with underscore.js, so no for loops please.

I need to merge objects from two different arrays, matched upon a common identifier.

For example, given:

var basic = [{
              id: '1',
              name: 'someName',
             {...} ]

var ext= [{
              id: '1',
              job: 'someJob',
             {...} ]

Result should be:

var combined = [{
                 id: '1',
                 name: 'someName',
                 job: 'someJob',
                {...} ]


by S.Kraus at July 21, 2014 10:58 AM

Parallelize Scala's Iterator

Please note: This is not a duplicate question, since this question specifies on all methods Iterator has, not just map and flatMap. Therefore Future.traverse are not a good answer.

Let's say I have this simple statement:

(1 to 100).toSet.subsets.find(f)

It works perfectly. It's lazy, doesn't use a lot of memory and returns as soon as one element is found. The problem starts when you want to parallelize it. You might say, it's Scala, there has to be .par or Iterator, but there isn't.

The proposed solution on the internet is to use .grouped, but it's not as good as I'd want. Why?

val it = (1 to 100).toSet.subsets.grouped(1000000).map(_.par.find(f)).flatten
if (it.hasNext) Some( else None
  1. Uses much more memory. I know it's still O(1), but let's be perfect here :)

  2. It's not perfectly parallelizable (by Amdahl's law). When .grouped is consuming the iterator for next block of million elements, all but one thread waits. This is especially problematic if iterator is expensive to consume. Moreover, there has to be an overhead of spawning a new set of threads to work on a new block.

  3. Produces more complicated / longer code (see example). It would shorten code a bit if Iterator had .nextOption, but still.

Is there anything else despite programming my own producer-consumer model (iterator is producer, threads are consumers) and then final reduce step?

by Rok Kralj at July 21, 2014 10:53 AM

How to remove an item from a list in Scala having only its index?

I have a list as follows:

val internalIdList: List[Int] = List()

internalIdList = (11, 12, 13, 14, 15)

From this list would remove the third element in order to obtain:

internalIdList = (11, 12, 14, 15)

I can not use a ListBuffer, are obliged to maintain the existing structure. How can I do?

Thanks to all

by YoBre at July 21, 2014 10:50 AM


Lustiger Vortrag von djb. Er macht sich da Gedanken, ...

Lustiger Vortrag von djb. Er macht sich da Gedanken, wie man, wenn man die NSA ist, dafür sorgen könnte, dass die Software da draußen schön unsicher bleibt. Money Quote (aus dem Cybersecurity-Blablah der US-Regierung):
Subcategories in Framework: 98.
… promoting secure software: 0.

This is how the money is spent.

Inhaltlich hat er natürlich völlig Recht. Das ganze System baut darauf auf, dass alle Beteiligten mit Bullshit abgelenkt werden, damit es nicht dazu kommt, dass jemand tatsächlich was tut, um die Lage zu verbessern.

July 21, 2014 10:02 AM


Convert DFA to Regular Expression

In this old exam-task I don't understand all the steps to convert the DFA below to a Regular Expression. The q_2 state is eliminated first. The provided solution to eliminate q2 is:

If we first eliminate q2 we obtain an intermediate automata with 3 states (q0 , q1 and q3 ) such that: 1. We go from q0 to q1 with RE a+ba 2. We go from q0 to q3 with RE bb 3. We go from q1 to q1 with RE ba 4. We go from q1 to q3 with RE a+bb

I don't understand nr2. q3 can also be reached using RE aa. Why is this left out?

enter image description here


enter image description here


enter image description here

by user20232 at July 21, 2014 09:31 AM


Scala framework for a Rest API Server?

We are thinking on moving our Rest API Server (it is inside the web service, on Symfony PHP) to Scala for several reasons: speed, no overhead, less CPU, less code, scalability, etc. I didn't know Scala until several days ago but I've been enjoying what I've been learning these days with the Scala book and all the blog posts and questions (it's not so ugly!)

I have the following options:

  • build the Rest API Server from scratch
  • use a tiny Scala web framework like Scalatra
  • use Lift

Some things that I will have to use: HTTP requests, JSON output, MySQL (data), OAuth, Memcache (cache), Logs, File uploads, Stats (maybe Redis).

What would you recommend?

by fesja at July 21, 2014 09:24 AM

What are alternatives to JDBC driver for access PostgreSQL database

I'm using official JDBC driver for PostgreSQL, but I'm stuck with the following issues:

  • No support for PostgreSQL-ish data structures such as UUIDs.
  • Common JDBC weirdness, such as:
    • No function to escape values for consuming by PostgreSQL.
    • Limited support for executing heterogeneous statements in batch.
    • No rewriting of multiple insert statements into single insert statement when inserting many rows in one table.

So, the question — is there any PostgreSQL database driver which can leverage full power of PostgreSQL without much boilerplate? I'm also use Scala language for development, so if driver is designed specifically for Scala it would be so much awesome awesome.

by andreypopp at July 21, 2014 09:10 AM


Worked examples of applying Ito's lemma

In most textbooks Ito's lemma is derived (on different levels of technicality depending on the intended audience) and then only the classic examples of Geometric Brownian motion and the Black-Scholes equation are given.

My question
I am looking for references where lots of worked examples of applying Ito's lemma are given in an easy to follow, step by step fashion. Also more advanced cases should be covered.

by vonjd at July 21, 2014 09:07 AM


Hibernating to Encrypted softraid(4) Now Supported

With this commit, Mike Larkin (mlarkin@) has added support for hibernating to encrypted softraid(4) devices. This is what he had to say when asked about it:

After RLE support (which went in in Slovenia), the next thing on the list to tackle was softraid crypto. Theo provided the initial idea on how to get the block transforms and crypto bits working over lunch one day in Slovenia and after about three or four days of on-and-off hacking this week, we had it working.


July 21, 2014 09:02 AM



Apache Spark:Which one is more efficient?

Code is writed by scala,I'm wonder which one is more efficient:


or :

    //if word contains xx,fuc will return (),how to deal with it ?>if(!word.contain("xx")(word,1))
    //result:Array((), (c,1), (d,1))

If word contains xx,fuc will return (),how to deal with it ?

by MrQuestion at July 21, 2014 09:00 AM

What is the inverse of intercalate, and how to implement it?

This question discusses how to interleave two lists in an alternating fashion, i.e. intercalate them.

  • What is the inverse of "intercalate" called?
  • Is there an idiomatic way to implement this in Scala?

by DNA at July 21, 2014 08:58 AM


g2k14: Stefan Sperling on wireless drivers

I spent most of this hackathon looking at problems in wifi drivers.

I wasn't exactly sure in advance which problems I wanted to work on. So I packed a bunch of hardware, including several USB wifi adapters, (rsu(4), 2x run(4), rum(4), urtwn(4), zyd(4)), some miniPCIe cards (an unsupported cousin of urtwn(4) named Realtek 8188CE, unsupported athn(4) AR9485, bwi(4)), two laptops, and an access point. This left me with more than enough toys for a week.

July 21, 2014 08:55 AM

g2k14: Florian Obser in IPv6 land

I arrived in Ljubljana somewhat tired so I started the first day off with some light ping(8) and ping6(8) hacking. Some unifdef(1) application for
#ifdef FEATURE_THAT_EXISTS_SINCE_FOREVER_BUT_MAYBE_WE_DONT_HAVE_IT and some cleanup by hand. The idea is to have ping(8) and ping6(8) be the same binary like traceroute(8) and traceroute6(8).

July 21, 2014 08:54 AM


Does Modern Portfolio Theory align with EMH?

I came to the conclusion that in literature Markowitz' Portfolio Theory is believed to be compliant with the Efficient Market Hypothesis. The weakest form states that the current price fully incorporates information contained in the past history of prices only. That is, nobody can detect mispriced securities and “beat” the market by analyzing past prices.

I understand this means that future price movements are independent from what happened until now. In MPT though, to allocate funds between assets, we estimate the expected return and covariances from past prices. If we want to be compliant with EMH, shouldn't we believe that those estimates make no sense?

by Karol Przybylak at July 21, 2014 08:47 AM


Use '==' instead of equals for string comparison in Scala

Why is it recommended to use == rather than .equals for string comparison in Scala? There are a lot of questions in StackOverflow that do not recommend the use of reference equality for String comparison in Java but why is it advised to do the exact reverse in Scala?

by Core_Dumped at July 21, 2014 08:47 AM

SBT 0.13 taskKey macro doesn't work with [Unit]?

lazy val buildDb = taskKey[Unit]("Initializes the database")

buildDb := {
  (compile in Compile).value
  val s: TaskStreams = streams.value"Building database")
  try {
  } catch {
    case e: Throwable =>
      sys.error("Failed to initialize the database: " + e.getMessage)
  }"Finished building database")

This produces the following error

C:\work\server\build.sbt:98: error: type mismatch;
 found   : Unit
 required: T"Finished building database")
[error] Type error in expression

But if I define it like this lazy val buildDb = taskKey[String]("Initializes the database") and then add to the last line in the task "Happy end!" string everything seem to work. Am I to blame, or something wrong with the macro?

by yevgeniy mordovkin at July 21, 2014 08:45 AM


g2k14: Ingo Schwarze on manly stuff

In the week right before the hackathon, I have done quite a bit of work cleaning up mandoc(1) warning and error messages. The goal is to provide more, more precise, and more readily understandable information to the user, in particular mentioning in the messages which section titles, macro names, and arguments each individual message is related to, and which workaround or fallback mandoc(1) has chosen, if any.

July 21, 2014 08:32 AM


Sort list of string with localization in scala

i want to sort a list of strings. I know that's not difficult in scala but my problem is, that i need to sort lists in different languages. For example i know that i can sort strings in english very easily. But what's about the russian language or the romanian one? What is the best practice for sorting strings in multiple languages in scala? Does the scala sorting implementation support only english letters?

In java i would do something like this:

Collator coll = Collator.getInstance(locale);
Collections.sort(words, coll);

I hope someone out there can help me. Thanks in advance Nico.

by Nico at July 21, 2014 08:25 AM





ansible : how to pass multiple commands

I tried this :

- command: ./configure chdir=/src/package/
- command: /usr/bin/make chdir=/src/package/
- command: /usr/bin/make install chdir=/src/package/

which works but I guess there is something more.. neat.

So I tried this :

from : multiple commands in the same line for bruker topspin which give me back "no such file or directory"

- command: ./configure;/usr/bin/make;/usr/bin/make install chdir=/src/package/

I tried this too :

but I couldn't find the right syntax to put :

- command: "{{ item }}" chdir=/src/package/
      /usr/bin/make install

That does not work, saying there is a quote issue.

Anyone ? Thank you

by John Doe at July 21, 2014 07:49 AM


Share your stories about algorithms in action, in real-life.

Have a personal algorithm that you use on a daily basis? Own an electrical appliance with smart behavior? Noticed algorithms in action, in real life? Let's share them.

Here's an interesting "optimization" algorithm I've noticed in my living area.

I live in a residential block that is 25-stories high. The 2 elevators/lifts (US/UK designation respectively) used to be slow moving machines that caused huge waiting times during morning rush hours when people were leaving home for work. So recently, the elevators got an uplift. Besides being mechanically upgraded, it seems that their software was made intelligent.

1) If there is exactly one idle elevator (A) while B is moving upwards, A will move to the ground floor to wait.

2) If there is exactly one idle elevator (A) while B is moving down towards the ground floor, A will move to the 17th floor to wait.

3) If both elevators are idle, one will be randomly chosen to move to ground floor, and the other to the 17th floor to wait.

A and B are inter-changeable.

What is this achieving? Firstly, the lift engineers have recognized that the ground floor is the most accessed floor, so having one elevator wait there at all times will likely minimize the sum of waiting time throughout the day.

Then, what's so special about the 17th floor? Turns out that if you have idle elevators at the ground (1st) and 17th floors, the maximum amount of floors that either elevator has to traverse to reach a passenger anywhere is minimized at 8 (minimax strategy). For example, the elevator at the ground floor can reach floors 2-9 within 8 floors, while the elevator at the 17th floor can reach floors 9-25 within 8 floors.

We no longer have two derpy elevators that idly wait at the 25th floor, in the worst case. Simple algorithm that has made my life better!

submitted by crimsonSoldier
[link] [41 comments]

July 21, 2014 07:43 AM



Which textbook can I use after a high school CS course?

I just finished the AP Computer Science course in high school and since my school does not have any further classes, I was thinking about getting a textbook and continuing my study. APCS goes through Big O, Arrays + ArrayLists, Recursion, Linked Lists, a rough overview of some other data structures and some sorting and searching algorithms (all in Java).

I will start Calculus in school next year (if that helps to determine math level).

Since I was thinking about participating in CS tournaments, I thought it would be useful to learn more about algorithms and data structures.

I was looking at two textbooks, The Algorithm Design Manual by Steven S Skiena and Introduction to Algorithms by Cormen et. al. I was wondering which one, if any of these, would best fit my situation.

by user20276 at July 21, 2014 07:26 AM


How to apply new changes without dropping current data in the db.(Playframework evaluations)

I'm using play 2.2.1 with scala. And I have a managing database evaluations. When I run my application with changes on the database, it'll drop all my datas. At this moment evaluationplugin = disabled is comment in. If I comment out, it does not apply my changes.
For example. I have a users table and there are id, f_name, l_name

id   f_name  l_name
1.   khazo   rasp

And I want to add age field to this table without losing data. I've added this field in scala files.It works properly. I'm assuming I need to write script in 1.sql for some alter command, but I don't want to write script.

How to apply new changes without dropping current data in the db. I've read this documentation. Thanks in advance.

by Khwarezm Shah at July 21, 2014 07:24 AM

Fred Wilson

Free International Roaming With A Premium Upsell

I just landed in Berlin after an overnight flight from the US.

In the past, turning on your phone after landing overseas could be an expensive experience as the phone downloads all the email you received since taking off at international mobile data rates.

I’ve used a host of techniques over the years to avoid the experience of landing, turning on my phone, and immediately getting a text message that I’ve blown past my international data roaming cap.

I’ve turned off mobile data and waited until I got to hotel WiFi to download my email but that meant no mobile data for directions to the hotel. I’ve bought SIM cards in airports. And more recently I’ve rented a pocket WiFi before traveling overseas.

But last year the Gotham Gal and I switched back to T-Mobile after they introduced free low bandwidth international data roaming for all customers in the US.

Here is the experience when I land. I turn on the phone, it finds the local mobile network, connects, and my phone lights up with notifications and emails start coming in.

In addition I get a text message from T-Mobile offering to upgrade me to an international data pass that offers 4G in 100MB buckets at roughly $10/100MB.

I buy the upgrade every time and am happy to pay for the higher speeds.

But the important thing here is the customer experience. No longer do customers have to fear turning on their phone. No longer do customers have to jump through hoops to procure an affordable mobile data plan. If you want faster speeds, T-Mobile makes it drop dead simple to upgrade right on your phone.

This approach to international mobile data should be adopted by all the mobile carriers. It’s a great experience.

by Fred Wilson at July 21, 2014 07:24 AM



Planet Clojure

Machine Learning in Clojure - part 2

I am trying to implement the material from the Machine Learning course on Coursera in Clojure.

My last post was about doing linear regression with 1 variable. This post will show that the same process works for multiple variables, and then explain why we represent the problem with matrices.

The only code in this post is calling the functions introduced in the last one. I also use the same examples, so post this will make a lot more sense if you read that one first.

For reference, here is the linear regression function:

(defn linear-regression [x Y a i]
(let [m (first (cl/size Y))
X (add-ones x)]
(loop [Theta (cl/zeros 1 (second (cl/size X))) i i]
(if (zero? i)
(let [ans (cl/* X (cl/t Theta))
diffs (cl/- ans Y)
dx (cl/* (cl/t diffs) X)
adjust-x (cl/* dx (/ a m))]
(recur (cl/- Theta adjust-x)
(dec i)))))))

Because the regression function works with matrices, it does not need any changes to run a regression over multiple variables.

Some Examples

In the English Premier League, a team gets 3 points for a win, and 1 point for a draw. Trying to find a relationship between wins and points gets close to the answer.

(->> (get-matrices [:win] :pts)
(print-results "wins->points"))

** wins->points **
A 1x2 matrix
1.24e+01 2.82e+00

When we add a second variable, the number of draws, we get close enough to ascribe the difference to rounding error.

(->> (get-matrices [:win :draw] :pts)
(print-results "wins+draws->points"))

** wins+draws->points **
A 1x3 matrix
-2.72e-01 3.01e+00 1.01e+00

In the last post, I asserted that scoring goals was the key to success in soccer.

(->> (get-matrices [:for] :pts)
(print-results "for->points"))

** for->points **
A 1x2 matrix
2.73e+00 9.81e-01

If you saw Costa Rica in the World Cup, you know that defense counts for a lot too. Looking at both goals for and against can give a broader picture.

(->> (get-matrices [:for :against] :pts)
(print-results "for-against->pts"))

** for-against->pts **
A 1x3 matrix
3.83e+01 7.66e-01 -4.97e-01

The league tables contain 20 fields of data, and the code works for any number of variables. Will adding more features (variables) make for a better model?

We can expand the model to include whether the goals were scored at home or away.

(->> (get-matrices [:for-h :for-a :against-h :against-a] :pts)
(print-results "forh-fora-againsth-againsta->pts"))

** forh-fora-againsth-againsta->pts **
A 1x5 matrix
3.81e+01 7.22e-01 8.26e-01 -5.99e-01 -4.17e-01

The statistical relationship we have found suggests that that goals scored on the road are with .1 points more than those scored at home. The difference in goals allowed is even greater; they cost .6 points at home and only .4 on the road.

Wins and draws are worth the same number of points, no matter where the game takes place, so what is going on?

In many sports there is a “home field advantage”, and this is certainly true in soccer. A team that is strong on the road is probably a really strong team, so the relationship we have found may indeed be accurate.

Adding more features indiscriminately can lead to confusion.

(->> (get-matrices [:for :against :played :gd :for-h :for-a] :pts)
(map *)
(print-results "kitchen sink”))

** kitchen sink **
(0.03515239958218979 0.17500425607459014 -0.22696465757628984 1.3357911841232217 0.4019689136508527 0.014497060396707949 0.1605071956778842)

When I printed out this result the first time, the parameter representing the number of games played displayed as a decimal point with no digit before or after. Multiplying each term by 1 got the numbers to appear. Weird.

The :gd stands for “goal difference” it is the difference between the number of goals that a team scores and the number they give up. Because we are also pulling for and against, this is a redundant piece of information. Pulling home and away goals for makes the combined goals-for column redundant as well.

All of the teams in the sample played the same number of games, so that variable should not have influenced the model. Looking at the values, our model says that playing a game is worth 1.3 points, and this is more important than all of the other factors combined. Adding that piece of data removed information.

Let’s look at one more model with redundant data. Lets look at goals for, against and the goal difference, which is just the difference of the two.

(->> (get-matrices [:for :against :gd] :pts)
(print-results "for-against-gd->pts"))

** for-against-gd->pts **
A 1x4 matrix
3.83e+01 3.45e-01 -7.57e-02 4.21e-01

points = 38.3 + 0.345 * goals-for - 0.0757 * goals-against + 0.421 * goal-difference

The first term, Theta[0] is right around 38. If a team neither scores nor allows any goals during a season, they will draw all of their matches, earning 38 points. I didn’t notice that the leading term was 38 in all of the cases that included both goals for and against until I wrote this model without the exponents.

Is this model better or worse than the one that looks at goals for and goals against, without goal difference. I can’t decide.

Why Matrices?

Each of our training examples have a series of X values, and one corresponding Y value. Our dataset contains 380 examples (20 teams * 19 seasons).
Our process is to make a guess as to the proper value for each parameter to multiply the X values by and compare the results in each case to the Y value. We use the differences between the product of our guesses, and the real life values to improve our guesses.

This could be done with a loop. With m examples and n features we could do something like

for i = 1 to m 
guess = 0
for j = 1 to n
guess = guess + X[i, j] * Theta[j]
end for j
difference[i] = guess - Y
end for i

We would need another loop to calculate the new values for Theta.

Matrices have operations defined that replace the above loops. When we multiply the X matrix by the Theta vector, for each row of X, we multiply each element by the corresponding element in Theta, and add the products together to get the first element of the result.

Matrix subtraction requires two matrices that are the same size. The result of subtraction is a new matrix that is the same size, where each element is the difference of the corresponding elements in the original matrices.

Using these two operations, we can replace the loops above with

Guess = X * Theta
Difference = Guess - Y

Clearly the notation is shorter. The other advantage is that there are matrix libraries that are able to do these operations much more efficiently than can be done with loops.

There are two more operations that our needed in the linear regression calculations. One is multiplying matrices by a single number, called a scalar. When multiplying a matrix by a number, multiply each element by that number. [1 2 3] * 3 = [3 6 9].

The other operation we perform is called a transpose. Transposing a matrix turns all of its rows into columns, and its columns into rows. In our examples, the size of X is m by n, and the size of Theta is 1 x n. We don’t have any way to multiply an m by n matrix and a 1 by n matrix, but we can multiply a m by n matrix and an n by 1 matrix. The product will be an m by 1 matrix.

In the regression function there are a couple of transposes to make the dimensions line up. That is the meaning of the cl/t expression. cl is an alias for the Clatrix matrix library.

Even though we replaced a couple of calculations that could have been done in loops with matrix calculations, we are still performing these calculations in a series of iterations. There is a technique for calculating linear regression without the iterative process called Normal Equation.

I am not going to discuss normal equation for two reasons. First, I don’t understand the mathematics. Second the process we use, Gradient Descent, can be used with other types of machine learning techniques, and normal equation cannot.

by Rick at July 21, 2014 06:15 AM


Planet Clojure

Clojure Gazette 1.85

Clojure Gazette 1.85
Magic, Games, Shootout

Clojure Gazette

Issue 1.85 July 20, 2014


Greetings loyal readers,

Yes, it's true. I have the best newsletter readers on the planet. You all are awesome and I'm continually blown away with the support you show me each time I ask. :)

And now, I ask for something again ;)

I've been writing the Clojure Gazette for over two years now. It's brought me joy. It's brought me pain. But I keep at it and I have grown to love it. It started on the free plan of the newsletter service but graduated some time ago to the not-free plan. I've paid for everything myself and continued to build the list.

Although the growth has been steady in the last year, the slope was way lower that I would have liked. I put any marketing efforts on the back burner, not really worrying about it too much. But now, I can't ignore it.

I have recently reworked the Clojure Gazette landing page to be a little bit snazzier and I'm starting to link to it more regularly from my blog. But I need your help.

There are two ways you can help: one is to send me a testimonial. Just a short blurb about how your life is better because you receive the Gazette and a profile-style picture. Be honest! Why do you read the Gazette? I can't promise I'll put yours on the landing page, but I will send you a nice thank you. A lot of people have sent me great emails over the years, but I never asked permission to publish them so I don't feel right putting them on the page.

The second way to help is by sharing it with your friends. It came as a surprise to me, but some people have never heard of the Clojure Gazette! If you know one of these people, it is your duty to inform them of its existence ;) And if you have any kind of internet presence (blog, Twitter, podcast, etc), I would greatly appreciate links, tweets, likes, reblogs, shares, plugs, +1s, hearts, favorites, thumbs up, high fives, pins, retweets, blogs, posts, or whatever your respective social networking technology uses. In return, I will issue fives, high fives, and low fives (unless you're too slow) to any and all at events that I frequent.

Thanks in advance!

Eric Normand <>

PS Please follow the DISCUSS links and comment!

Of Mages and Grimoires

Reid McKenzie has created new Clojure documentation site called Grimoire and this post discusses the reasoning behind it. It looks like a cool project to make the documentation very useful for quick access (faster than ClojureDocs). It also has lots of examples and the collection is growing. I'm glad someone is working on this. Check out, as an example, the page for defprotocol. It includes, in one place, the docstring, examples, and source. I'd love to see hyperlinks in the code to their respective definitions. DISCUSS

Clojure web server shootout

Charts and graphs comparing the performance of different Clojure web server options. I'm surprised to see Jetty Ring fare so poorly. A great service to the community. It's a collaborative repo and they are accepting benchmarks. DISCUSS

Brute - Entity Component System for Clojure

I have a confession to make: ever since I was young, I've wanted to program games. So, far from bein an unbiased link curator, I do have a softspot for game-related programming posts. This one is an Entity Component System, which is a pattern for composing game objects instead of writing them from scratch each time or using inheritance. And now it works in ClojureScriptDISCUSS

Room Key Case Study

I almost didn't link to this because it was not very technical, but then I thought that it was such a good document on so many other levels that it was worth sharing. It is well designed, well written, and well scoped. It would make a great post to email to a project manager to convince them to give Clojure and Datomic a try. DISCUSS


If game programming was one of my first programming loves, programming languages have to be second. And that love has been much more requited :) ki is an interesting language: it's a sweet.js macro that expands to Javascript, but inside the macro, it's a Lisp. The compiler can run in your browser. It uses mori for its persistent data structures. Very interesting! DISCUSS

Building Single Page Apps with Reagent

Om is all-the-rage, but there are alternative React libraries out there. Reagent is one of them and this post begins with a discussion of the advantages of Reagent over Om, then goes on to show the code and what kinds of things Reagent is good at. DISCUSS

Elm – functional reactive dreams + missile command

A smart, in-depth look at Elm, which is a great new language that's worth watching. DISCUSS


A Ring adapter embedded in nginx? Wowee, that's fast! (It's currently winning the Clojure web server shootout.) DISCUSS
Copyright © 2014 LispCast, All rights reserved.

unsubscribe from this list    change email address 

by Clojure Gazette at July 21, 2014 05:35 AM

Wes Felter Personal Cloud Platform Crowdfunding Personal Cloud Platform Crowdfunding:

I’ve been going on about the personal cloud on this blog for years. Sandstorm is the first project that I would back with my own money. They’re discarding all the user-hostile Linux baggage to create a platform that just works.

July 21, 2014 05:22 AM

Planet Clojure

Recommended Reading : Clojure for the Brave and the True ( for Beginners)

If you are learning Clojure, one of the hottest languages, there are lots of good books, web sites and more.

A relatively new resource from :


by admin at July 21, 2014 04:55 AM


How to measure programming language succinctness?

I want to explore the notion of quantifying the amount of succinctness a programming language provides. That is, the amount a high-level language reduces the complex.

This idea of "simplification" is a factor of text-wise reduction (fewer characters needed to express a complex concept, à la Algorithmic Information Theory) and another, less easy-to-quantify concept of maintainability. Fleshing out this latter concept, it is clear it has to do with how easily one can establish programmer consensus for the given task (i.e. how many programmers of the language would put it back the same way you've expressed it or otherwise agree on the best implementation between different implementations of the same problem?).

I will define the "Kolmogorov Quotient" so that higher numbers for a given language denote a reduction in the complexity of solving the problem in the given language.

The metric for "text-wise reduction" should incorporate a constant factor based on (non-identifier) symbols used in the language and source text. These factors will be the same across all languages implemented (i.e. designed) for a given architecture (e.g. VonNeumann architecture vs. Symbolics) and will be a measure of the significance of the symbol; i.e. the topology of the language tokens. (These terms, alas, will also need to be fleshed out and defined.)

Once the basic premise and a methodology above is agreed to, it is only a matter of a rough constant of difference for any specific implementation/architecture. (That is, as long as the architecture is the same across all measurements, the number should be valid and comparable between languages.)

But it could go something like this: Pick a language "close to the machine", like C or Assembly, and measure the amount of bytes of machine code it used to implement a standard suite(*) of common, non-threaded programming tasks (base_language_count). Then code the exact same functionality in the language you are wanting to measure (without using external libraries) and count the number of bytes of source code (test_language_count).

KQuotient = (base_language_count / test_language_count) / number_of_equal_programs.

"number_of_equal_programs" is the number of programs fluent programmers of the language agree that are the best and equal solutions to the problem. (I will define "equal programs" as those who's output is the same for every input.)

The first ratio should always be greater than 1.0. My only concern is that for each programming language, one could reduce the KQuotient number simply by changing each keyword to a single character.

(*) "standard suite of common programming tasks...": I see two main categories:

  1. Data-processing suite, limited to simple text I/O (computation towards the machine)
  2. GUI suite (computation towards the user)

The purpose of this idea is to end the tiring "language wars" about whose language is the best. By giving a quantitative metric, people can at least argue better.

by Mark J at July 21, 2014 04:42 AM


Org Users: What did it take you a long time to understand about org-mode?

I love Org Mode, but I haven't read all or even most of the manual. I just read this passage:

Important: Scheduling an item in Org mode should not be understood in the same way that we understand scheduling a meeting. Setting a date for a meeting is just a simple appointment, you should mark this entry with a simple plain timestamp, to get this item shown on the date where it applies. This is a frequent misunderstanding by Org users. In Org mode, scheduling means setting a date when you want to start working on an action item.

I had previously had this misunderstanding. Now I don't, but I suspect I'm still misunderstanding / under-utilizing other parts of Org. What did you misunderstand or not know about for a long time?

submitted by birdmanx35
[link] [15 comments]

July 21, 2014 04:30 AM


Issue understanding splitting data in Scala using "randomSplit" for Machine Learning purpose

Hi I am new in MLlib and I am reading the documents on Spark website about it. I have difficulty to understand why in the following code we need to cache "0" for training and "1" for testing:

  val splits = data.randomSplit(Array(0.6, 0.4), seed = 11L)
  val training = splits(0).cache()
  val test = splits(1)

Can anyone help me understanding the reason? As far as I know we need positive and negative samples so "1" can be positive and "0" can be negative, why it is divided like this?

Thank you!

by Rubbic at July 21, 2014 04:27 AM

can't get cookie after add cookie to a jsoup session

The following code will print an empty of "{}", why I can't see any cookie. Since I set the cookie in the second line, shouldn't expect return cookie?

var res = Jsoup.connect(url)
  .cookie("JSESSIONID", "somesessionID")  
  .data(postParamsList: _*)

The following will print the cookie after comment out the second line.

var res = Jsoup.connect(url)
//  .cookie("JSESSIONID", "somesessionID")  
  .data(postParamsList: _*)

by Daniel Wu at July 21, 2014 04:18 AM

Portland Pattern Repository



State of FreeBSD driver support on mobile hardware?

I'm currently a Slackware user out of the (perhaps misplaced) belief that the BSDs do not have good mobile hardware support, namely wireless and acpi power management. I'm generally fond of the simplicity of FreeBSD and found Slackware to be very similar in the regard, but with, generally, better driver support. Has this changed in recent years? I was reading the thread about the state of graphics support and some of the comments seemed encouraging. If it helps to have a hardware reference, I will be using a Thinkpad x61.

submitted by ewood87
[link] [8 comments]

July 21, 2014 03:44 AM


Just about every web app scanner I've looked at recently, including my beloved BurpSuite, injects references to their own infrastructure (URL, DNS, email, etc) in a way that could leak test and vulnerability data.

I don't necessarily want to call this 'shady' at this point, but i think it's messed up that almost none of these vendors openly disclose the fact that certain tests could leak vulnerability data to their infrastructure unless you take action to prevent it. (Acunetix discloses it with AcuMonitor) The only way to know for sure is to run a full suite of tests to see if you can find any such reference, but given some tests are 'optimized', there's really no way to ensure that you're seeing everything given any particular test scenario.

At this point I run all tests through a proxy that looks for this type of thing and both logs and rewrites it, but it's not always easy to detect this type of thing. It's probably not an issue for 99% of the folks using the product, but it is an issue for me. Not only because of data sensitivity concerns, but sometimes the test flat out isn't going to work because it's an internal site with no outside access. So you wind up with false negatives because the whole thing is tucked away.


submitted by dungoogled
[link] [3 comments]

July 21, 2014 03:17 AM



Proxying sbt ivy repositories with artifactory

I'm trying to do almost exactly what is described in the official sbt documentation, at I'd like to proxy both maven and ivy packages, and it seems like only Artifactory supports both. I went through the installation and got things roughly working, except that I couldn't point my remote repository (for local caching) at e.g., sbt-plugin-releases because it uses a custom repository layout (including the sbt version), which isn't allowed in the free version of Artifactory.

How do other people do this? It can't be uncommon if there's a whole documentation page on it. Does doing sbt stuff with a proxied ivy repository have to cost me $2750 a year? It's a little frustrating because I don't need most of the Artifactory Pro features, but I don't think I can get by without the custom layouts. Nexus and Archiva only seem to support Maven.

by copumpkin at July 21, 2014 03:07 AM

Portland Pattern Repository



Scala Programming

Let xs be a list of 100 elements in scala. Which of the following list operation is least expensive (takes shortest time) to complete based on the implementation described in the class?

A. xs.last

B. xs.tail

C. xs.init

D. xs.length

by user3859123 at July 21, 2014 02:56 AM


Proof of concept: ClojureScript source in HTML files

Live demo and github repository. Derived from ClojureScript compiled in ClojureScript, though any mistakes and bad code are mine.

My thinking was as follows: if his ClojureScript REPL can compile from strings entered into a text area, why can’t it compile from strings that are in an HTML document? Answer: it can.

I have put this all together without a thorough understanding of the original (as will be obvious when you look at the code).

submitted by jdeisenberg
[link] [1 comment]

July 21, 2014 02:55 AM


Portland Pattern Repository


arXiv Discrete Mathematics

Partitioning two-coloured complete multipartite graphs into monochromatic paths. (arXiv:1407.5083v1 [math.CO])

We show that any complete $k$-partite graph $G$ on $n$ vertices, with $k \ge 3$, whose edges are two-coloured, can be covered by two vertex-disjoint monochromatic paths of distinct colours, under the necessary assumption that the largest partition class of $G$ contains at most $n/2$ vertices. This extends known results for complete and complete bipartite graphs.

Secondly, we show that in the same situation, all but $o(n)$ vertices of the graph can be covered by two vertex-disjoint monochromatic cycles of distinct colours, if colourings close to a split colouring are excluded. The same holds for bipartite balanced graphs.

As a consequence of the above results, we prove that for $k\geq 2$, any complete $k$-partite graph whose edges are two-coloured can be covered by at most 14 vertex-disjoint monochromatic cycles (and for $k=2$, this number drops to 12). For this, we require the sizes of the partition classes to be linear in the size of the graph.

by <a href="">Oliver Schaudt</a>, <a href="">Maya Stein</a> at July 21, 2014 01:30 AM

Separation dimension of bounded degree graphs. (arXiv:1407.5075v1 [math.CO])

The 'separation dimension' of a graph $G$ is the smallest natural number $k$ for which the vertices of $G$ can be embedded in $\mathbb{R}^k$ such that any pair of disjoint edges in $G$ can be separated by a hyperplane normal to one of the axes. Equivalently, it is the smallest possible cardinality of a family $\mathcal{F}$ of total orders of the vertices of $G$ such that for any two disjoint edges of $G$, there exists at least one total order in $\mathcal{F}$ in which all the vertices in one edge precede those in the other. In general, the maximum separation dimension of a graph on $n$ vertices is $\Theta(\log n)$. In this article, we focus on bounded degree graphs and show that the separation dimension of a graph with maximum degree $d$ is at most $2^{9log^{\star} d} d$. We also demonstrate that the above bound is nearly tight by showing that, for every $d$, almost all $d$-regular graphs have separation dimension at least $\lceil d/2\rceil$.

by <a href="">Noga Alon</a>, <a href="">Manu Basavaraju</a>, <a href="">L. Sunil Chandran</a>, <a href="">Rogers Mathew</a>, <a href="">Deepak Rajendraprasad</a> at July 21, 2014 01:30 AM

An Alloy Verification Model for Consensus-Based Auction Protocols. (arXiv:1407.5074v1 [cs.SE])

Max Consensus-based Auction (MCA) protocols are an elegant approach to establish conflict-free distributed allocations in a wide range of network utility maximization problems. A set of agents independently bid on a set of items, and exchange their bids with their first hop-neighbors for a distributed (max-consensus) winner determination. The use of MCA protocols was proposed, $e.g.$, to solve the task allocation problem for a fleet of unmanned aerial vehicles, in smart grids, or in distributed virtual network management applications. Misconfigured or malicious agents participating in a MCA, or an incorrect instantiation of policies can lead to oscillations of the protocol, causing, $e.g.$, Service Level Agreement (SLA) violations.

In this paper, we propose a formal, machine-readable, Max-Consensus Auction model, encoded in the Alloy lightweight modeling language. The model consists of a network of agents applying the MCA mechanisms, instantiated with potentially different policies, and a set of predicates to analyze its convergence properties. We were able to verify that MCA is not resilient against rebidding attacks, and that the protocol fails (to achieve a conflict-free resource allocation) for some specific combinations of policies. Our model can be used to verify, with a "push-button" analysis, the convergence of the MCA mechanism to a conflict-free allocation of a wide range of policy instantiations.

by <a href="">Saber Mirzaei</a>, <a href="">Flavio Esposito</a> at July 21, 2014 01:30 AM

Solving Large Imperfect Information Games Using CFR+. (arXiv:1407.5042v1 [cs.GT])

Counterfactual Regret Minimization and variants (e.g. Public Chance Sampling CFR and Pure CFR) have been known as the best approaches for creating approximate Nash equilibrium solutions for imperfect information games such as poker. This paper introduces CFR$^+$, a new algorithm that typically outperforms the previously known algorithms by an order of magnitude or more in terms of computation time while also potentially requiring less memory.

by <a href="">Oskari Tammelin</a> at July 21, 2014 01:30 AM

Power law scaling and "Dragon-Kings" in distributions of intraday financial drawdowns. (arXiv:1407.5037v1 [q-fin.ST])

We investigate the distributions of epsilon-drawdowns and epsilon-drawups of the most liquid futures financial contracts of the world at time scales of 30 seconds. The epsilon-drawdowns (resp. epsilon- drawups) generalise the notion of runs of negative (resp. positive) returns so as to capture the risks to which investors are arguably the most concerned with. Similarly to the distribution of returns, we find that the distributions of epsilon-drawdowns and epsilon-drawups exhibit power law tails, albeit with exponents significantly larger than those for the return distributions. This paradoxical result can be attributed to (i) the existence of significant transient dependence between returns and (ii) the presence of large outliers (dragon-kings) characterizing the extreme tail of the drawdown/drawup distributions deviating from the power law. The study of the tail dependence between the sizes, speeds and durations of drawdown/drawup indicates a clear relationship between size and speed but none between size and duration. This implies that the most extreme drawdown/drawup tend to occur fast and are dominated by a few very large returns. We discuss both the endogenous and exogenous origins of these extreme events.

by <a href="">Vladimir Filimonov</a>, <a href="">Didier Sornette</a> at July 21, 2014 01:30 AM

A Fully Distributed Reactive Power Optimization and Control Method for Active Distribution Networks. (arXiv:1407.5032v1 [math.OC])

This paper proposes a fully distributed reactive power optimization algorithm that can obtain the global optimum of non-convex problems for distribution networks without a central coordinator. Second-order cone (SOC) relaxation is used to achieve exact convexification. A fully distributed algorithm is then formulated corresponding to the given division of areas based on an alternating direction method of multipliers (ADMM) algorithm, which is greatly simplified by exploiting the structure of active distribution networks (ADNs). The problem is solved for each area with very little interchange of boundary information between neighboring areas. The standard ADMM algorithm is extended using a varying penalty parameter to improve convergence. The validity of the method is demonstrated via numerical simulations on an IEEE 33-node distribution network, a PG&E 69-node distribution system, and an extended 137-node system.

by <a href="">Weiye Zheng</a>, <a href="">Wenchuan Wu</a>, <a href="">Boming Zhang</a>, <a href="">Hongbin Sun</a>, <a href="">Liu Yibing</a> at July 21, 2014 01:30 AM

To Reach or not to Reach? Efficient Algorithms for Total-Payoff Games. (arXiv:1407.5030v1 [cs.GT])

Quantitative games are two-player zero-sum games played on directed weighted graphs. We consider variants of usual payoff functions: total-payoff, mean-payoff, discounted-payoff, where we add a reachability objective, i.e., Player 1 wants to reach a target while minimising his payoff. In the mean-payoff and discounted-payoff cases, we show that reachability and classical versions of the game are inter-reducible. In the case of reachability total-payoff, we introduce an efficient value iteration algorithm to compute the values and optimal strategies (when they exist), that runs in pseudo-polynomial time. This contribution allows us to derive the first (to our knowledge) value iteration algorithm for classical total-payoff games, running in pseudo-polynomial time.

by <a href="">Thomas Brihaye</a>, <a href="">Gilles Geeraerts</a>, <a href="">Axel Haddad</a>, <a href="">Benjamin Monmege</a> at July 21, 2014 01:30 AM

Causal Non-Linear Financial Networks. (arXiv:1407.5020v1 [q-fin.ST])

In our previous study we have presented an approach to studying lead--lag effect in financial markets using information and network theories. Methodology presented there, as well as previous studies using Pearson's correlation for the same purpose, approached the concept of lead--lag effect in a naive way. In this paper we further investigate the lead--lag effect in financial markets, this time treating them as causal effects. To incorporate causality in a manner consistent with our previous study, that is including non-linear interdependencies, we base this study on a generalisation of Granger causality in the form of transfer entropy, or equivalently a special case of conditional (partial) mutual information. This way we are able to produce networks of stocks, where directed links represent causal relationships for a specific time lag. We apply this procedure to stocks belonging to the NYSE 100 index for various time lags, to investigate the short-term causality on this market, and to comment on the resulting Bonferroni networks.

by <a href="">Pawe&#x142; Fiedor</a> at July 21, 2014 01:30 AM

A Framework for HI Spectral Source Finding Using Distributed-Memory Supercomputing. (arXiv:1407.4958v1 [astro-ph.IM])

The latest generation of radio astronomy interferometers will conduct all sky surveys with data products consisting of petabytes of spectral line data. Traditional approaches to identifying and parameterising the astrophysical sources within this data will not scale to datasets of this magnitude, since the performance of workstations will not keep up with the real-time generation of data. For this reason, it is necessary to employ high performance computing systems consisting of a large number of processors connected by a high-bandwidth network. In order to make use of such supercomputers substantial modifications must be made to serial source finding code. To ease the transition, this work presents the Scalable Source Finder Framework, a framework providing storage access, networking communication and data composition functionality, which can support a wide range of source finding algorithms provided they can be applied to subsets of the entire image. Additionally, the Parallel Gaussian Source Finder was implemented using SSoFF, utilising Gaussian filters, thresholding, and local statistics. PGSF was able to search on a 256GB simulated dataset in under 24 minutes, significantly less than the 8 to 12 hour observation that would generate such a dataset.

by <a href="">Stefan Westerlund</a>, <a href="">Christopher Harris</a> at July 21, 2014 01:30 AM

Proceedings 3rd Workshop on Synthesis. (arXiv:1407.4937v1 [cs.LO])

The idea of synthesis, i.e., the process of automatically computing implementations from their specifications, has recently gained a lot of momentum in the contexts of software engineering and reactive system design. While it is widely believed that, due to complexity/undecidability issues, synthesis cannot completely replace manual engineering, it can assist the process of designing the intricate pieces of code that most programmers find challenging, or help with orchestrating tasks in reactive environments. The SYNT workshop aims to bring together researchers interested in synthesis to discuss and present ongoing and mature work on all aspects of automated synthesis and its applications.

The third iteration of the workshop took place in Vienna, Austria, and was co-located with the 26th International Conference on Computer Aided Verification, held in the context of the Vienna Summer of Logic in July 2014. The workshop included eight contributed talks and four invited talks. In addition, it featured a special session about the Syntax-Guided Synthesis Competition (SyGuS) and the SyntComp Synthesis competition.

by <a href="">Krishnendu Chatterjee</a> (IST Austria), <a href="">R&#xfc;diger Ehlers</a> (University of Bremen), <a href="">Susmit Jha</a> (Intel Strategic CAD Lab) at July 21, 2014 01:30 AM

Your Voice Assistant is Mine: How to Abuse Speakers to Steal Information and Control Your Phone. (arXiv:1407.4923v1 [cs.CR])

Previous research about sensor based attacks on Android platform focused mainly on accessing or controlling over sensitive device components, such as camera, microphone and GPS. These approaches get data from sensors directly and need corresponding sensor invoking permissions.

This paper presents a novel approach (GVS-Attack) to launch permission bypassing attacks from a zero permission Android application (VoicEmployer) through the speaker. The idea of GVS-Attack utilizes an Android system built-in voice assistant module -- Google Voice Search. Through Android Intent mechanism, VoicEmployer triggers Google Voice Search to the foreground, and then plays prepared audio files (like "call number 1234 5678") in the background. Google Voice Search can recognize this voice command and execute corresponding operations. With ingenious designs, our GVS-Attack can forge SMS/Email, access privacy information, transmit sensitive data and achieve remote control without any permission.

Also we found a vulnerability of status checking in Google Search app, which can be utilized by GVS-Attack to dial arbitrary numbers even when the phone is securely locked with password. A prototype of VoicEmployer has been implemented to demonstrate the feasibility of GVS-Attack in real world. In theory, nearly all Android devices equipped with Google Services Framework can be affected by GVS-Attack. This study may inspire application developers and researchers rethink that zero permission doesn't mean safety and the speaker can be treated as a new attack surface.

by <a href="">Wenrui Diao</a>, <a href="">Xiangyu Liu</a>, <a href="">Zhe Zhou</a>, <a href="">Kehuan Zhang</a> at July 21, 2014 01:30 AM

Integrating R and Hadoop for Big Data Analysis. (arXiv:1407.4908v1 [cs.DC])

Analyzing and working with big data could be very diffi cult using classical means like relational database management systems or desktop software packages for statistics and visualization. Instead, big data requires large clusters with hundreds or even thousands of computing nodes. Offi cial statistics is increasingly considering big data for deriving new statistics because big data sources could produce more relevant and timely statistics than traditional sources. One of the software tools successfully and wide spread used for storage and processing of big data sets on clusters of commodity hardware is Hadoop. Hadoop framework contains libraries, a distributed fi le-system (HDFS), a resource-management platform and implements a version of the MapReduce programming model for large scale data processing. In this paper we investigate the possibilities of integrating Hadoop with R which is a popular software used for statistical computing and data visualization. We present three ways of integrating them: R with Streaming, Rhipe and RHadoop and we emphasize the advantages and disadvantages of each solution.

by <a href="">Bogdan Oancea</a>, <a href="">Raluca Mariana Dragoescu</a> at July 21, 2014 01:30 AM

ADHA: Automatic Data layout framework for Heterogeneous Architectures. (arXiv:1407.4859v1 [cs.DC])

Data layouts play a crucial role in determining the performance of a given application running on a given architecture. Existing parallel programming frameworks for both multicore and heterogeneous systems leave the onus of selecting a data layout to the programmer. Therefore, shifting the burden of data layout selection to optimizing compilers can greatly enhance programmer productivity and application performance. In this work, we introduce {\ADHA}: a two-level hierarchal formulation of the data layout problem for modern heterogeneous architectures. We have created a reference implementation of ADHA in the Heterogeneous Habanero-C (H2C) parallel programming system. ADHA shows significant performance benefits of up to 6.92$\times$ compared to manually specified layouts for two benchmark programs running on a CPU+GPU heterogeneous platform.

by <a href="">Deepak Majeti</a>, <a href="">Kuldeep S. Meel</a>, <a href="">Rajkishore Barik</a>, <a href="">Vivek Sarkar</a> at July 21, 2014 01:30 AM

Partial Quantifier Elimination. (arXiv:1407.4835v1 [cs.LO])

We consider the problem of Partial Quantifier Elimination (PQE). Given formula exists(X)[F(X,Y) & G(X,Y)], where F, G are in conjunctive normal form, the PQE problem is to find a formula F*(Y) such that F* & exists(X)[G] is logically equivalent to exists(X)[F & G]. We solve the PQE problem by generating and adding to F clauses over the free variables that make the clauses of F with quantified variables redundant. The traditional Quantifier Elimination problem (QE) is a special case of PQE where G is empty so all clauses of the input formula with quantified variables need to be made redundant. The importance of PQE is twofold. First, many problems are more naturally formulated in terms of PQE rather than QE. Second, in many cases PQE can be solved more efficiently than QE. We describe a PQE algorithm based on the machinery of dependency sequents and give experimental results showing the promise of PQE.

by <a href="">Eugene Goldberg</a>, <a href="">Panagiotis Manolios</a> at July 21, 2014 01:30 AM

The Information Service Evaluation (ISE) Model. (arXiv:1407.4831v1 [cs.CY])

Information services are an inherent part of our everyday life. Especially since ubiquitous cities are being developed all over the world their number is increasing even faster. They aim at facilitating the production of information and the access to the needed information and are supposed to make life easier. Until today many different evaluation models (among others, TAM, TAM 2, TAM 3, UTAUT and MATH) have been developed to measure the quality and acceptance of these services. Still, they only consider subareas of the whole concept that represents an information service. As a holistic and comprehensive approach, the ISE Model studies five dimensions that influence adoption, use, impact and diffusion of the information service: information service quality, information user, information acceptance, information environment and time. All these aspects have a great impact on the final grading and of the success (or failure) of the service. Our model combines approaches, which study subjective impressions of users (e.g., the perceived service quality), and user-independent, more objective approaches (e.g., the degree of gamification of a system). Furthermore, we adopt results of network economics, especially the Success breeds success-principle.

by <a href="">Laura Schumann</a>, <a href="">Wolfgang G. Stock</a> at July 21, 2014 01:30 AM

Mixed-integer Quadratic Programming is in NP. (arXiv:1407.4798v1 [cs.DM])

Mixed-integer quadratic programming is the problem of optimizing a quadratic function over points in a polyhedral set where some of the components are restricted to be integral. In this paper, we prove that the decision version of mixed-integer quadratic programming is in NP, thereby showing that it is NP-complete. This is established by showing that if the decision version of mixed-integer quadratic programming is feasible, then there exists a solution of polynomial size. This result generalizes and unifies classical results that quadratic programming is in NP and integer linear programming is in NP.

by <a href="">Alberto Del Pia</a>, <a href="">Santanu S. Dey</a>, <a href="">Marco Molinaro</a> at July 21, 2014 01:30 AM

Query Complexity of Approximate Nash Equilibria. (arXiv:1306.6686v3 [cs.GT] UPDATED)

We study the query complexity of approximate notions of Nash equilibrium in games with a large number of players $n$. Our main result states that for $n$-player binary-action games and for constant $\varepsilon$, the query complexity of an $\varepsilon$-well-supported Nash equilibrium is exponential in $n$. One of the consequences of this result is an exponential lower bound on the rate of convergence of adaptive dynamics to approxiamte Nash equilibrium.

by <a href="">Yakov Babichenko</a> at July 21, 2014 01:30 AM



`return` in a scala function literal

I'm trying to do something like this:

var fun : (Int,Int) => Double = (a,b) =>
  // do something
  return 1.0

However, my IDE complaints with Return statement outside method definition. So how do I explicitly give a return statement in a function literal in scala?

by OneZero at July 21, 2014 12:59 AM

How does Spray's "jsonFormatX" work?

I can convert my case class to json using the Spray-Json library by doing this:

case class Foo(a1:Int, a2:Int)
    implicit val fooFormat = jsonFormat2(Foo) // notice the "2" at the end

Looking through the source code, I don't see anything like def jsonFormat1... def jsonFormat2... jsonFormat3.. and so on... How are the developers of this library doing this type of thing? See here

by goo at July 21, 2014 12:54 AM

Planet Theory

Improved ancestry labeling scheme for trees

Authors: Søren Dahlgaard, Mathias Bæk Tejs Knudsen, Noy Rotbart
Download: PDF
Abstract: We consider ancestry labeling schemes: Given a rooted tree T, assign a binary string (label) to each node, such that given the labels of any two nodes, one can determine whether the first is an ancestor of the second in T.

Recently, Fraigniaud and Korman [STOC'10] showed that such labels can be assigned using log n + O(loglog n) bits per label, solving a long standing open question and matching a lower bound of log n + Omega(loglog n) due to Alstrup et al. [SODA'03]. In this paper we present an alternative ancestry labeling scheme using log n + 2loglog n + O(1) bits. Similar to the previously known schemes, our scheme relies on intervals and can encode any tree in O(n) time. Rather than using complicated decompositions, our scheme uses approximate interval sizes, which gives a clean algorithm that is simple enough to be taught and implemented.

July 21, 2014 12:40 AM