Spark saving to Cassandra with Scala TTL Option -


i use cassandra spark connector in scala. here sample code working save data cassandra. 

val data = rdd.map (f => new cassandrarow(indexedseq("pk", "count"), indexedseq(f._1, f._2.tostring())))  data.savetocassandra("keyspace", "table")

then, try use ttl. here sample code adding ttl.

adding import

import com.datastax.spark.connector.writer.{ttloption, writeconf}

and adding ttl savetocassandra

data.savetocassandra ("keyspace", "table", writeconf = writeconf(ttl = ttloption.constant(604800))

however, when compiled it, gets errors.

error messages

[error] bad symbolic reference. signature in ttloption.class refers term streaming [error] in package org.apache.spark not available. [error] may missing current classpath, or version on [error] classpath might incompatible version used when compiling ttloption.class. [error] 1 error found [error] (compile:compile) compilation failed [error] total time: 4 s, completed mar 31, 2015 11:55:14

build.sbt

name := "cassandra"  version := "1.0"  scalaversion := "2.10.4"  librarydependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1"  librarydependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1"  librarydependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2"

is there problem code??

this known bug in current connector relating dependency on streaming spark libraries.

https://datastax-oss.atlassian.net/browse/sparkc-113

if blocking , need workaround can include spark-streaming library dependency in build file.

librarydependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"

Comments

Post a Comment

Popular posts from this blog

cakephp - simple blog with croogo -

i not problem, made simple blog in croogo, problem is not adding or modifying or delete, because arabic language ?? function add in controller `public function admin_add() { $this->set('title_for_layout', __('add part')); if (!empty($this->request->data)) { $this->part->create(); if ($this->part->save($this->request->data)) { $this->session->setflash(__('the part has been saved'), 'default', array('class' => 'success')); $this->redirect(array('action' => 'index'));//, $this->part->id)); } else { $this->session->setflash(__('the part not saved. please, try again.'), 'default', array('class' => 'error')); } } }` => model `class part extends appmodel { public $name = 'part'; var $belongst...
Read more

How to group boxplot outliers in gnuplot -

i have large set of data points. try plot them boxplot, of outliers exact same value , represented on line beside each other. found how set horizontal distance between outliers in gnuplot boxplot , doesn't much, apparently not possible. is possible group outliers together, print 1 point , print number in brackets beside indicate how many points there are? think make more readable in graph. for information, have 3 boxplots 1 x value , times 6 in 1 graph. using gnuplot 5 , played around pointsize, doesn't reduce distance anymore. hope can help! edit: set terminal pdf set output 'dat.pdf' file0 = 'dat1.dat' file1 = 'dat2.dat' file2 = 'dat3.dat' set pointsize 0.2 set notitle set xlabel 'x' set ylabel 'y' header = system('head -1 '.file0); n = words(header) set xtics ('' 1) set [i=1:n] xtics add (word(header, i) i) set style data boxplot plot file0 using (1-0.25):1:(0.2) boxplot lw 2 lc rgb '#8b0000...
Read more

bash - Performing variable substitution in a string -

i have string contains use of variable, , i'd substitute variable's value string. right best have is #!/bin/bash foo='$name; echo bar' name="the name" expanded="$(eval echo "$foo")" echo "$expanded" which has obvious defects: prints the name bar while i'd print the name; echo bar instead of eval can bash's regex matching bash's string replacement: foo='$name; echo bar' name="the name" [[ "$foo" =~ \$([[:alnum:]]+) ]] && s="${!bash_rematch[1]}" && expanded="${foo/\$${bash_rematch[1]}/$s}" echo "$expanded" name; echo bar
Read more