scala - How to use GzipCodec or BZip2Codec for shuffle spill compression with Spark shell -

-

so when start spark shell -dspark.io.compression.codec=org.apache.hadoop.io.compress.gzipcodec get.

due space limitations on our cluster, want use more aggressive compression codec how use bzip2codec , avoid exception? possible?

java.lang.nosuchmethodexception: org.apache.hadoop.io.compress.bzip2codec.<init>(org.apache.spark.sparkconf)     @ java.lang.class.getconstructor0(class.java:2810)     @ java.lang.class.getconstructor(class.java:1718)     @ org.apache.spark.io.compressioncodec$.createcodec(compressioncodec.scala:48)     @ org.apache.spark.io.compressioncodec$.createcodec(compressioncodec.scala:42)     @ org.apache.spark.broadcast.httpbroadcast$.initialize(httpbroadcast.scala:106)     @ org.apache.spark.broadcast.httpbroadcastfactory.initialize(httpbroadcast.scala:70)     @ org.apache.spark.broadcast.broadcastmanager.initialize(broadcast.scala:81)     @ org.apache.spark.broadcast.broadcastmanager.<init>(broadcast.scala:68)     @ org.apache.spark.sparkenv$.create(sparkenv.scala:175)     @ org.apache.spark.sparkcontext.<init>(sparkcontext.scala:141)     @ org.apache.spark.repl.sparkiloop.createsparkcontext(sparkiloop.scala:956)     @ $iwc$$iwc.<init>(<console>:8)     @ $iwc.<init>(<console>:14)     @ <init>(<console>:16)     @ .<init>(<console>:20)     @ .<clinit>(<console>)     @ .<init>(<console>:7)     @ .<clinit>(<console>)     @ $print(<console>)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:606)     @ org.apache.spark.repl.sparkimain$readevalprint.call(sparkimain.scala:772)     @ org.apache.spark.repl.sparkimain$request.loadandrun(sparkimain.scala:1040)     @ org.apache.spark.repl.sparkimain.loadandrunreq$1(sparkimain.scala:609)     @ org.apache.spark.repl.sparkimain.interpret(sparkimain.scala:640)     @ org.apache.spark.repl.sparkimain.interpret(sparkimain.scala:604)     @ org.apache.spark.repl.sparkiloop.reallyinterpret$1(sparkiloop.scala:795)     @ org.apache.spark.repl.sparkiloop.interpretstartingwith(sparkiloop.scala:840)     @ org.apache.spark.repl.sparkiloop.command(sparkiloop.scala:752)     @ org.apache.spark.repl.sparkiloopinit$$anonfun$initializespark$1.apply(sparkiloopinit.scala:119)     @ org.apache.spark.repl.sparkiloopinit$$anonfun$initializespark$1.apply(sparkiloopinit.scala:118)     @ org.apache.spark.repl.sparkimain.bequietduring(sparkimain.scala:258)     @ org.apache.spark.repl.sparkiloopinit$class.initializespark(sparkiloopinit.scala:118)     @ org.apache.spark.repl.sparkiloop.initializespark(sparkiloop.scala:55)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1$$anonfun$apply$mcz$sp$5.apply$mcv$sp(sparkiloop.scala:912)     @ org.apache.spark.repl.sparkiloopinit$class.runthunks(sparkiloopinit.scala:140)     @ org.apache.spark.repl.sparkiloop.runthunks(sparkiloop.scala:55)     @ org.apache.spark.repl.sparkiloopinit$class.postinitialization(sparkiloopinit.scala:102)     @ org.apache.spark.repl.sparkiloop.postinitialization(sparkiloop.scala:55)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1.apply$mcz$sp(sparkiloop.scala:929)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1.apply(sparkiloop.scala:883)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1.apply(sparkiloop.scala:883)     @ scala.tools.nsc.util.scalaclassloader$.savingcontextloader(scalaclassloader.scala:135)     @ org.apache.spark.repl.sparkiloop.process(sparkiloop.scala:883)     @ org.apache.spark.repl.sparkiloop.process(sparkiloop.scala:981)     @ org.apache.spark.repl.main$.main(main.scala:31)     @ org.apache.spark.repl.main.main(main.scala)

running hadoop checknative:

14/06/13 17:41:24 info bzip2.bzip2factory: loaded & initialized native-bzip2 library system-native 14/06/13 17:41:24 info zlib.zlibfactory: loaded & initialized native-zlib library native library checking: hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 zlib:   true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1 lz4:    true revision:99 bzip2:  true /lib/x86_64-linux-gnu/libbz2.so.1

after starting shell simple import statement should allow use bzip2 - compression , how want, although still unclear on question.

import org.apache.hadoop.io.compress.bzip2codec

and when starting shell, executed following command , gave me no error, try check if you're entering correctly.

./spark-shell -d spark.io.compression.codec=org.apache.hadoop.io.compress.bzip2codec

this of-course in "~/spark/bin/"

similarly gzip:

./spark-shell -d spark.io.compression.codec=org.apache.hadoop.io.compress.gzipcodec

Comments

Post a Comment

Popular posts from this blog

database - VFP Grid + SQL server 2008 - grid not showing correctly -

-
i having problem, have grid called "gridbasica" , this: thisform.gridbase.recordsource = "sgviemp" sgviemp cursor made executing select sqlserver command sqlexec. when want modify registry, go form , the update there. so, when come form grid, grid isnt grid anymore, big white rectangle. reading around said: delete recordsource , asingt again. thisform.gridbase.recordsource = "" *--i sqlexe here thisform.gridbase.recordsource = "sgviemp" i , works, but, when click button modify again cursos updated modify form doesnt load fields did first time. could me out that? thanks in advance. --------------- edit ------------------ i have been doing recommended: lnselect = select(0) n = sqlexec(thisform.conexion,"select emp_ccodigo codigo, emp_cnombre 'nombre o razon social', emp_cnumrif 'r.i.f' sgviempr","sgviemp1") set echo on suspend select sgviemp zap in sgviemp *-- browse *--zaps correct
Read more

jquery - Set jPicker field to empty value -

-
im using jpicker color selector in form. have clear value set window/icon 'no color selected' state. i've tried following: $(':input[name="color"]').val(''); //access value directly. (does not affect jpicker) 2: $.jpicker.list[0].color.active.val('hex', '', this); //set color black , input value 000000, instead of empty string 3: $.jpicker.list[0].color.active.val(null, '', this); //does nothing none of 3 methods seems behave properly. in advance. this works me :) $.jpicker.list[0].color.active.val('v', null); here example: http://jsfiddle.net/3up7n/
Read more

.htaccess - htaccess convert request to clean url and add slash at the end of the url -

-
ok , searched more 4 days , more 20 posts here in stach or in websites without kind of success first here htaccess file rewriteengine on rewriterule ^([a-za-z0-9_]+)/?$ index.php?page=$1 [nc] rewriterule ^([a-za-z0-9_]+)\/([a-za-z0-9_]+)\/?$ index.php?page=$1&action=$2 [nc] i tried many codes 1 worked me can call request , works me www.example.com/pagename => www.example.com/index.php?page=pagename here questions , please provide me information "in detail" understand code well 1- how convert request automatically www.example.com/index.php?page=pagename www.example.com/pagename 2- if tried add slash @ end shows me page without images or styling mean www.example.com/pagename => works www.example.com/pagename/ => doesn't work that's :) for first part need 301 rule this: rewritecond %{the_request} /index\.php\?page=([^\s&]+) [nc] rewriterule ^ /%1? [r=301,l] for second part use absolute path in css, js, images files r
Read more