scala - How to use GzipCodec or BZip2Codec for shuffle spill compression with Spark shell -

-

so when start spark shell -dspark.io.compression.codec=org.apache.hadoop.io.compress.gzipcodec get.

due space limitations on our cluster, want use more aggressive compression codec how use bzip2codec , avoid exception? possible?

java.lang.nosuchmethodexception: org.apache.hadoop.io.compress.bzip2codec.<init>(org.apache.spark.sparkconf)     @ java.lang.class.getconstructor0(class.java:2810)     @ java.lang.class.getconstructor(class.java:1718)     @ org.apache.spark.io.compressioncodec$.createcodec(compressioncodec.scala:48)     @ org.apache.spark.io.compressioncodec$.createcodec(compressioncodec.scala:42)     @ org.apache.spark.broadcast.httpbroadcast$.initialize(httpbroadcast.scala:106)     @ org.apache.spark.broadcast.httpbroadcastfactory.initialize(httpbroadcast.scala:70)     @ org.apache.spark.broadcast.broadcastmanager.initialize(broadcast.scala:81)     @ org.apache.spark.broadcast.broadcastmanager.<init>(broadcast.scala:68)     @ org.apache.spark.sparkenv$.create(sparkenv.scala:175)     @ org.apache.spark.sparkcontext.<init>(sparkcontext.scala:141)     @ org.apache.spark.repl.sparkiloop.createsparkcontext(sparkiloop.scala:956)     @ $iwc$$iwc.<init>(<console>:8)     @ $iwc.<init>(<console>:14)     @ <init>(<console>:16)     @ .<init>(<console>:20)     @ .<clinit>(<console>)     @ .<init>(<console>:7)     @ .<clinit>(<console>)     @ $print(<console>)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:606)     @ org.apache.spark.repl.sparkimain$readevalprint.call(sparkimain.scala:772)     @ org.apache.spark.repl.sparkimain$request.loadandrun(sparkimain.scala:1040)     @ org.apache.spark.repl.sparkimain.loadandrunreq$1(sparkimain.scala:609)     @ org.apache.spark.repl.sparkimain.interpret(sparkimain.scala:640)     @ org.apache.spark.repl.sparkimain.interpret(sparkimain.scala:604)     @ org.apache.spark.repl.sparkiloop.reallyinterpret$1(sparkiloop.scala:795)     @ org.apache.spark.repl.sparkiloop.interpretstartingwith(sparkiloop.scala:840)     @ org.apache.spark.repl.sparkiloop.command(sparkiloop.scala:752)     @ org.apache.spark.repl.sparkiloopinit$$anonfun$initializespark$1.apply(sparkiloopinit.scala:119)     @ org.apache.spark.repl.sparkiloopinit$$anonfun$initializespark$1.apply(sparkiloopinit.scala:118)     @ org.apache.spark.repl.sparkimain.bequietduring(sparkimain.scala:258)     @ org.apache.spark.repl.sparkiloopinit$class.initializespark(sparkiloopinit.scala:118)     @ org.apache.spark.repl.sparkiloop.initializespark(sparkiloop.scala:55)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1$$anonfun$apply$mcz$sp$5.apply$mcv$sp(sparkiloop.scala:912)     @ org.apache.spark.repl.sparkiloopinit$class.runthunks(sparkiloopinit.scala:140)     @ org.apache.spark.repl.sparkiloop.runthunks(sparkiloop.scala:55)     @ org.apache.spark.repl.sparkiloopinit$class.postinitialization(sparkiloopinit.scala:102)     @ org.apache.spark.repl.sparkiloop.postinitialization(sparkiloop.scala:55)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1.apply$mcz$sp(sparkiloop.scala:929)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1.apply(sparkiloop.scala:883)     @ org.apache.spark.repl.sparkiloop$$anonfun$process$1.apply(sparkiloop.scala:883)     @ scala.tools.nsc.util.scalaclassloader$.savingcontextloader(scalaclassloader.scala:135)     @ org.apache.spark.repl.sparkiloop.process(sparkiloop.scala:883)     @ org.apache.spark.repl.sparkiloop.process(sparkiloop.scala:981)     @ org.apache.spark.repl.main$.main(main.scala:31)     @ org.apache.spark.repl.main.main(main.scala)

running hadoop checknative:

14/06/13 17:41:24 info bzip2.bzip2factory: loaded & initialized native-bzip2 library system-native 14/06/13 17:41:24 info zlib.zlibfactory: loaded & initialized native-zlib library native library checking: hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 zlib:   true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1 lz4:    true revision:99 bzip2:  true /lib/x86_64-linux-gnu/libbz2.so.1

after starting shell simple import statement should allow use bzip2 - compression , how want, although still unclear on question.

import org.apache.hadoop.io.compress.bzip2codec

and when starting shell, executed following command , gave me no error, try check if you're entering correctly.

./spark-shell -d spark.io.compression.codec=org.apache.hadoop.io.compress.bzip2codec

this of-course in "~/spark/bin/"

similarly gzip:

./spark-shell -d spark.io.compression.codec=org.apache.hadoop.io.compress.gzipcodec

Comments

Post a Comment

Popular posts from this blog

C# random value from dictionary and tuple -

-
i have following code, , need random item contains string "region_1" dictionary. public static dictionary<int, tuple<string, int, currencytype>> itemarray = new dictionary<int, tuple<string, int, currencytype>>() { { 0xb3e, new tuple<string, int, currencytype>("region_1", 1500, currencytype.fame) } }; public static int generateitemid(string shopid) { var generateditem = itemarray.elementat(new random().next(0, itemarray.count)).key; } how select this? it's not possible efficiently... first create static random @ class level... prevent non-random behaviour if run query on short period of time... (it's seeded discrete clock) static random rnd = new random(); then: var item = itemarray.values .where(t => t.item1 == shopid) .orderby(_ => rnd.next()) .firstordefault()
Read more

cgi - How do I interpret URLs without extension as files rather than missing directories in nginx? -

-
so i'm trying install zoneminder , have run under nginx, has few compiled files need run using fast-cgi. these files lack extension. if file has extension, there no issue , nginx interprets file , return 404 if can't find it. if has no extension, assume it's directory , return 404 when can't find sort of index page. here have now: # security camera domain. server { listen 888; server_name mydomain.com; root /srv/http/zm; # enable php support. include php.conf; location / { index index.html index.htm index.php; } # enable cgi support. location ~ ^/cgi-bin/(.+)$ { alias /srv/cgi-bin/zm/; fastcgi_pass unix:/run/fcgiwrap.sock; fastcgi_param script_filename $document_root/$1; include fastcgi.conf; } } the idea is, under cgi-bin directory goes through fastcgi pipe. any idea on howi can fix this? upon closer inspection, realized zoneminder has 2 cgi sc...
Read more

.htaccess - htaccess convert request to clean url and add slash at the end of the url -

-
ok , searched more 4 days , more 20 posts here in stach or in websites without kind of success first here htaccess file rewriteengine on rewriterule ^([a-za-z0-9_]+)/?$ index.php?page=$1 [nc] rewriterule ^([a-za-z0-9_]+)\/([a-za-z0-9_]+)\/?$ index.php?page=$1&action=$2 [nc] i tried many codes 1 worked me can call request , works me www.example.com/pagename => www.example.com/index.php?page=pagename here questions , please provide me information "in detail" understand code well 1- how convert request automatically www.example.com/index.php?page=pagename www.example.com/pagename 2- if tried add slash @ end shows me page without images or styling mean www.example.com/pagename => works www.example.com/pagename/ => doesn't work that's :) for first part need 301 rule this: rewritecond %{the_request} /index\.php\?page=([^\s&]+) [nc] rewriterule ^ /%1? [r=301,l] for second part use absolute path in css, js, images files r...
Read more