hadoop - Sqoop installation export and import from postgresql -

i v'e installed sqoop , testing . tried export data hdfs postgresql using sqoop. when run it throws following exception : java.io.ioexception: can't export data, please check task tracker logs . think there may have been problem in installation.

the file content :

ustnu 45 mb1ba 0 gnbco 76 izp10 39 b2aoo 45 si7eg 93 5sc4k 60 2ihfv 2 u2a48 16 yvy6r 51 lnhsv 26 mz2yn 65 80gp3 43 wk5ag 85 vufyp 93 p077j 94 f1oj5 11 lxjkg 72 0h7np 99 dk406 25 g4krp 76 fw3u0 80 6ld59 1 07khx 91 f1s88 72 bnb0v 85 a2qm7 79 z6cat 81 0m3do 23 m0s09 44 kivwd 13 gnud0 78 um93a 20 19bhv 75 4of3s 75 5hfen 16 

this posgres table:

    table "public.mysort"  column |  type   | modifiers  --------+---------+-----------  name   | text    |   marks  | integer |  

the sqoop command is:

sqoop export --connect jdbc:postgresql://localhost/testdb --username akshay --password akshay --table mysort -m 1 --export-dir mysort/input 

followed error:

warning: /usr/lib/hcatalog not exist! hcatalog jobs fail. please set $hcat_home root of hcatalog installation. 14/06/11 18:28:06 warn tool.basesqooptool: setting password on command-line insecure. consider using -p instead. 14/06/11 18:28:06 info manager.sqlmanager: using default fetchsize of 1000 14/06/11 18:28:06 info tool.codegentool: beginning code generation 14/06/11 18:28:06 info manager.sqlmanager: executing sql statement: select t.* "mysort" t limit 1 14/06/11 18:28:06 info orm.compilationmanager: hadoop_mapred_home /usr/local/hadoop note: /tmp/sqoop-hduser/compile/0402ad4b5cf7980040264af35de406cb/mysort.java uses or overrides deprecated api. note: recompile -xlint:deprecation details. 14/06/11 18:28:07 info orm.compilationmanager: writing jar file: /tmp/sqoop-hduser/compile/0402ad4b5cf7980040264af35de406cb/mysort.jar 14/06/11 18:28:07 info mapreduce.exportjobbase: beginning export of mysort slf4j: class path contains multiple slf4j bindings. slf4j: found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/staticloggerbinder.class] slf4j: found binding in [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/staticloggerbinder.class] slf4j: see http://www.slf4j.org/codes.html#multiple_bindings explanation. slf4j: actual binding of type [org.slf4j.impl.log4jloggerfactory] java hotspot(tm) 64-bit server vm warning: have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 might have disabled stack guard. vm try fix stack guard now. it's highly recommended fix library 'execstack -c <libfile>', or link '-z noexecstack'. 14/06/11 18:28:22 warn util.nativecodeloader: unable load native-hadoop library platform... using builtin-java classes applicable 14/06/11 18:28:22 info configuration.deprecation: mapred.jar deprecated. instead, use mapreduce.job.jar 14/06/11 18:28:23 info configuration.deprecation: mapred.reduce.tasks.speculative.execution deprecated. instead, use mapreduce.reduce.speculative 14/06/11 18:28:23 info configuration.deprecation: mapred.map.tasks.speculative.execution deprecated. instead, use mapreduce.map.speculative 14/06/11 18:28:23 info configuration.deprecation: mapred.map.tasks deprecated. instead, use mapreduce.job.maps 14/06/11 18:28:23 info client.rmproxy: connecting resourcemanager @ /0.0.0.0:8032 14/06/11 18:28:24 info input.fileinputformat: total input paths process : 1 14/06/11 18:28:24 info input.fileinputformat: total input paths process : 1 14/06/11 18:28:25 info mapreduce.jobsubmitter: number of splits:1 14/06/11 18:28:25 info mapreduce.jobsubmitter: submitting tokens job: job_1402488523460_0003 14/06/11 18:28:25 info impl.yarnclientimpl: submitted application application_1402488523460_0003 14/06/11 18:28:25 info mapreduce.job: url track job: http://localhost:8088/proxy/application_1402488523460_0003/ 14/06/11 18:28:25 info mapreduce.job: running job: job_1402488523460_0003 14/06/11 18:28:46 info mapreduce.job: job job_1402488523460_0003 running in uber mode : false 14/06/11 18:28:46 info mapreduce.job:  map 0% reduce 0% 14/06/11 18:29:04 info mapreduce.job: task id : attempt_1402488523460_0003_m_000000_0, status : failed error: java.io.ioexception: can't export data, please check task tracker logs     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:112)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:39)     @ org.apache.hadoop.mapreduce.mapper.run(mapper.java:145)     @ org.apache.sqoop.mapreduce.autoprogressmapper.run(autoprogressmapper.java:64)     @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:764)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:340)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:168)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:163) caused by: java.util.nosuchelementexception     @ java.util.arraylist$itr.next(arraylist.java:839)     @ mysort.__loadfromfields(mysort.java:198)     @ mysort.parse(mysort.java:147)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:83)     ... 10 more  14/06/11 18:29:23 info mapreduce.job: task id : attempt_1402488523460_0003_m_000000_1, status : failed error: java.io.ioexception: can't export data, please check task tracker logs     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:112)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:39)     @ org.apache.hadoop.mapreduce.mapper.run(mapper.java:145)     @ org.apache.sqoop.mapreduce.autoprogressmapper.run(autoprogressmapper.java:64)     @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:764)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:340)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:168)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:163) caused by: java.util.nosuchelementexception     @ java.util.arraylist$itr.next(arraylist.java:839)     @ mysort.__loadfromfields(mysort.java:198)     @ mysort.parse(mysort.java:147)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:83)     ... 10 more  14/06/11 18:29:42 info mapreduce.job: task id : attempt_1402488523460_0003_m_000000_2, status : failed error: java.io.ioexception: can't export data, please check task tracker logs     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:112)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:39)     @ org.apache.hadoop.mapreduce.mapper.run(mapper.java:145)     @ org.apache.sqoop.mapreduce.autoprogressmapper.run(autoprogressmapper.java:64)     @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:764)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:340)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:168)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:163) caused by: java.util.nosuchelementexception     @ java.util.arraylist$itr.next(arraylist.java:839)     @ mysort.__loadfromfields(mysort.java:198)     @ mysort.parse(mysort.java:147)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:83)     ... 10 more  14/06/11 18:30:03 info mapreduce.job:  map 100% reduce 0% 14/06/11 18:30:03 info mapreduce.job: job job_1402488523460_0003 failed state failed due to: task failed task_1402488523460_0003_m_000000 job failed tasks failed. failedmaps:1 failedreduces:0  14/06/11 18:30:03 info mapreduce.job: counters: 9     job counters          failed map tasks=4         launched map tasks=4         other local map tasks=3         data-local map tasks=1         total time spent maps in occupied slots (ms)=69336         total time spent reduces in occupied slots (ms)=0         total time spent map tasks (ms)=69336         total vcore-seconds taken map tasks=69336         total megabyte-seconds taken map tasks=71000064 14/06/11 18:30:03 warn mapreduce.counters: group filesystemcounters deprecated. use org.apache.hadoop.mapreduce.filesystemcounter instead 14/06/11 18:30:03 info mapreduce.exportjobbase: transferred 0 bytes in 100.1476 seconds (0 bytes/sec) 14/06/11 18:30:03 warn mapreduce.counters: group org.apache.hadoop.mapred.task$counter deprecated. use org.apache.hadoop.mapreduce.taskcounter instead 14/06/11 18:30:03 info mapreduce.exportjobbase: exported 0 records. 14/06/11 18:30:03 error tool.exporttool: error during export: export job failed! 

this log file :

2014-06-11 17:54:37,601 warn [main] org.apache.hadoop.conf.configuration: job.xml:an attempt override final parameter: mapreduce.job.end-notification.max.retry.interval;  ignoring. 2014-06-11 17:54:37,602 warn [main] org.apache.hadoop.conf.configuration: job.xml:an attempt override final parameter: mapreduce.job.end-notification.max.attempts;  ignoring. 2014-06-11 17:54:52,678 warn [main] org.apache.hadoop.util.nativecodeloader: unable load native-hadoop library platform... using builtin-java classes applicable 2014-06-11 17:54:52,777 info [main] org.apache.hadoop.metrics2.impl.metricsconfig: loaded properties hadoop-metrics2.properties 2014-06-11 17:54:52,846 info [main] org.apache.hadoop.metrics2.impl.metricssystemimpl: scheduled snapshot period @ 10 second(s). 2014-06-11 17:54:52,847 info [main] org.apache.hadoop.metrics2.impl.metricssystemimpl: maptask metrics system started 2014-06-11 17:54:52,855 info [main] org.apache.hadoop.mapred.yarnchild: executing tokens: 2014-06-11 17:54:52,855 info [main] org.apache.hadoop.mapred.yarnchild: kind: mapreduce.job, service: job_1402488523460_0002, ident: (org.apache.hadoop.mapreduce.security.token.jobtokenidentifier@971d0d8) 2014-06-11 17:54:52,901 info [main] org.apache.hadoop.mapred.yarnchild: sleeping 0ms before retrying again. got null now. 2014-06-11 17:54:53,165 info [main] org.apache.hadoop.mapred.yarnchild: mapreduce.cluster.local.dir child: /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1402488523460_0002 2014-06-11 17:54:53,249 warn [main] org.apache.hadoop.conf.configuration: job.xml:an attempt override final parameter: mapreduce.job.end-notification.max.retry.interval;  ignoring. 2014-06-11 17:54:53,249 warn [main] org.apache.hadoop.conf.configuration: job.xml:an attempt override final parameter: mapreduce.job.end-notification.max.attempts;  ignoring. 2014-06-11 17:54:53,393 info [main] org.apache.hadoop.conf.configuration.deprecation: session.id deprecated. instead, use dfs.metrics.session-id 2014-06-11 17:54:53,689 info [main] org.apache.hadoop.mapred.task:  using resourcecalculatorprocesstree : [ ] 2014-06-11 17:54:53,899 info [main] org.apache.hadoop.mapred.maptask: processing split: paths:/user/hduser/mysort/input/data.txt:0+891082 2014-06-11 17:54:53,904 info [main] org.apache.hadoop.conf.configuration.deprecation: map.input.file deprecated. instead, use mapreduce.map.input.file 2014-06-11 17:54:53,904 info [main] org.apache.hadoop.conf.configuration.deprecation: map.input.start deprecated. instead, use mapreduce.map.input.start 2014-06-11 17:54:53,904 info [main] org.apache.hadoop.conf.configuration.deprecation: map.input.length deprecated. instead, use mapreduce.map.input.length 2014-06-11 17:54:54,028 error [main] org.apache.sqoop.mapreduce.textexportmapper:  2014-06-11 17:54:54,028 error [main] org.apache.sqoop.mapreduce.textexportmapper: exception raised during data export 2014-06-11 17:54:54,028 error [main] org.apache.sqoop.mapreduce.textexportmapper:  2014-06-11 17:54:54,028 error [main] org.apache.sqoop.mapreduce.textexportmapper: exception:  java.util.nosuchelementexception     @ java.util.arraylist$itr.next(arraylist.java:839)     @ mysort.__loadfromfields(mysort.java:198)     @ mysort.parse(mysort.java:147)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:83)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:39)     @ org.apache.hadoop.mapreduce.mapper.run(mapper.java:145)     @ org.apache.sqoop.mapreduce.autoprogressmapper.run(autoprogressmapper.java:64)     @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:764)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:340)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:168)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:163) 2014-06-11 17:54:54,030 error [main] org.apache.sqoop.mapreduce.textexportmapper: on input: ustnu 45 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper: on input file: hdfs://localhost:9000/user/hduser/mysort/input/data.txt 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper: @ position 0 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper:  2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper: processing split: 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper: paths:/user/hduser/mysort/input/data.txt:0+891082 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper:  2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper: issue might not caused current input 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper: due batching nature of export. 2014-06-11 17:54:54,031 error [main] org.apache.sqoop.mapreduce.textexportmapper:  2014-06-11 17:54:54,032 info [thread-12] org.apache.sqoop.mapreduce.autoprogressmapper: auto-progress thread finished. keepgoing=false 2014-06-11 17:54:54,033 warn [main] org.apache.hadoop.security.usergroupinformation: priviledgedactionexception as:hduser (auth:simple) cause:java.io.ioexception: can't export data, please check task tracker logs 2014-06-11 17:54:54,033 warn [main] org.apache.hadoop.mapred.yarnchild: exception running child : java.io.ioexception: can't export data, please check task tracker logs     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:112)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:39)     @ org.apache.hadoop.mapreduce.mapper.run(mapper.java:145)     @ org.apache.sqoop.mapreduce.autoprogressmapper.run(autoprogressmapper.java:64)     @ org.apache.hadoop.mapred.maptask.runnewmapper(maptask.java:764)     @ org.apache.hadoop.mapred.maptask.run(maptask.java:340)     @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:168)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1548)     @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:163) caused by: java.util.nosuchelementexception     @ java.util.arraylist$itr.next(arraylist.java:839)     @ mysort.__loadfromfields(mysort.java:198)     @ mysort.parse(mysort.java:147)     @ org.apache.sqoop.mapreduce.textexportmapper.map(textexportmapper.java:83)     ... 10 more  2014-06-11 17:54:54,037 info [main] org.apache.hadoop.mapred.task: runnning cleanup task 

any in resolving issue appreciated.

here complete procedure installation , import , export commands sqoop. hope may helpful one. 1 tried , tested me , works.

download : apache.mirrors.tds.net/sqoop/1.4.4/sqoop-1.4.4.bin__hadoop-2.0.4-alpha.tar.gz  sudo mv sqoop-1.4.4.bin__hadoop-2.0.4-alpha.tar.gz /usr/lib/sqoop  copy paste followingtwo lines in .bashrc export sqoop_home=/usr/lib/sqoop export path=$path:$sqoop_home/bin  go /usr/lib/sqoop/conf folder , copy sqoop-env-template.sh new file sqoop-env.sh , modify export hadoop_home ,hbase_home,etc installation directory  download postgresql conector jar file jdbc.postgresql.org/download/postgresql-9.3-1101.jdbc41.jar  create directory manager.d in sqoop/conf/  create file postgresql in conf/  , add following line in org.postgresql.driver=/usr/lib/sqoop/lib/postgresql-9.3-1101.jdbc41.jar  name connector.jar file accordingly  

for export

create user in postgres: createuser -p -s -e ace enter password new role: ace enter again: ace   create database testdb owner ace tablespace ace;  create table stud1(id int,name text);  create file student.txt add lines such as: 1,ace 2,iloveapis  hadoop fs -put student.txt   sqoop export --connect jdbc:postgresql://localhost:5432/testdb --username ace --password ace --table stud1 -m 1 --export-dir student.txt  check in postgres: select * stud1; 

for import:

sqoop import --connect jdbc:postgresql://localhost:5432/testdb --username akshay --password akshay --table stud1 --m 1  hadoop fs -ls -r stud1 

expected output:

-rw-r--r--   1 hduser supergroup          0 2014-06-13 18:10 stud1/_success -rw-r--r--   1 hduser supergroup         21 2014-06-13 18:10 stud1/part-m-00000   hadoop fs -cat stud1/part-m-00000 

expected output:

1,ace 2,iloveapis   hadoop fs -copytolocal stud1/part-m-00000 $home/imported_data.txt