i have distant hadoop cluster of machines (cloudera cdh4). , i'm trying run pig script computer. here java code so:
import org.apache.pig.exectype; import org.apache.pig.pigserver; import org.apache.pig.backend.executionengine.execexception; import org.apache.pig.data.tuple; public class testpig { public static void main(string args[]){ pigserver pigserver; try { /** on définit les propriétés */ properties props = new properties(); props.setproperty("fs.default.name", "hdfs://master.node.ip.adress:8020"); props.setproperty("mapred.job.tracker", "master.node.ip.adress:8021"); system.setproperty("javax.xml.parsers.documentbuilderfactory", "com.sun.org.apache.xerces.internal.jaxp.documentbuilderfactoryimpl"); /** mapreduce mode */ pigserver = new pigserver(exectype.mapreduce, props); /** pig script's path */ pigserver.registerscript("/user/admin/data/script.pig"); /** printing pig script's output */ iterator<tuple> results = pigserver.openiterator("a"); while(results.hasnext()) system.out.println(results.next().todelimitedstring("\t")); } catch (execexception e) { e.printstacktrace(); } catch (ioexception e) { e.printstacktrace(); } } } when launch program, have following error:
13/07/19 10:44:02 info executionengine.hexecutionengine: connecting hadoop file system at: hdfs://master.node.ip.adress:8020 13/07/19 10:44:23 info ipc.client: retrying connect server: master.cs236cloud.internal/master.node.ip.adress:8020. tried 0 time(s); maxretries=45
...
13/07/19 10:59:27 info ipc.client: retrying connect server: master.cs236cloud.internal/master.node.ip.adress:8020. tried 43 time(s); maxretries=45 13/07/19 10:59:48 info ipc.client: retrying connect server: master.cs236cloud.internal/master.node.ip.adress:8020. tried 44 time(s); maxretries=45
exception in thread "main" java.lang.runtimeexception: failed create datastorage @ org.apache.pig.backend.hadoop.datastorage.hdatastorage.init(hdatastorage.java:75) @ org.apache.pig.backend.hadoop.datastorage.hdatastorage.(hdatastorage.java:58) @ org.apache.pig.backend.hadoop.executionengine.hexecutionengine.init(hexecutionengine.java:204) @ org.apache.pig.backend.hadoop.executionengine.hexecutionengine.init(hexecutionengine.java:117) @ org.apache.pig.impl.pigcontext.connect(pigcontext.java:240) @ org.apache.pig.pigserver.(pigserver.java:213) @ org.apache.pig.pigserver.(pigserver.java:198) @ org.apache.pig.pigserver.(pigserver.java:194) @ testpig.main(testpig.java:42) caused by: java.net.sockettimeoutexception: call master.cs236cloud.internal/master.node.ip.adress:8020 failed on socket timeout exception: java.net.sockettimeoutexception: 20000 millis timeout while waiting channel ready connect. ch : java.nio.channels.socketchannel[connection-pending remote=master.cs236cloud.internal/master.node.ip.adress:8020] @ org.apache.hadoop.ipc.client.wrapexception(client.java:1140) @ org.apache.hadoop.ipc.client.call(client.java:1112) @ org.apache.hadoop.ipc.rpc$invoker.invoke(rpc.java:229) @ $proxy1.getprotocolversion(unknown source) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(unknown source) @ sun.reflect.delegatingmethodaccessorimpl.invoke(unknown source) @ java.lang.reflect.method.invoke(unknown source) @ org.apache.hadoop.io.retry.retryinvocationhandler.invokemethod(retryinvocationhandler.java:85) @ org.apache.hadoop.io.retry.retryinvocationhandler.invoke(retryinvocationhandler.java:62) @ $proxy1.getprotocolversion(unknown source) @ org.apache.hadoop.ipc.rpc.checkversion(rpc.java:422) @ org.apache.hadoop.hdfs.dfsclient.createnamenode(dfsclient.java:183) @ org.apache.hadoop.hdfs.dfsclient.(dfsclient.java:281) @ org.apache.hadoop.hdfs.dfsclient.(dfsclient.java:245) @ org.apache.hadoop.hdfs.distributedfilesystem.initialize(distributedfilesystem.java:100) @ org.apache.hadoop.fs.filesystem.createfilesystem(filesystem.java:1437) @ org.apache.hadoop.fs.filesystem.access$200(filesystem.java:66) @ org.apache.hadoop.fs.filesystem$cache.get(filesystem.java:1455) @ org.apache.hadoop.fs.filesystem.get(filesystem.java:254) @ org.apache.hadoop.fs.filesystem.get(filesystem.java:123) @ org.apache.pig.backend.hadoop.datastorage.hdatastorage.init(hdatastorage.java:72) ... 8 more caused by: java.net.sockettimeoutexception: 20000 millis timeout while waiting channel ready connect. ch : java.nio.channels.socketchannel[connection-pending remote=master.cs236cloud.internal/master.node.ip.adress:8020] @ org.apache.hadoop.net.socketiowithtimeout.connect(socketiowithtimeout.java:213) @ org.apache.hadoop.net.netutils.connect(netutils.java:511) @ org.apache.hadoop.net.netutils.connect(netutils.java:481) @ org.apache.hadoop.ipc.client$connection.setupconnection(client.java:453) @ org.apache.hadoop.ipc.client$connection.setupiostreams(client.java:579) @ org.apache.hadoop.ipc.client$connection.access$2100(client.java:202) @ org.apache.hadoop.ipc.client.getconnection(client.java:1243) @ org.apache.hadoop.ipc.client.call(client.java:1087) ... 28 more
i'm sure master listening @ 8020 port. indeed, when launch command:
netstat -an | grep 8020
i've got result:
tcp 0 0 master.local.ip.adress:8020 0.0.0.0:* listen
i want give core-site.xml file:
<?xml version="1.0" encoding="utf-8"?> <!--autogenerated cloudera cm on 2013-07-12t10:43:15.666z--> <configuration> <property> <name>fs.defaultfs</name> <value>hdfs://master.cs236cloud.internal:8020</value> </property> <property> <name>fs.trash.interval</name> <value>1</value> </property> <property> <name>io.file.buffer.size</name> <value>65536</value> </property> <property> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.defaultcodec,org.apache.hadoop.io.compress.gzipcodec,org.apache.hadoop.io.compress.bzip2codec,org.apache.hadoop.io.compress.deflatecodec,org.apache.hadoop.io.compress.snappycodec,org.apache.hadoop.io.compress.lz4codec</value> </property> <property> <name>hadoop.security.authentication</name> <value>simple</value> </property> <property> <name>hadoop.rpc.protection</name> <value>authentication</value> </property> <property> <name>hadoop.security.auth_to_local</name> <value>default</value> </property> </configuration> here result of jps command on master machine:
3250 datanode 4468 main 2776 headlampserver 2541 runjar 5496 jps 4467 eventcatcherservice 2502 quorumpeermain 2650 jobtracker 3082 runjar 2597 hregionserver 2594 tasktracker 2629 hmaster 2520 3003 secondarynamenode 4553 main 3414 bootstrap 2549 alertpublisher 3172 namenode 2127 main 4583 main 3350 bootstrap i searched on internet solutions, didn't found works.
dou have idea on how solve problem ?
thanks.
Comments
Post a Comment