Comments for joe crobak's website http://www.crobak.org sharing what I find Fri, 10 May 2013 18:21:40 +0000 hourly 1 http://wordpress.org/?v=3.5.1 Comment on Workflow Engines for Hadoop by Alvin http://www.crobak.org/2012/07/workflow-engines-for-hadoop/#comment-619 Alvin Fri, 10 May 2013 18:21:40 +0000 http://www.crobak.org/?p=111#comment-619 Oozie does support rerun w/o making every action a sub-workflow. Specify oozie.wf.rerun.failnodes=true in the job.properties file and rerun the same workflow with the -rerun option and the previous job number as an argument.

]]> Comment on Workflow Engines for Hadoop by ivan provalov http://www.crobak.org/2012/07/workflow-engines-for-hadoop/#comment-617 ivan provalov Thu, 07 Feb 2013 23:54:35 +0000 http://www.crobak.org/?p=111#comment-617 Joe,

Great comparison. For Oozie workflows visualization, there is also a VizOozie. It parses a workflow xml file and generates a dot file. You can use graphviz’s dot to convert it to any format:

https://github.com/iprovalo/vizoozie

See this blog for more info:

http://info.lucidworks.com/blog/bid/266880/LucidWorks-Big-Data-Oozie-Workflow-With-VizOozie

Regards,

Ivan Provalov

]]> Comment on Moving wordpress blog to lighttpd by raspberry pi, lighttpd and wordpress pretty permalink | Rants, grunts and chants http://www.crobak.org/2011/01/moving-wordpress-blog-to-lighttpd/#comment-615 raspberry pi, lighttpd and wordpress pretty permalink | Rants, grunts and chants Fri, 28 Dec 2012 14:03:59 +0000 http://www-test.crobak.org/?p=3#comment-615 [...] finds these links useful: http://www.crobak.org/2011/01/moving-wordpress-blog-to-lighttpd/ http://longspine.com/how-to/wordpress-pretty-permalinks-on-lighttpd/ [...]

]]> Comment on Getting Started with Apache Hadoop 0.23.0 by Keith Wiley http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/#comment-613 Keith Wiley Fri, 02 Nov 2012 18:36:51 +0000 http://www.crobak.org/?p=85#comment-613 Gah! Even though a comment pointed out that you forgot to mention formatting the namenode, and even though you replied that you would update the article…the omission is still there. I spent quite a while trying to figure out why the datanode would start but not the namenode (I figured it out by investigating the namenode error log in logs/).

You should really put that update in the article. :-)

Cheers!

]]> Comment on Getting Started with Apache Hadoop 0.23.0 by Hardik http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/#comment-611 Hardik Fri, 07 Sep 2012 00:12:17 +0000 http://www.crobak.org/?p=85#comment-611 I get the same FileNotFoundException running “pi” example, anyone with some idea pleas help

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.1.jar pi -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -libjars share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.1.jar 16 10000
12/09/06 19:56:10 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
Number of Maps = 16
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Starting Job
12/09/06 19:56:22 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
12/09/06 19:56:22 INFO input.FileInputFormat: Total input paths to process : 16
12/09/06 19:56:23 INFO mapreduce.JobSubmitter: number of splits:16
12/09/06 19:56:25 INFO mapred.ResourceMgrDelegate: Submitted application application_1346972652940_0002 to ResourceManager at /0.0.0.0:8040
12/09/06 19:56:27 INFO mapreduce.Job: The url to track the job: http://10.215.12.11:8088/proxy/application_1346972652940_0002/
12/09/06 19:56:27 INFO mapreduce.Job: Running job: job_1346972652940_0002
12/09/06 19:56:55 INFO mapreduce.Job: Job job_1346972652940_0002 running in uber mode : false
12/09/06 19:56:55 INFO mapreduce.Job: map 0% reduce 0%
12/09/06 19:56:56 INFO mapreduce.Job: Job job_1346972652940_0002 failed with state FAILED due to: Application application_1346972652940_0002 failed 1 times due to AM Container for appattempt_1346972652940_0002_000001 exited with exitCode: 1 due to:
.Failing this attempt.. Failing the application.
12/09/06 19:56:56 INFO mapreduce.Job: Counters: 0
Job Finished in 35.048 seconds
java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/user/hardikpandya/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1685)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1709)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)

]]> Comment on Getting Started with Apache Hadoop 0.23.0 by dheeren@yahoo.com http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/#comment-610 dheeren@yahoo.com Tue, 04 Sep 2012 23:54:43 +0000 http://www.crobak.org/?p=85#comment-610 To start history server use
$ sbin/mr-jobhistory-daemon.sh start

With cdh401 I was unable to start history server using
$ ysbin/arn-daemon.sh start historyserver

starting historyserver, logging to /tmp/yarn-hhhhuser-historyserver-rd1-nn1-1-sfm.ops.sfdc.net.out
Exception in thread “main” java.lang.NoClassDefFoundError: historyserver
Caused by: java.lang.ClassNotFoundException: historyserver
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: historyserver. Program will exit.

]]> Comment on Workflow Engines for Hadoop by Manoj http://www.crobak.org/2012/07/workflow-engines-for-hadoop/#comment-606 Manoj Tue, 07 Aug 2012 18:17:43 +0000 http://www.crobak.org/?p=111#comment-606 Good stuff!

]]> Comment on Getting Started with Apache Hadoop 0.23.0 by rashmi http://www.crobak.org/2011/12/getting-started-with-apache-hadoop-0-23-0/#comment-534 rashmi Wed, 18 Jul 2012 19:06:35 +0000 http://www.crobak.org/?p=85#comment-534 Hi,

For hadoop-2.0.0 installation on two linux machines, what should be values of fs.defaultFS and dfs.name.dir and dfs.data.dir properties on both name nodes????

one machine hostname is rsi-nod-nsn1 and another one is rsi-nod-nsn2…

i want to make both federated namenodes.. and both should be used as datanodes too..

what should be configuration changes for the same? i am not finding masters, mapred-site.xml, and hadoop-env.sh files in hadoopHome/etc/hadoop folder… how do i make changes for these files?

]]> Comment on two puppet tricks: combining arrays and local tests by Callum http://www.crobak.org/2011/02/two-puppet-tricks-combining-arrays-and-local-tests/#comment-520 Callum Tue, 10 Jul 2012 13:24:45 +0000 http://www.crobak.org/?p=58#comment-520 A simpler option was posted on etherized.com which uses:
(array1+array2+arrayX).flatten.join(',')

Thanks for posting this, helped me out in a bind, the solution of using inline_template works well in my case. :-)

]]> Comment on Workflow Engines for Hadoop by Nathan Bijnens http://www.crobak.org/2012/07/workflow-engines-for-hadoop/#comment-507 Nathan Bijnens Thu, 05 Jul 2012 18:09:21 +0000 http://www.crobak.org/?p=111#comment-507 Nice comparison. One addition, we use our internal Jenkins server on our test cluster as a job control engine. Quite flexible, and already completely integrated with the rest of the workflow. Of-course it lacks in certain aspects.

Nathan

]]>