MAC Nutch+MySQL集成笔记

发表于 2016年12月26日
周边

目的：Nutch爬虫引擎抓取的数据自动存入MySQL
隶属：Nutch+Hadoop+HBase(MySQL)+Elasticsearch+PHP 系列实践

目录表

MAC MySQL安装

不需要什么配置，就是next最后记住弹出的窗口里的密码就行，如：

mysql 默认密码

安装的时候忘记截图了，网上找了个配图。

下载地址：http://dev.mysql.com/downloads/mysql/

Nutch的安装与配置以及使用

1、Nutch-2.3.1下载：http://nutch.apache.org/downloads.html下载，然后解压至本地安装目录，如本地根目录为${NUTCH_HOME}；

2、配置nutch对mysql的支持，修改${APACHE_NUTCH_HOME}/ivy/ivy.xml文件，分别：

1)找到以下行取消注释

1	<dependency org="mysql" name="mysql-connector-java" rev="5.1.18" conf="*->default"/>

2)修改以下行

默认为

1	<dependency org="org.apache.gora" name="gora-core" rev="0.3" conf="*->default"/>

修改后为

1	<dependency org="org.apache.gora" name="gora-core" rev="0.2.1" conf="*->default"/>

3)取消注释以下行

1	<dependency org="org.apache.gora" name="gora-sql" rev="0.1.1-incubating" conf="*->default" />

注释：上2)、3)如果不修改会有异常异常信息为

Exception in thread “main” Java.lang.ClassNotFoundException:org.apache.gora.sql.store.SqlStore

3、数据库连接配置

编辑${NUTCH_HOME}/conf/gora.properties文件，注释掉默认的数据库连接配置，同时添加以下配置内容：

###############################

# MySQL properties

###############################

gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver

gora.sqlstore.jdbc.url=jdbc:mysql://192.168.58.1:3306/nutch?createDatabaseIfNotExist=true

gora.sqlstore.jdbc.user=root

gora.sqlstore.jdbc.password=

写上你需要连接的数据库地址以及用户名密码

4、修改nutch-site配置文件

将以下内容添加至${NUTCH_HOME}/conf/nutch-site.xml中的configuration节点中

<name>http.agent.name</name>

<value>LiuXun Nutch Spider</value>

</property>

<name>http.accept.language</name>

<description>Value of the “Accept-Language” request header field.

This allows selecting non-English language as default one to retrieve.

It is a useful setting for search engines build for certain national group.

</description>

</property>

<name>parser.character.encoding.default</name>

<description>The character encoding to fall back to when no other information

is available</description>

</property>

<name>storage.data.store.class</name>

<value>org.apache.gora.sql.store.SqlStore</value>

<description>The Gora DataStore class for storing and retrieving data.

Currently the following stores are available: ….

</description>

</property>

//特别添加

<name>generate.batch.id</name>

</property>

5、编译Nutch-2.3.1

进入${NUTCH_HOME}目录下执行ant命令：ant runtime
编译成功后${NUTCH_HOME}目录下会有runtime这个目录

编译Nutch

➜ apache-nutch-2.3.1 (master) ✗ ant

Buildfile: /Users/hackgyj/apache-nutch-2.3.1/build.xml

Trying to override old definition of task javac

[taskdef] Could not load definitions from resource org/sonar/ant/antlib.xml. It could not be found.

ivy-probe-antlib:

ivy-download:

[taskdef] Could not load definitions from resource org/sonar/ant/antlib.xml. It could not be found.

ivy-download-unchecked:

ivy-init-antlib:

ivy-init:

init:

[mkdir] Created dir: /Users/hackgyj/apache-nutch-2.3.1/build

[mkdir] Created dir: /Users/hackgyj/apache-nutch-2.3.1/build/classes

[mkdir] Created dir: /Users/hackgyj/apache-nutch-2.3.1/build/release

[mkdir] Created dir: /Users/hackgyj/apache-nutch-2.3.1/build/test

[mkdir] Created dir: /Users/hackgyj/apache-nutch-2.3.1/build/test/classes

clean-lib:

resolve-default:

[ivy:resolve] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::

[ivy:resolve] :: loading settings :: file = /Users/hackgyj/apache-nutch-2.3.1/ivy/ivysettings.xml

上面报错了，需要下载sonar的jar包（sonar-ant-task-2.2.jar），并将jar包放到解压好的apache-nutch-2.3.1文件夹内的lib文件内内。由于需要连接网络下载资源，需要一些时间，根据网络情况时间不等，我自己用了大概一小时！

然后命令行执行：

ant clear

再执行

1	ant runtime

OK，没再出错，编译成功，目录下多出：build、runtime两个文件夹，其中runtime就是编译好的目录。

6、网页抓取以及配置

进入${NUTCH_HOME}/runtime/local目录下
设置抓取的网站

执行命令

mkdir -p urls //建议爬虫连接文件夹

echo 'http://www.oschina.net/' > urls/seed.txt //写入爬取的连接

bin/nutch crawl urls -depth 3 -topN 5 //开始爬虫工作

Error: JAVA_HOME is not set.

提示JAVA_HOME未设置

MAC OS X El Capitan 10.11.6 查找和设置$JAVA_HOME，命令如下

➜ ~ (master) ✗ which java

/usr/bin/java

➜ ~ (master) ✗ ls -l /usr/bin/java

lrwxr-xr-x 1 root wheel 74 Oct 20 2015 /usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java

➜ ~ (master) ✗ ls -l /System/Library/Frameworks/JavaVM.framework/Versions

total 64

lrwxr-xr-x 1 root wheel 10 Oct 20 2015 1.4 -> CurrentJDK

lrwxr-xr-x 1 root wheel 10 Oct 20 2015 1.4.2 -> CurrentJDK

lrwxr-xr-x 1 root wheel 10 Oct 20 2015 1.5 -> CurrentJDK

lrwxr-xr-x 1 root wheel 10 Oct 20 2015 1.5.0 -> CurrentJDK

lrwxr-xr-x 1 root wheel 10 Oct 20 2015 1.6 -> CurrentJDK

lrwxr-xr-x 1 root wheel 10 Oct 20 2015 1.6.0 -> CurrentJDK

drwxr-xr-x 10 root wheel 340 Oct 7 13:55 A

lrwxr-xr-x 1 root wheel 1 Oct 20 2015 Current -> A

lrwxr-xr-x 1 root wheel 52 Oct 20 2015 CurrentJDK -> /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents

➜ ~ (master) ✗ java -version

java version "1.8.0_91"

Java(TM) SE Runtime Environment (build 1.8.0_91-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

➜ ~ (master) ✗ /usr/libexec/java_home -V

Matching Java Virtual Machines (3):

1.8.0_91, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

1.6.0_65-b14-468, x86_64: "Java SE 6" /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

1.6.0_65-b14-468, i386: "Java SE 6" /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

//打开用户配置文件

//添加路径：export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

➜ ~ (master) ✗ open ~/.profile

//保存后刷新用户配置

➜ ~ (master) ✗ source ~/.profile

➜ ~ (master) ✗ echo $JAVA_HOME

/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

Command crawl is deprecated, please use bin/crawl instead

当执行bin/nutch crawl urls -depth 3 -topN 5时显示这个错误，经查资料发现是因为Nutch2.3.1不支持这么写了。

1.7和2.2.1及以上版本用bin/crawl取代bin/nutch crawl.正确的写法：

1	bin/crawl url/ test 5

好了，能执行了，但问题又出现：

Exception in thread "main" Java.lang.NoClassDefFoundError: org/apache/avro/ipc/ByteBufferOutputStream

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:191)

at org.apache.nutch.storage.StorageUtils.getDataStoreClass(StorageUtils.java:93)

at org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:77)

at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:218)

at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)

at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)

Caused by: java.lang.ClassNotFoundException: org.apache.avro.ipc.ByteBufferOutputStream

at java.NET.URLClassLoader$1.run(URLClassLoader.java:366)

at java.Net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

... 9 more

崩溃的感觉，再查发现答案是，Nutch2.3.1不支持MySQL，What………………

解决方法是：

要么使用2.2x版本，要么退回使用nutch1.x版本
或者更换MySQL为hbase存储

显示我的选择是，放弃nutch2.3.1使用nutch2.2.1。浪费我大量时间！

如出现下面的错误，请搜索本文“特别添加”来解决。

Exception in thread "main" java.lang.RuntimeException: job failed: name=generate: null, jobid=job_local200289520_0002

at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:55)

at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:199)

at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68)

at org.apache.nutch.crawl.Crawler.run(Crawler.java:152)

at org.apache.nutch.crawl.Crawler.run(Crawler.java:250)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.nutch.crawl.Crawler.main(Crawler.java:257)

nutch2.2.1成功

➜ local (master) ✗ bin/nutch crawl urls -depth 3 -topN 5

InjectorJob: Using class org.apache.gora.sql.store.SqlStore as the Gora storage class.

InjectorJob: total number of urls rejected by filters: 0

InjectorJob: total number of urls injected after normalization and filtering: 1

Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.

FetcherJob: threads: 10

FetcherJob: parsing: false

FetcherJob: resuming: false

FetcherJob : timelimit set for : -1

Using queue mode : byHost

Fetcher: threads: 10

QueueFeeder finished: total 1 records. Hit by time limit :0

nutch2.2.1的安装及配置和上面一样，其中细节版本号错误等看错误信息修正就行，最后成功。

nutch命令前面章节介绍到了执行完在mysql中即查看到爬虫抓取的内容，如下图：

nutch安装

标签：Elasticsearch , MySQL , Nutch 原文连接：MAC Nutch+MySQL集成笔记 所有媒体，可在保留署名、 原文连接 的情况下转载，若非则不得使用我方内容。

《罗生门》人性高分悬疑片 PHPMailer 5.2.17 - Remote Code Execution