安装pyspider,python3.7 成功,但是不能启动pyspider的解决办法

在python中安装pyspider,发现很多不成功,把每一步出错分享给大家

第一步安装 pyspider

pip3 install pyspider

报错如下:

ERROR: Command errored out with exit status 10:
     command: 'c:\users\administrator\appdata\local\programs\python\python37\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\pip-install-f0ohewg0\\pycurl\\setup.py'"'"'; __file__='"'"'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\pip-install-f0ohewg0\\pycurl\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: C:\Users\ADMINI~1\AppData\Local\Temp\pip-install-f0ohewg0\pycurl\
    Complete output (1 lines):
    Please specify --curl-dir=/path/to/built/libcurl
    ----------------------------------------
ERROR: Command errored out with exit status 10: python setup.py egg_info Check the logs for full command output.

报这个错误是因为缺pycurl这个包,通过以下网址进行下载

https://pypi.tuna.tsinghua.edu.cn/simple/pycurl/

刚开始下载的是3.7版本的,因为当前的系统python是python3.7这个版本

pycurl-7.43.0.5.win-amd64-py3.7.exe

注意:要下载 pycurl-7.43.0.5.win-amd64-py3.6.exe 否则后面会报错

当我们pycurl-7.43.0.5.win-amd64-py3.7.exe下载并且安装完成之后

执行如下命令启动pyspider

pyspider all

报错如下:

最后一句出现

File "C:\Users\Administrator\AppData\Local\Programs\Python\python37\lib\site-packages\pyspider\run.py", line 231

async=True, get_object=False, no_input=False):

SyntaxError: invalid syntax

asyncawait从 python3.7 开始已经加入保留关键字中.  所以async不能作为函数的参数名.

既然有关键字冲突,我们就需要降低python的版本,当前测试通过的python的版本为python-3.6.8-amd64.exe

第二步、下载python-3.6.8-amd64.exe进行安装

第三步、下载pycurl-7.43.0.5.win-amd64-py3.6.exe进行安装

再次执行:

pyspider all

报错如下:

c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\libs\utils.py:196: FutureWarning: timeout is not supported on your platform.
  warnings.warn("timeout is not supported on your platform.", FutureWarning)
[W 201229 16:18:48 run:413] phantomjs not found, continue running without it.
[I 201229 16:18:50 result_worker:49] result_worker starting...
[I 201229 16:18:50 processor:211] processor starting...
[I 201229 16:18:50 scheduler:647] scheduler starting...
[I 201229 16:18:50 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 201229 16:18:51 tornado_fetcher:638] fetcher starting...
[I 201229 16:18:51 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 201229 16:18:51 tornado_fetcher:671] fetcher exiting...
[I 201229 16:18:51 app:84] webui exiting...
[I 201229 16:18:51 scheduler:663] scheduler exiting...
[I 201229 16:18:51 result_worker:66] result_worker exiting...
[I 201229 16:18:51 processor:229] processor exiting...
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\Scripts\pyspider-script.py", line 11, in <module>
    load_entry_point('pyspider==0.3.10', 'console_scripts', 'pyspider')()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\run.py", line 754, in main
    cli()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\run.py", line 384, in webui
    app.run(host=host, port=port)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\webui\app.py", line 59, in run
    from .webdav import dav_app
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\webui\webdav.py", line 219, in <module>
    dav_app = WsgiDAVApp(config)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\wsgidav\wsgidav_app.py", line 134, in __init__
    _check_config(config)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\wsgidav\wsgidav_app.py", line 118, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.

原因是因为WsgiDAV发布了版本 pre-release 3.x。

解决方法如下:

在安装包中找到pyspider的资源包,然后找到webui文件里面的webdav.py文件打开,修改第209行即可。

'domaincontroller': NeedAuthController(app),

修改为:

# 'domaincontroller': NeedAuthController(app),
    'http_authenticator':{
        'HTTPAuthenticator':NeedAuthController(app),
    },

 再次执行:

pyspider all

pyspider all命令报错如下:ImportError: cannot import name 'DispatcherMiddleware' from 'werkzeug.wsgi'

c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\libs\utils.py:196: FutureWarning: timeout is not supported on your platform.
  warnings.warn("timeout is not supported on your platform.", FutureWarning)
[W 201229 16:18:01 run:413] phantomjs not found, continue running without it.
[I 201229 16:18:03 result_worker:49] result_worker starting...
[I 201229 16:18:03 processor:211] processor starting...
[I 201229 16:18:03 scheduler:647] scheduler starting...
[I 201229 16:18:03 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 201229 16:18:03 tornado_fetcher:638] fetcher starting...
[I 201229 16:18:03 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 201229 16:18:03 tornado_fetcher:671] fetcher exiting...
[I 201229 16:18:03 app:84] webui exiting...
[I 201229 16:18:03 scheduler:663] scheduler exiting...
[I 201229 16:18:04 result_worker:66] result_worker exiting...
[I 201229 16:18:04 processor:229] processor exiting...
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\Scripts\pyspider-script.py", line 11, in <module>
    load_entry_point('pyspider==0.3.10', 'console_scripts', 'pyspider')()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\run.py", line 754, in main
    cli()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\run.py", line 384, in webui
    app.run(host=host, port=port)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\webui\app.py", line 64, in run
    from werkzeug.wsgi import DispatcherMiddleware
ImportError: cannot import name 'DispatcherMiddleware'

因为pyspider源码版本缘故,把版本降下来,将wsgidav替换为2.4.1

python -m pip  uninstall wsgidav     # 卸载
python -m pip install werkzeug==2.4.1     #安装2.4.1版本

 安装成功后,重启启动pyspider依旧报错:

发现还是版本的原因,werkzeug的版本为1.0.0,这个版本中没有DispatcherMiddleware方法,所以还是降低版本。

注意降低版本不能低于0.15版本,因为我们的pyspider要求大于0.15版本以上

python -m pip  uninstall werkzeug # 卸载
python -m pip install werkzeug==0.16.1   #安装0.16.1版本

提示信息如下:

Collecting werkzeug==0.16.1
  Downloading https://files.pythonhosted.org/packages/c2/e4/a859d2fe516f466642fa5c6054fd9646271f9da26b0cac0d2f37fc858c8f/Werkzeug-0.16.1-py2.py3-none-any.whl (327kB)
    100% |████████████████████████████████| 327kB 234kB/s
Installing collected packages: werkzeug
  Found existing installation: Werkzeug 1.0.1
    Uninstalling Werkzeug-1.0.1:
      Successfully uninstalled Werkzeug-1.0.1
Successfully installed werkzeug-0.16.1
You are using pip version 18.1, however version 20.3.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

证明安装成功了

再次启动pyspider all终于成功!!!

提示信息如下:

c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\pyspider\libs\utils.py:196: FutureWarning: timeout is not supported on your platform.
  warnings.warn("timeout is not supported on your platform.", FutureWarning)
[W 201229 16:22:21 run:413] phantomjs not found, continue running without it.
[I 201229 16:22:23 result_worker:49] result_worker starting...
[I 201229 16:22:23 processor:211] processor starting...
[I 201229 16:22:23 scheduler:647] scheduler starting...
[I 201229 16:22:24 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 201229 16:22:24 tornado_fetcher:638] fetcher starting...
[I 201229 16:22:24 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 201229 16:22:24 app:76] webui running on 0.0.0.0:5000
[I 201229 16:23:24 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0