Hadoop: драйвер клиента Python для HiveServer2 не удается установить

Я пытаюсь установить драйвер клиента Python для HiveServer2: https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-PythonClientDriver

Установки говорят, что: "Клиентский драйвер Python для HiveServer2 доступен в https://github.com/BradRuderman/pyhs2 Он включает в себя все необходимые пакеты, такие как обертки SASL и Thrift.

Тем не менее, запуск pip install pyhs2 завершается неудачей при компиляции SASL (см. ниже). У меня Hadoop 2.2.0 установлен и работает на localhost. Пожалуйста, помогите, установить клиент Python.

[[email protected] /]# pip install pyhs2
Requirement already satisfied (use --upgrade to upgrade): pyhs2 in /usr/lib/python2.6/site-packages
Downloading/unpacking sasl (from pyhs2)
  Downloading sasl-0.1.3.tar.gz
  Running setup.py (path:/tmp/pip_build_root/sasl/setup.py) egg_info for package sasl
Downloading/unpacking thrift (from pyhs2)
  Downloading thrift-0.9.1.tar.gz
  Running setup.py (path:/tmp/pip_build_root/thrift/setup.py) egg_info for package thrift
Installing collected packages: sasl, thrift
  Running setup.py install for sasl
    building '_saslwrapper' extension
                                                                                                                                                                                                                                                                                                                                                                    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Isasl -I/usr/include/python2.6 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-2.6/sasl/saslwrapper.o
sasl/saslwrapper.cpp:21:23: error: sasl/sasl.h: No such file or directory
sasl/saslwrapper.cpp:51: error: ‘sasl_interact_t’ has not been declared
sasl/saslwrapper.cpp:54: error: ‘sasl_conn_t’ has not been declared
sasl/saslwrapper.cpp:54: error: ‘sasl_secret_t’ has not been declared
sasl/saslwrapper.cpp:57: error: ISO C++ forbids declaration of ‘sasl_conn_t’ with no type
sasl/saslwrapper.cpp:57: error: expected ‘;’ before ‘*’ token
sasl/saslwrapper.cpp:58: error: ‘sasl_callback_t’ does not name a type
sasl/saslwrapper.cpp:71: error: ISO C++ forbids declaration of ‘sasl_secret_t’ with no type
sasl/saslwrapper.cpp:71: error: expected ‘;’ before ‘*’ token
sasl/saslwrapper.cpp: In constructor ‘saslwrapper::ClientImpl::ClientImpl()’:
sasl/saslwrapper.cpp:35: error: class ‘saslwrapper::ClientImpl’ does not have any field named ‘conn’
sasl/saslwrapper.cpp:35: error: class ‘saslwrapper::ClientImpl’ does not have any field named ‘secret’
sasl/saslwrapper.cpp: In destructor ‘saslwrapper::ClientImpl::~ClientImpl()’:
sasl/saslwrapper.cpp:36: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:36: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp:36: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘void saslwrapper::ClientImpl::lastCallback()’:
sasl/saslwrapper.cpp:49: error: ‘SASL_CB_LIST_END’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::init()’:
sasl/saslwrapper.cpp:83: error: ‘sasl_client_init’ was not declared in this scope
sasl/saslwrapper.cpp:84: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:85: error: ‘sasl_errstring’ was not declared in this scope
sasl/saslwrapper.cpp:91: error: ‘SASL_CB_GETREALM’ was not declared in this scope
sasl/saslwrapper.cpp:93: error: ‘SASL_CB_USER’ was not declared in this scope
sasl/saslwrapper.cpp:94: error: ‘SASL_CB_AUTHNAME’ was not declared in this scope
sasl/saslwrapper.cpp:97: error: ‘SASL_CB_PASS’ was not declared in this scope
sasl/saslwrapper.cpp:99: error: ‘SASL_CB_PASS’ was not declared in this scope
sasl/saslwrapper.cpp:107: error: ‘SASL_NEED_PROXY’ was not declared in this scope
sasl/saslwrapper.cpp:109: error: ‘callbacks’ was not declared in this scope
sasl/saslwrapper.cpp:109: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:109: error: ‘sasl_client_new’ was not declared in this scope
sasl/saslwrapper.cpp:110: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:111: error: ‘sasl_errstring’ was not declared in this scope
sasl/saslwrapper.cpp:115: error: ‘sasl_security_properties_t’ was not declared in this scope
sasl/saslwrapper.cpp:115: error: expected ‘;’ before ‘secprops’
sasl/saslwrapper.cpp:117: error: ‘secprops’ was not declared in this scope
sasl/saslwrapper.cpp:124: error: ‘SASL_SEC_PROPS’ was not declared in this scope
sasl/saslwrapper.cpp:124: error: ‘sasl_setprop’ was not declared in this scope
sasl/saslwrapper.cpp:125: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:127: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp:133: error: ‘SASL_AUTH_EXTERNAL’ was not declared in this scope
sasl/saslwrapper.cpp:134: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:136: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp:141: error: ‘SASL_SSF_EXTERNAL’ was not declared in this scope
sasl/saslwrapper.cpp:142: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:144: error: ‘sasl_dispose’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::setAttr(const std::string&, const std::string&)’:
sasl/saslwrapper.cpp:163: error: ‘secret’ was not declared in this scope
sasl/saslwrapper.cpp:164: error: ‘sasl_secret_t’ was not declared in this scope
sasl/saslwrapper.cpp:164: error: expected primary-expression before ‘)’ token
sasl/saslwrapper.cpp:164: error: expected ‘;’ before ‘malloc’
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::start(const std::string&, saslwrapper::output_string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:199: error: ‘sasl_interact_t’ was not declared in this scope
sasl/saslwrapper.cpp:199: error: ‘prompt’ was not declared in this scope
sasl/saslwrapper.cpp:205: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:205: error: ‘sasl_client_start’ was not declared in this scope
sasl/saslwrapper.cpp:206: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:208: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:209: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:209: error: ‘SASL_CONTINUE’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::step(const std::string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:222: error: ‘sasl_interact_t’ was not declared in this scope
sasl/saslwrapper.cpp:222: error: ‘prompt’ was not declared in this scope
sasl/saslwrapper.cpp:227: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:227: error: ‘sasl_client_step’ was not declared in this scope
sasl/saslwrapper.cpp:228: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:230: error: ‘SASL_INTERACT’ was not declared in this scope
sasl/saslwrapper.cpp:231: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp:231: error: ‘SASL_CONTINUE’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::encode(const std::string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:244: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:244: error: ‘sasl_encode’ was not declared in this scope
sasl/saslwrapper.cpp:245: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::decode(const std::string&, saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:265: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:265: error: ‘sasl_decode’ was not declared in this scope
sasl/saslwrapper.cpp:266: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::getUserId(saslwrapper::output_string&)’:
sasl/saslwrapper.cpp:282: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:282: error: ‘SASL_USERNAME’ was not declared in this scope
sasl/saslwrapper.cpp:282: error: ‘sasl_getprop’ was not declared in this scope
sasl/saslwrapper.cpp:283: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘bool saslwrapper::ClientImpl::getSSF(saslwrapper::output_int*)’:
sasl/saslwrapper.cpp:294: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:294: error: ‘SASL_SSF’ was not declared in this scope
sasl/saslwrapper.cpp:294: error: ‘sasl_getprop’ was not declared in this scope
sasl/saslwrapper.cpp:295: error: ‘SASL_OK’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘void saslwrapper::ClientImpl::addCallback(long unsigned int, void*)’:
sasl/saslwrapper.cpp:311: error: ‘callbacks’ was not declared in this scope
sasl/saslwrapper.cpp: In member function ‘void saslwrapper::ClientImpl::setError(const std::string&, int, const std::string&, const std::string&)’:
sasl/saslwrapper.cpp:323: error: ‘conn’ was not declared in this scope
sasl/saslwrapper.cpp:324: error: ‘sasl_errdetail’ was not declared in this scope
sasl/saslwrapper.cpp:326: error: ‘sasl_errstring’ was not declared in this scope
sasl/saslwrapper.cpp: At global scope:
sasl/saslwrapper.cpp:338: error: variable or field ‘interact’ declared void
sasl/saslwrapper.cpp:338: error: ‘sasl_interact_t’ was not declared in this scope
sasl/saslwrapper.cpp:338: error: ‘prompt’ was not declared in this scope
error: command 'gcc' failed with exit status 1

Ответ 1

pyhs2 отлично, он упаковывает несколько необходимых python пакетов для dbapi, среди них sasl. Обратите внимание, что если вы pip install sasl, вы получите ту же ошибку.

sasl (пакет python) зависит от libsasl2-dev (на машине Debian/Ubuntu). Ошибка, которую вы видите, заключается в том, что компилятор не может найти библиотеки, которые, как предполагается, должны быть обработаны.

Если вы sudo apt-get install libsasl2-dev и повторите попытку, это скорее всего будет исправлено.

изменить Основываясь на вашем комментарии, здесь можно найти надежный способ заставить это работать:

curl -O -L ftp://ftp.cyrusimap.org/cyrus-sasl/cyrus-sasl-2.1.26.tar.gz
tar xzf cyrus-sasl-2.1.26.tar.gz
cd cyrus-sasl-2.1.26
./configure && make install

Ответ 2

Pyhs2 имеет следующие зависимости:

  • gcc-С++

  • python-devel.x86_64

  • Cyrus-SASL-devel.x86_64

Итак, запустите этот

sudo yum install gcc-c++ python-devel.x86_64 cyrus-sasl-devel.x86_64

sudo pip install pyhs2

Ответ 3

Если у вас есть ошибка в Windows при установке "sasl" с

pip install sasl

вы можете установить предварительно скомпилированный пакет.

Загрузите sasl-пакет для python36 в x64 из ссылки: https://www.lfd.uci.edu/~gohlke/pythonlibs/#sasl

Затем запустите pip install sasl-0.2.1-cp36-cp36m-win_amd64.whl