Structure
The Problem
In my projects RestAPI I have a function called "search" which makes a request to SOLR to get some search results. To make the request I use Pysolr. Normally all my project components run in the same Docker network but when I run my RestAPI tests using Pytest the RestAPI is started separately. Pysolr then tries to connect to my SOLR Nodes using the Docker container names, which will obviously not work.
What I want
I need Pysolr to connect to the SOLR Node using localhost, but I can't figure out how to get this working.
What I tried
I tried to change my docker-compose-override.yml file in a way that my SOLR Nodes are running on localhost and not on their docker names. And I tried to change the various environment variables for the Zookeeper and SOLR servers / hosts to use localhost. I also tried to somehow set in Pysolr directly the SOLR Nodes.
What I expected
I expected that Pysolr would use localhost:Port to connect to the SOLR Nodes.
Code
docker-compose.yml
search1:
image: solr:9.8.0
restart: always
ports:
- "8983:8983"
networks:
- default
- search
environment:
ZK_HOST: search-zoo1,search-zoo2,search-zoo3
SOLR_HOST: search1
SOLR_PORT: 8983
SOLR_MODULES: jwt-auth
depends_on:
- search-zoo1
- search-zoo2
- search-zoo3
search2:
image: solr:9.8.0
restart: always
networks:
- default
- search
environment:
ZK_HOST: search-zoo1,search-zoo2,search-zoo3
SOLR_HOST: search2
SOLR_PORT: 8983
SOLR_MODULES: jwt-auth
depends_on:
- search-zoo1
- search-zoo2
- search-zoo3
search3:
image: solr:9.8.0
restart: always
networks:
- default
- search
environment:
ZK_HOST: search-zoo1,search-zoo2,search-zoo3
SOLR_HOST: search3
SOLR_PORT: 8983
SOLR_MODULES: jwt-auth
depends_on:
- search-zoo1
- search-zoo2
- search-zoo3
search-zoo1:
image: zookeeper:3.9.3
networks:
- search
- default
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=search-zoo1:2888:3888;2181 server.2=search-zoo2:2888:3888;2181 server.3=search-zoo3:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok"
expose:
- "2181"
search-zoo2:
image: zookeeper:3.9.3
networks:
- search
- default
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=search-zoo1:2888:3888;2181 server.2=search-zoo2:2888:3888;2181 server.3=search-zoo3:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok"
expose:
- "2181"
search-zoo3:
image: zookeeper:3.9.3
networks:
- search
- default
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=search-zoo1:2888:3888;2181 server.2=search-zoo2:2888:3888;2181 server.3=search-zoo3:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: "mntr,conf,ruok"
expose:
- "2181"
docker-compose-override.yml
search1:
restart: "no"
ports:
- "8983:8983"
environment:
ZK_HOST: search-zoo1:2181,search-zoo2:2182,search-zoo3:2183
SOLR_JAVA_OPTS: -Dsolr.auth.jwt.allowOutboundHttp=true
SOLR_HOST: search1
SOLR_PORT: 8983
search2:
restart: "no"
ports:
- "8984:8983"
environment:
ZK_HOST: search-zoo1:2181,search-zoo2:2182,search-zoo3:2183
SOLR_JAVA_OPTS: -Dsolr.auth.jwt.allowOutboundHttp=true
SOLR_HOST: search2
SOLR_PORT: 8984
search3:
restart: "no"
ports:
- "8985:8983"
environment:
ZK_HOST: search-zoo1:2181,search-zoo2:2182,search-zoo3:2183
SOLR_JAVA_OPTS: -Dsolr.auth.jwt.allowOutboundHttp=true
SOLR_HOST: search3
SOLR_PORT: 8985
search-zoo1:
ports:
- "2181:2181"
environment:
ZOO_SERVERS: server.1=search-zoo1:2888:3888;2181 server.2=search-zoo2:2888:3888;2182 server.3=search-zoo3:2888:3888;2183
search-zoo2:
ports:
- "2182:2181"
environment:
ZOO_SERVERS: server.1=search-zoo1:2888:3888;2181 server.2=search-zoo2:2888:3888;2182 server.3=search-zoo3:2888:3888;2183
search-zoo3:
ports:
- "2183:2181"
environment:
ZOO_SERVERS: server.1=search-zoo1:2888:3888;2181 server.2=search-zoo2:2888:3888;2182 server.3=search-zoo3:2888:3888;2183
search.py
async def search(
skip: int = 0,
limit: int = 100,
params: SearchQueryParams = Depends(),
) -> Any:
"""
Search query.
"""
zookeeper = pysolr.ZooKeeper("localhost:2181,localhost:2182,localhost:2183")
solr = pysolr.SolrCloud(zookeeper, "tag", auth=JWTAuth(settings.SOLR_TOKEN))
res = solr.search(q=params.query, start=skip, rows=limit)
return SearchResults(data=res, count=res.hits)
Trace back
pysolr:pysolr.py:428 Failed to connect to server at http://search2:8984/solr/tag/select/?q=test-tag&start=0&rows=100&wt=json: HTTPConnectionPool(host='search2', port=8984): Max retries exceeded with url: /solr/tag/select/?q=test-tag&start=0&rows=100&wt=json (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x10ba89fd0>: Failed to resolve 'search2' ([Errno 8] nodename nor servname provided, or not known)"))
Traceback (most recent call last):
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 199, in _new_conn
sock = connection.create_connection(
(self._dns_host, self.port),
...<2 lines>...
socket_options=self.socket_options,
)
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.pyenv/versions/3.13.1/lib/python3.13/socket.py", line 975, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 495, in _make_request
conn.request(
~~~~~~~~~~~~^
method,
^^^^^^^
...<6 lines>...
enforce_content_length=enforce_content_length,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 441, in request
self.endheaders()
~~~~~~~~~~~~~~~^^
File "/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1091, in _send_output
self.send(msg)
~~~~~~~~~^^^^^
File "/.pyenv/versions/3.13.1/lib/python3.13/http/client.py", line 1035, in send
self.connect()
~~~~~~~~~~~~^^
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 279, in connect
self.sock = self._new_conn()
~~~~~~~~~~~~~~^^
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 206, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x10ba89fd0>: Failed to resolve 'search2' ([Errno 8] nodename nor servname provided, or not known)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/restapi/.venv/lib/python3.13/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 843, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/restapi/.venv/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='search2', port=8984): Max retries exceeded with url: /solr/tag/select/?q=test-tag&start=0&rows=100&wt=json (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x10ba89fd0>: Failed to resolve 'search2' ([Errno 8] nodename nor servname provided, or not known)"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/restapi/.venv/lib/python3.13/site-packages/pysolr.py", line 414, in _send_request
resp = requests_method(
url,
...<4 lines>...
auth=self.auth,
)
File "/restapi/.venv/lib/python3.13/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
File "/restapi/.venv/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/restapi/.venv/lib/python3.13/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/restapi/.venv/lib/python3.13/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='search2', port=8984): Max retries exceeded with url: /solr/tag/select/?q=test-tag&start=0&rows=100&wt=json (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x10ba89fd0>: Failed to resolve 'search2' ([Errno 8] nodename nor servname provided, or not known)"))