我一直在尝试创建一个在Dagster云中运行的容器,该容器采用ECS混合部署模型。我能够将容器推送到Dagster,但我不断收到此错误:
dagster_cloud.workspace.ecs.client.EcsServiceError: ECS service failed because task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a failed: CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH: unknown
Task logs:
runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH
For more information about the failure, check the ECS console for logs for task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a in cluster Dagster-Cloud-my-cluster-Cluster.
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1304, in _reconcile
self._wait_for_new_server_ready(
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 458, in _wait_for_new_server_ready
task_arn = self.client.wait_for_new_service(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 491, in wait_for_new_service
return self.check_service_has_running_task(
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 607, in check_service_has_running_task
self._raise_failed_task(task, container_name, logger)
File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 526, in _raise_failed_task
raise EcsServiceError(
字符串
我不知道为什么会发生这种情况,因为我确定我正在安装dagster和dagster-cloud,根据文档。我的Docker容器看起来像这样:
###############################################################################
# Base container
###############################################################################
FROM python:3.11 AS base
# Define the environment variables necessary to work with poetry
ENV POETRY_VERSION=1.5.1 \
POETRY_HOME="/opt/poetry" \
POETRY_VIRTUALENVS_IN_PROJECT=true \
POETRY_NO_INTERACTION=1
# Add the poetry bin to our path
ENV PATH="$POETRY_HOME/bin:$PATH"
###############################################################################
# Poetry installer container
###############################################################################
FROM base AS installer
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -
###############################################################################
# Container that actually builds the application
###############################################################################
FROM base AS builder
# Copy the poetry files from the poetry installer to this container
COPY --from=installer $POETRY_HOME $POETRY_HOME
# Describe the environment variables necessary to install our dependencies
ENV PYTHONFAULTHANDLER=1 \
PYTHONUNBUFFERED=1 \
PYTHONHASHSEED=random \
PIP_NO_CACHE_DIR=off \
PIP_DISABLE_PIP_VERSION_CHECK=on \
PIP_DEFAULT_TIMEOUT=100
# Set the working directory to one we can install to easily
WORKDIR /app
# Copy the poetry.lock and pyproject.toml files first to ensure that dependencies
# are only installed when they're updated
COPY poetry.lock pyproject.toml /app/
# Copy in our SSH key so we can retrieve the shared GitHub repo
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
# Install our project dependencies
RUN --mount=type=ssh poetry install --only dagster --no-ansi --no-root
# Build the project
COPY . /app/
RUN poetry build
###############################################################################
# Create a runtime environment that's much smaller
###############################################################################
FROM python:3.11-alpine AS runtime
# Set the working directory to where Dagster will look for the application
WORKDIR /opt/dagster/app
# Copy the project wheel file
COPY --from=builder /app/dist/*.whl /
# Install the wheel file using pip and then install dagster and dagster-cloud
RUN pip install --no-cache-dir /*.whl \
&& rm -rf /*.whl
型
对于这个包,我的pyproject.toml
文件列出了我的依赖项如下:
[tool.poetry.group.dagster.dependencies]
dagster = "^1.5.9"
dagster-aws = "^0.21.9"
pendulum = "^2.1.2"
pandas = "^2.1.3"
openpyxl = "^3.1.2"
dagster-cloud = "^1.5.12"
型
所以,这两个都应该安装,但显然他们没有。我在这里做错了什么?
2条答案
按热度按时间quhf5bfb1#
您可能忘记激活虚拟环境了?如果它位于/.venv,我建议添加
字符串
在Dockerfile的结尾。
tp5buhyn2#
这个容器有两个问题:
1.我正在推动整个虚拟环境,但这并不包括与一些软件包一起安装的可执行文件,这导致了错误。
1.安装wheel不会安装包的要求。
为了解决这些问题,我使用Poetry构建了一个
requirements.txt
文件,然后使用pip安装其中的需求。由于这些需求包括dagster
和dagster-cloud
,因此该软件包最终包含了两个必要的可执行文件。字符串