# Multi-Arch Build Pipeline — AWS Batch with Docker-outside-of-Docker (DooD)

## Status: IMPLEMENTED

This document describes the multi-architecture Docker image build pipeline using AWS Batch with Docker-outside-of-Docker (DooD).

## Problem

- AWS Batch simulation tasks need **ARM64** containers (Graviton compute — cheaper, better perf)
- EKS worker nodes are **AMD64** only (GovCloud `us-gov-west-1` limitation)
- A single-arch build node cannot build both architectures
- CodeBuild ARM64 is not available in `us-gov-west-1`

## Solution

Two parallel AWS Batch jobs — one on ARM64 compute, one on AMD64 compute — each builds Docker images natively for its target architecture using **Docker-outside-of-Docker (DooD)**.

DooD mounts the host EC2 instance's Docker socket (`/var/run/docker.sock`) into the `docker:cli` container. No Docker daemon runs inside the container — it uses the host's daemon directly. This avoids all DinD cgroup/storage driver issues.

## Architecture

```
sms-api (submit_build_image_job)
│
├── AWS Batch Job (ARM64 Graviton compute)
│   └── docker:cli container (DooD via mounted socket)
│       └── Clones vEcoli, builds + pushes vecoli:{commit}
│           (ARM64 task image for Batch simulation tasks)
│
└── AWS Batch Job (AMD64 x86 compute)
    └── docker:cli container (DooD via mounted socket)
        └── Clones vEcoli, builds + pushes vecoli:{commit}-submit
            (AMD64 submit image: vEcoli + Java + Nextflow for EKS K8s Job)
```

## What Gets Built

| Image | Architecture | Used By | Built By |
|-------|-------------|---------|----------|
| `vecoli:{commit}` | ARM64 | AWS Batch tasks (Graviton) | Batch ARM64 job |
| `vecoli:{commit}-submit` | AMD64 | K8s Nextflow head Job (EKS) | Batch AMD64 job |

## Infrastructure (CDK stack: `smsvpctest-build-batch`)

| Resource | Details |
|----------|---------|
| ARM64 compute env | Graviton4 (m8g.xlarge, c8g.xlarge), on-demand, 16 max vCPUs |
| AMD64 compute env | x86 (m7i.xlarge, c7i.xlarge), on-demand, 16 max vCPUs |
| ARM64 job queue | `vecoli-build-arm64` |
| AMD64 job queue | `vecoli-build-amd64` |
| Job definition | `vecoli-dind-build` — `docker:cli`, unprivileged, host Docker socket mounted |
| GitHub PAT | `vecoli-github-pat` in Secrets Manager |
| ECR repository | `vecoli` |

## Build Script Flow

Each Batch job runs this sequence inside the `docker:cli` container:

1. `apk add aws-cli git bash` — install tools (Alpine base)
2. `docker info` — verify host Docker socket is accessible
3. `aws secretsmanager get-secret-value` — fetch GitHub PAT
4. `git clone` — clone vEcoli private repo via HTTPS + PAT
5. `git checkout {commit}` — checkout specific commit
6. `aws ecr get-login-password | docker login` — authenticate to ECR
7. `bash build-and-push-ecr.sh` — build Docker image and push to ECR

The AMD64 job additionally builds the submit image (vEcoli + Java + Nextflow) on top of the base image.

## Key Learnings

- **DinD doesn't work in AWS Batch** — Docker daemon inside containers fails due to cgroup/storage driver issues even with privileged mode on EC2-backed compute
- **DooD works perfectly** — mounting host Docker socket is simple and reliable
- **`docker:cli`** (not `docker:dind`) — only need the CLI, not the daemon
- **`build-and-push-ecr.sh` requires `USER` env var** — set `export USER=${USER:-sms-api}` before calling it
- **GitHub private repos don't support `git fetch --depth 1 origin {hash}`** — use full single-branch clone instead
- **GitHub PAT auth** — `https://x-access-token:${PAT}@github.com/...` is the standard convention

## sms-api Code

- `simulation_service_k8s.py`: `_build_command()` generates the shell script, `_submit_batch_build()` submits to Batch, `_poll_batch_jobs()` polls for completion, `_run_build()` orchestrates both jobs
- `config.py`: `build_arm64_queue`, `build_amd64_queue`, `build_job_definition`, `build_git_secret_arn`
- Build status polling via `batch:DescribeJobs` — both must succeed for overall build to complete