Tag Archives: docker

Fixing docker does not want to update images

If you run docker pull, and you see an error like failed to register layer: Error processing tar file(exit status 2): fatal error: runtime: out of memory, there could be 2 reasons. Obvious one — not enough disk space for docker. You can fix it with

docker system prune -af
docker volume ls -qf dangling=true

1 2	docker system prune -af docker volume ls -qf dangling=true

WARNING: It will remove your unused resources like …

Create SQS queues on localstack container start

Put this file to your filesystem as ←anyname.sh→:

#!/usr/bin/env bash

set -euo pipefail

# enable debug
# set -x


echo "configuring sqs"
echo "==================="
LOCALSTACK_HOST=localhost
AWS_REGION=eu-central-1


# https://docs.aws.amazon.com/cli/latest/reference/sqs/create-queue.html
create_queue() {
	local QUEUE_NAME_TO_CREATE=$1
	awslocal --endpoint-url=http://${LOCALSTACK_HOST}:4566 sqs create-queue --queue-name ${QUEUE_NAME_TO_CREATE} --region ${AWS_REGION} --attributes VisibilityTimeout=30
}

create_queue "queue2"
create_queue "queue1"

#!/usr/bin/env bash

set -euo pipefail

# enable debug

# set -x

echo "configuring sqs"

echo "==================="

LOCALSTACK_HOST=localhost

AWS_REGION=eu-central-1

# https://docs.aws.amazon.com/cli/latest/reference/sqs/create-queue.html

create_queue() {

local QUEUE_NAME_TO_CREATE=$1

awslocal --endpoint-url=http://${LOCALSTACK_HOST}:4566 sqs create-queue --queue-name ${QUEUE_NAME_TO_CREATE} --region ${AWS_REGION} --attributes VisibilityTimeout=30

}

create_queue "queue2"

create_queue "queue1"

For instance, ./localstack_bootstrap/sqs_bootstrap.sh. Don’t forget to run chmod +x ./localstack_bootstrap/sqs_bootstrap.sh. Add this line to your docker-compose.yml: — ./localstack_bootstrap:/docker-entrypoint-initaws.d/. So it could look like this:

services:
  localstack:
    container_name: aws_sqs
    hostname: sqs
    image: localstack/localstack:latest
    environment:
      - AWS_DEFAULT_REGION=ap-southeast-1
      - EDGE_PORT=4566
      - SERVICES=sqs
    ports:
      - '4566:4566'
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - ./localstack_bootstrap:/docker-entrypoint-initaws.d/

services:

localstack:

container_name: aws_sqs

hostname: sqs

image: localstack/localstack:latest

environment:

- AWS_DEFAULT_REGION=ap-southeast-1

- EDGE_PORT=4566

- SERVICES=sqs

ports:

- '4566:4566'

volumes:

- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"

- "/var/run/docker.sock:/var/run/docker.sock"

- ./localstack_bootstrap:/docker-entrypoint-initaws.d/

Then your queues can be reached by URLs http://localhost:4566/000000000000/queue1 and http://localhost:4566/000000000000/queue2. Further reading: https://docs.aws.amazon.com/cli/latest/reference/sqs/create-queue.html https://joerg-pfruender.github.io/software/docker/microservices/testing/2020/01/25/Localstack_in_Docker.html

Using tcpdump for k8s pods

You need wireshark to be installed on your local machine — download (more on this below). If you don’t have tcpdump installed inside the pod, you can install it with

# get pod name
kubectl get po -Lversion,cfg,infra-cfg -lapp=<pod-tag>
kubectl exec -it <pod-name> sh
# inside the pod, alpine
apk add tcpdump

# get pod name

kubectl get po -Lversion,cfg,infra-cfg -lapp=<pod-tag>

kubectl exec -it <pod-name> sh

# inside the pod, alpine

apk add tcpdump

Then, on your local machine:

kubectl exec <pod-name> -- tcpdump -i eth0 -w - | wireshark -k -i -

1	kubectl exec <pod-name> -- tcpdump -i eth0 -w - \| wireshark -k -i -

If you want to see packets in grpc/http2, you can add a rule in Wireshark. For this, click …

DOCKER_CONTENT_TRUST=1 docker run --rm --name dd-agent -p 8126:8126 -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=REPLACE_WITH_NEW_KEY -e DD_SITE=datadoghq.eu datadog/agent

1	DOCKER_CONTENT_TRUST=1 docker run --rm --name dd-agent -p 8126:8126 -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=REPLACE_WITH_NEW_KEY -e DD_SITE=datadoghq.eu datadog/agent

mmock for mocking microservices

It is a very easy to set up though pretty powerful tool with support for delays and handling query params. It also has a console for checking all the requests and corresponding responses matched. Here it is https://github.com/jmartin82/mmock. Below I describe a way to use it with docker-compose for local development.

XDebug inside a docker container

That’s how you can debug your php app running in a docker container.

Integration tests with testcontainers-go

Here is the go library that simplifies integration tests with docker containers — https://github.com/testcontainers/testcontainers-go. That’s how you can use it to test sql — https://github.com/testcontainers/testcontainers-go/blob/master/docs/examples/cockroachdb.md. Project documentation — https://golang.testcontainers.org/features/docker_compose/ The idea is to prepare the environment, build your app, then make requests and check the DB state.

Read logs from docker container

tail -f `docker inspect --format='{{.LogPath}}' [containername]`

1	tail -f `docker inspect --format='{{.LogPath}}' [containername]`

Postgres in docker: random «unique violation» errors

While running postgres in docker (for development, of course), from time to time I just run into errors like «Duplicate Key Value Violates Unique Constraint». I found the solution (thanks to this answer, for sure — https://stackoverflow.com/a/47089825/801426): if you have a table called «[table]», just do this

SELECT setval('[table]_id_seq', (SELECT MAX(id) FROM [table]));

1	SELECT setval('[table]_id_seq', (SELECT MAX(id) FROM [table]));

Gitlab-ci: build go app with docker as a docker image

Preparations [TO BE UPDATED LATER] My gitlab-ci.yml

variables:
  PACKAGE_PATH: /go/src/gitlab.com/[username_for_gitlab]/[project_name]

  # https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest

stages:
  - dep
  - test
  - build

# A hack to make Golang-in-Gitlab happy
.anchors:
  - &inject-gopath
      mkdir -p $(dirname ${PACKAGE_PATH})
      && ln -s ${CI_PROJECT_DIR} ${PACKAGE_PATH}
      && cd ${PACKAGE_PATH}

dep:
  stage: dep
  image: golang:1.13.1-alpine
  before_script:
    - *inject-gopath
  script: go mod tidy && go mod vendor
  artifacts:
    name: "vendor-$CI_PIPELINE_ID"
    paths:
      - vendor/
    expire_in: 1 hour

test:
  stage: test
  dependencies:
    - dep
  image: docker:17
  services:
    - docker:dind
  before_script:
    - *inject-gopath
  script: docker build --target=test .

lint:
  stage: test
  dependencies:
    - dep
  image: docker:17
  services:
    - docker:dind
  before_script:
    - *inject-gopath
  script: docker build --target=lint .

build:
  stage: build
  dependencies:
    - dep
    - lint
    - test
  image: docker:17
  services:
    - docker:dind
  before_script:
    - *inject-gopath
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
registry
  script:
    - docker build --target=release --pull -t $CONTAINER_TEST_IMAGE .
    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
    - docker push $CONTAINER_TEST_IMAGE

variables:

PACKAGE_PATH: /go/src/gitlab.com/[username_for_gitlab]/[project_name]

# https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest

stages:

- dep

- test

- build

# A hack to make Golang-in-Gitlab happy

.anchors:

- &inject-gopath

mkdir -p $(dirname ${PACKAGE_PATH})

&& ln -s ${CI_PROJECT_DIR} ${PACKAGE_PATH}

&& cd ${PACKAGE_PATH}

dep:

stage: dep

image: golang:1.13.1-alpine

before_script:

- *inject-gopath

script: go mod tidy && go mod vendor

artifacts:

name: "vendor-$CI_PIPELINE_ID"

paths:

- vendor/

expire_in: 1 hour

test:

stage: test

dependencies:

- dep

image: docker:17

services:

- docker:dind

before_script:

- *inject-gopath

script: docker build --target=test .

lint:

stage: test

dependencies:

- dep

image: docker:17

services:

- docker:dind

before_script:

- *inject-gopath

script: docker build --target=lint .

build:

stage: build

dependencies:

- dep

- lint

- test

image: docker:17

services:

- docker:dind

before_script:

- *inject-gopath

- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

registry

script:

- docker build --target=release --pull -t $CONTAINER_TEST_IMAGE .

- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE

- docker push $CONTAINER_TEST_IMAGE

My Dockerfile

FROM golang:1.13.1-alpine AS build
WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]
COPY . .
RUN go get -u github.com/jessevdk/go-assets-builder
RUN apk add --no-cache make
RUN make build #some pareparations and after that: go build -i -o ./[path_to_bin] ./cmd/[...]

FROM alpine:3.10 AS release
COPY --from=build /go/src/gitlab.com/[username_for_gitlab]/[project_name]/[path_to_bin] /usr/local/bin/[appname]
RUN apk add --no-cache make bash curl tzdata
EXPOSE 3000 3001
ENTRYPOINT ["/usr/local/bin/[appname]"]

FROM golang:1.13.1-alpine AS test
ENV CGO_ENABLED=0
WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]
COPY . .
RUN apk add --no-cache make bash
RUN make test

FROM golangci/golangci-lint:v1.20 AS lint
WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]
COPY . .
RUN make lint-full

# to check things locally
# docker build --target=check_container .
# docker images
# docker run -it <hash> /bin/bash
FROM golang:1.13.1-alpine AS check_container
WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]
COPY . .
RUN apk add --no-cache make bash
RUN /bin/bash

FROM release

FROM golang:1.13.1-alpine AS build

WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]

COPY . .

RUN go get -u github.com/jessevdk/go-assets-builder

RUN apk add --no-cache make

RUN make build #some pareparations and after that: go build -i -o ./[path_to_bin] ./cmd/[...]

FROM alpine:3.10 AS release

COPY --from=build /go/src/gitlab.com/[username_for_gitlab]/[project_name]/[path_to_bin] /usr/local/bin/[appname]

RUN apk add --no-cache make bash curl tzdata

EXPOSE 3000 3001

ENTRYPOINT ["/usr/local/bin/[appname]"]

FROM golang:1.13.1-alpine AS test

ENV CGO_ENABLED=0

WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]

COPY . .

RUN apk add --no-cache make bash

RUN make test

FROM golangci/golangci-lint:v1.20 AS lint

WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]

COPY . .

RUN make lint-full

# to check things locally

# docker build --target=check_container .

# docker images

# docker run -it <hash> /bin/bash

FROM golang:1.13.1-alpine AS check_container

WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name]

COPY . .

RUN apk add --no-cache make bash

RUN /bin/bash

FROM release

Pulling from your private registry Create deploy token as described here. After that, you can do this:

docker login -u [deploy_username] -p [deploy_token] registry.gitlab.com
docker pull registry.gitlab.com/[username_for_gitlab]/[project_name]:master
docker run -d --name [container_name_you_like] --env-file [path-to-.env] --net host --rm registry.gitlab.com/[username_for_gitlab]/[project_name]:master

docker login -u [deploy_username] -p [deploy_token] registry.gitlab.com

docker pull registry.gitlab.com/[username_for_gitlab]/[project_name]:master

docker run -d --name [container_name_you_like] --env-file [path-to-.env] --net host --rm registry.gitlab.com/[username_for_gitlab]/[project_name]:master

Useful links: https://dev.to/hypnoglow/how-to-make-friends-with-golang-docker-and-gitlab-ci-4bil https://docs.gitlab.com/ee/ci/docker/using_docker_build.html https://medium.com/@tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae https://docs.gitlab.com/ee/ci/yaml/ (full description of gitlab-ci.yml) https://about.gitlab.com/blog/2017/11/27/go-tools-and-gitlab-how-to-do-continuous-integration-like-a-boss/ https://gitlab.com/hypnoglow/example-go-docker-gitlab/blob/master/.gitlab-ci.yml Somewhat useful: https://blog.lwolf.org/post/how-to-build-tiny-golang-docker-images-with-gitlab-ci/ https://angristan.xyz/build-push-docker-images-gitlab-ci/ https://docs.gitlab.com/ee/user/packages/container_registry/#use-images-from-gitlab-container-registry https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-centos-7 (it says to do sudo usermod -aG docker …