Tag Archives: docker
Fixing docker does not want to update images
If you run docker pull, and you see an error like failed to register layer: Error processing tar file(exit status 2): fatal error: runtime: out of memory, there could be 2 reasons. Obvious one — not enough disk space for docker. You can fix it with
1 2 |
docker system prune -af docker volume ls -qf dangling=true |
WARNING: It will remove your unused resources like …
Create SQS queues on localstack container start
Put this file to your filesystem as ←anyname.sh→:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
#!/usr/bin/env bash set -euo pipefail # enable debug # set -x echo "configuring sqs" echo "===================" LOCALSTACK_HOST=localhost AWS_REGION=eu-central-1 # https://docs.aws.amazon.com/cli/latest/reference/sqs/create-queue.html create_queue() { local QUEUE_NAME_TO_CREATE=$1 awslocal --endpoint-url=http://${LOCALSTACK_HOST}:4566 sqs create-queue --queue-name ${QUEUE_NAME_TO_CREATE} --region ${AWS_REGION} --attributes VisibilityTimeout=30 } create_queue "queue2" create_queue "queue1" |
For instance, ./localstack_bootstrap/sqs_bootstrap.sh. Don’t forget to run chmod +x ./localstack_bootstrap/sqs_bootstrap.sh. Add this line to your docker-compose.yml: — ./localstack_bootstrap:/docker-entrypoint-initaws.d/. So it could look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
services: localstack: container_name: aws_sqs hostname: sqs image: localstack/localstack:latest environment: - AWS_DEFAULT_REGION=ap-southeast-1 - EDGE_PORT=4566 - SERVICES=sqs ports: - '4566:4566' volumes: - "${TMPDIR:-/tmp/localstack}:/tmp/localstack" - "/var/run/docker.sock:/var/run/docker.sock" - ./localstack_bootstrap:/docker-entrypoint-initaws.d/ |
Then your queues can be reached by URLs http://localhost:4566/000000000000/queue1 and http://localhost:4566/000000000000/queue2. Further reading: https://docs.aws.amazon.com/cli/latest/reference/sqs/create-queue.html https://joerg-pfruender.github.io/software/docker/microservices/testing/2020/01/25/Localstack_in_Docker.html
Using tcpdump for k8s pods
You need wireshark to be installed on your local machine — download (more on this below). If you don’t have tcpdump installed inside the pod, you can install it with
1 2 3 4 5 |
# get pod name kubectl get po -Lversion,cfg,infra-cfg -lapp=<pod-tag> kubectl exec -it <pod-name> sh # inside the pod, alpine apk add tcpdump |
Then, on your local machine:
1 |
kubectl exec <pod-name> -- tcpdump -i eth0 -w - | wireshark -k -i - |
If you want to see packets in grpc/http2, you can add a rule in Wireshark. For this, click …
Datadog in a docker for integration tests and local dev
1 |
DOCKER_CONTENT_TRUST=1 docker run --rm --name dd-agent -p 8126:8126 -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=REPLACE_WITH_NEW_KEY -e DD_SITE=datadoghq.eu datadog/agent |
mmock for mocking microservices
It is a very easy to set up though pretty powerful tool with support for delays and handling query params. It also has a console for checking all the requests and corresponding responses matched. Here it is https://github.com/jmartin82/mmock. Below I describe a way to use it with docker-compose for local development.
XDebug inside a docker container
That’s how you can debug your php app running in a docker container.
Integration tests with testcontainers-go
Here is the go library that simplifies integration tests with docker containers — https://github.com/testcontainers/testcontainers-go. That’s how you can use it to test sql — https://github.com/testcontainers/testcontainers-go/blob/master/docs/examples/cockroachdb.md. Project documentation — https://golang.testcontainers.org/features/docker_compose/ The idea is to prepare the environment, build your app, then make requests and check the DB state.
Read logs from docker container
1 |
tail -f `docker inspect --format='{{.LogPath}}' [containername]` |
Postgres in docker: random «unique violation» errors
While running postgres in docker (for development, of course), from time to time I just run into errors like «Duplicate Key Value Violates Unique Constraint». I found the solution (thanks to this answer, for sure — https://stackoverflow.com/a/47089825/801426): if you have a table called «[table]», just do this
1 |
SELECT setval('[table]_id_seq', (SELECT MAX(id) FROM [table])); |
Gitlab-ci: build go app with docker as a docker image
Preparations [TO BE UPDATED LATER] My gitlab-ci.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
variables: PACKAGE_PATH: /go/src/gitlab.com/[username_for_gitlab]/[project_name] # https://docs.gitlab.com/ee/ci/docker/using_docker_build.html CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest stages: - dep - test - build # A hack to make Golang-in-Gitlab happy .anchors: - &inject-gopath mkdir -p $(dirname ${PACKAGE_PATH}) && ln -s ${CI_PROJECT_DIR} ${PACKAGE_PATH} && cd ${PACKAGE_PATH} dep: stage: dep image: golang:1.13.1-alpine before_script: - *inject-gopath script: go mod tidy && go mod vendor artifacts: name: "vendor-$CI_PIPELINE_ID" paths: - vendor/ expire_in: 1 hour test: stage: test dependencies: - dep image: docker:17 services: - docker:dind before_script: - *inject-gopath script: docker build --target=test . lint: stage: test dependencies: - dep image: docker:17 services: - docker:dind before_script: - *inject-gopath script: docker build --target=lint . build: stage: build dependencies: - dep - lint - test image: docker:17 services: - docker:dind before_script: - *inject-gopath - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY registry script: - docker build --target=release --pull -t $CONTAINER_TEST_IMAGE . - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE - docker push $CONTAINER_TEST_IMAGE |
My Dockerfile
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
FROM golang:1.13.1-alpine AS build WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name] COPY . . RUN go get -u github.com/jessevdk/go-assets-builder RUN apk add --no-cache make RUN make build #some pareparations and after that: go build -i -o ./[path_to_bin] ./cmd/[...] FROM alpine:3.10 AS release COPY --from=build /go/src/gitlab.com/[username_for_gitlab]/[project_name]/[path_to_bin] /usr/local/bin/[appname] RUN apk add --no-cache make bash curl tzdata EXPOSE 3000 3001 ENTRYPOINT ["/usr/local/bin/[appname]"] FROM golang:1.13.1-alpine AS test ENV CGO_ENABLED=0 WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name] COPY . . RUN apk add --no-cache make bash RUN make test FROM golangci/golangci-lint:v1.20 AS lint WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name] COPY . . RUN make lint-full # to check things locally # docker build --target=check_container . # docker images # docker run -it <hash> /bin/bash FROM golang:1.13.1-alpine AS check_container WORKDIR /go/src/gitlab.com/[username_for_gitlab]/[project_name] COPY . . RUN apk add --no-cache make bash RUN /bin/bash FROM release |
Pulling from your private registry Create deploy token as described here. After that, you can do this:
1 2 3 |
docker login -u [deploy_username] -p [deploy_token] registry.gitlab.com docker pull registry.gitlab.com/[username_for_gitlab]/[project_name]:master docker run -d --name [container_name_you_like] --env-file [path-to-.env] --net host --rm registry.gitlab.com/[username_for_gitlab]/[project_name]:master |
Useful links: https://dev.to/hypnoglow/how-to-make-friends-with-golang-docker-and-gitlab-ci-4bil https://docs.gitlab.com/ee/ci/docker/using_docker_build.html https://medium.com/@tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae https://docs.gitlab.com/ee/ci/yaml/ (full description of gitlab-ci.yml) https://about.gitlab.com/blog/2017/11/27/go-tools-and-gitlab-how-to-do-continuous-integration-like-a-boss/ https://gitlab.com/hypnoglow/example-go-docker-gitlab/blob/master/.gitlab-ci.yml Somewhat useful: https://blog.lwolf.org/post/how-to-build-tiny-golang-docker-images-with-gitlab-ci/ https://angristan.xyz/build-push-docker-images-gitlab-ci/ https://docs.gitlab.com/ee/user/packages/container_registry/#use-images-from-gitlab-container-registry https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-centos-7 (it says to do sudo usermod -aG docker …