Category Archives: golang

Useful links for proto generator development

https://rotemtam.com/2021/03/22/creating-a-protoc-plugin-to-gen-go-code/ https://medium.com/@tim.r.coulson/writing-a-protoc-plugin-with-google-golang-org-protobuf-cd5aa75f5777 https://github.com/nametake/protoc-gen-gohttp https://github.com/drekle/protoc-gen-goexample/blob/c60883c9711f3cfbd5191fd697651dbd067e6fdd/main.go#L135 https://github.com/grpc-ecosystem/grpc-gateway/blob/24434e22fb9734f1a62c81c4ea246125d7844645/examples/internal/proto/examplepb/echo_service_grpc.pb.go https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen#Options.Run https://github.com/lyft/protoc-gen-star https://go.dev/blog/protobuf-apiv2 https://docs.buf.build/build/usage/#configuration https://github.com/grpc-ecosystem/grpc-gateway https://github.com/googleapis/googleapis/tree/master/google/api https://github.com/bufbuild/buf https://stackoverflow.com/questions/36540427/cannot-find-package-google-protobuf https://pkg.go.dev/google.golang.org/protobuf https://developers.google.com/protocol-buffers/docs/reference/go-generated#package https://stackoverflow.com/questions/61666805/correct-format-of-protoc-go-package https://grpc.io/docs/languages/go/quickstart/ https://github.com/protocolbuffers/protobuf/tree/master/src/google/protobuf https://github.com/gogo/protobuf/blob/b03c65ea87cdc3521ede29f62fe3ce239267c1bc/protobuf/google/protobuf/descriptor.proto

Go study

A list of resources for learning golang https://github.com/ardanlabs/gotraining/tree/master/reading

Nullable fields in json generated from protobuf file

Native support With grpc v2 for go, protofile’s optional option is supported, so you don’t need to import proto extensions anymore and use things like google.protobuf.StringValue. my.proto

message FeeView {
    optional string name = 1 [json_name = "name"];
}

message FeeView {

optional string name = 1 [json_name = "name"];

}

go mapper

var name *string
if fee.Name != nil {
	name = &fee.Name
}

fee := &desc.FeeView{
	Name: name,
}

var name *string

if fee.Name != nil {

name = &fee.Name

}

fee := &desc.FeeView{

Name: name,

}

Legacy way with extensions Null for nil value my.proto

import "google/protobuf/wrappers.proto";

message FeeView {
    google.protobuf.StringValue name = 1 [json_name = "name"];
}

import "google/protobuf/wrappers.proto";

message FeeView {

google.protobuf.StringValue name = 1 [json_name = "name"];

}

go mapper

import "github.com/gogo/protobuf/types"

var name *types.StringValue
if fee.Name != nil {
	name = &types.StringValue{
		Value: *fee.Name,
	}
}

fee := &desc.FeeView{
	Name: name,
}

import "github.com/gogo/protobuf/types"

var name *types.StringValue

if fee.Name != nil {

name = &types.StringValue{

Value: *fee.Name,

}

fee := &desc.FeeView{

Name: name,

}

Which will lead to this json response

{"translation_key":"a fee","name":null,"value":5,"style":"body","text_color":"grey","tag":null}

1	{"translation_key":"a fee","name":null,"value":5,"style":"body","text_color":"grey","tag":null}

Omit a field …

Talk about pros and cons of GraphQL

Pros: Very flexible — clients decide which fields of which entities they need. Documentation and test UI are available by default. Cons: Code looks ugly. N+1 selects problem is hard to solve (most probably — with some ad-hoc). It’s harder to test as you don’t know all the use cases. Not a binary protocol — …

Debugging go app in Jetbrains Goland on m1

First download a proper version of GoLand here — select .dmg (MacOS Apple Silicon) here (direct link — https://www.jetbrains.com/go/download/download-thanks.html?type=eap&platform=macM1&build=211.6556.11&code=GO). Then make sure you are using arm version of go (go version go1.16.2 darwin/arm64). If not, download and install arm version of golang (like «go1.16.2.darwin-arm64.pkg») from the list here — https://golang.org/dl/. Now it should work. With …

Example of using embed in go

Nice article on using go templates with embed introduced in go1.16: https://philipptanlak.com/web-frontends-in-go/. That’s the code:

Embedding assets to go binary in go 1.16

That’s how you can use it:

package server

import "embed"

// content holds our static web server content.
//go:embed image/* template/*
//go:embed html/index.html
var content embed.FS
data, _ := f.ReadFile("html/index.html")
print(string(data))

package server

import "embed"

// content holds our static web server content.

//go:embed image/* template/*

//go:embed html/index.html

var content embed.FS

data, _ := f.ReadFile("html/index.html")

print(string(data))

Or embedding one file directly into a variable:

import _ "embed"

//go:embed hello.txt
var s string
print(s)

import _ "embed"

//go:embed hello.txt

var s string

print(s)

More details — https://golang.org/pkg/embed/

Forward multiple kubernetes pods for local development

kubefwd is a great tool that can forward a lot of pods at once to your local machine. The only downside for me is that it touches your local /etc/hosts. That’s how I use it:

sudo KUBECONFIG=<path to your k8s config> kubefwd svc -l "app in (first-api, second-api, third-service)"

1	sudo KUBECONFIG=<path to your k8s config> kubefwd svc -l "app in (first-api, second-api, third-service)"

It does not have huge documentation, but it is written in golang, and you can check how the source …

Force ipv4 for golang http client

Sometimes you want to prevent your http client from using ipv6/tcp6. That’s how you do it.

dialer := &net.Dialer{
	Timeout:   dialTimeout,
	KeepAlive: keepAliveTimeout,
}

transport := &http.Transport{
	Proxy:             http.ProxyFromEnvironment,
	ForceAttemptHTTP2: false,
	DialContext: func(ctx context.Context, network string, addr string) (net.Conn, error) {
		ipv4, err := resolveIPv4(addr)
		if err != nil {
			return nil, err
		}

		return dialer.DialContext(ctx, network, ipv4)
	},
}

httpClient := &http.Client{
	Transport: transport,
	Timeout:   args.HTTPTimeout,
}

// resolveIPv4 resolves an address to IPv4 address.
func resolveIPv4(addr string) (string, error) {
	url := strings.Split(addr, ":")

	m := new(dns.Msg)
	m.SetQuestion(dns.Fqdn(url[0]), dns.TypeA)
	m.RecursionDesired = true

	config, _ := dns.ClientConfigFromFile("/etc/resolv.conf")
	c := new(dns.Client)
	r, _, err := c.Exchange(m, net.JoinHostPort(config.Servers[0], config.Port))
	if err != nil {
		return "", err
	}
	for _, ans := range r.Answer {
		if a, ok := ans.(*dns.A); ok {
			url[0] = a.A.String()
		}
	}

	return strings.Join(url, ":"), nil
}

dialer := &net.Dialer{

Timeout: dialTimeout,

KeepAlive: keepAliveTimeout,

}

transport := &http.Transport{

Proxy: http.ProxyFromEnvironment,

ForceAttemptHTTP2: false,

DialContext: func(ctx context.Context, network string, addr string) (net.Conn, error) {

ipv4, err := resolveIPv4(addr)

if err != nil {

return nil, err

}

return dialer.DialContext(ctx, network, ipv4)

}

httpClient := &http.Client{

Transport: transport,

Timeout: args.HTTPTimeout,

}

// resolveIPv4 resolves an address to IPv4 address.

func resolveIPv4(addr string) (string, error) {

url := strings.Split(addr, ":")

m := new(dns.Msg)

m.SetQuestion(dns.Fqdn(url[0]), dns.TypeA)

m.RecursionDesired = true

config, _ := dns.ClientConfigFromFile("/etc/resolv.conf")

c := new(dns.Client)

r, _, err := c.Exchange(m, net.JoinHostPort(config.Servers[0], config.Port))

if err != nil {

return "", err

}

for _, ans := range r.Answer {

if a, ok := ans.(*dns.A); ok {

url[0] = a.A.String()

}

return strings.Join(url, ":"), nil

}

Mocking for unit-tests and e2e-tests in golang

Some patterns to test your golang app. Mocking an interface

	mockS3 := &mocks.S3API{}

	mockResultFn := func(input *s3.ListObjectsInput) *s3.ListObjectsOutput {
		output := &s3.ListObjectsOutput{}
		output.SetCommonPrefixes([]*s3.CommonPrefix{
			&s3.CommonPrefix{
				Prefix: aws.String("2017-01-01"),
			},
		})
		return output
	}

	// NB: .Return(...) must return the same signature as the method being mocked.
	//     In this case it's (*s3.ListObjectsOutput, error).
	mockS3.On("ListObjects", mock.MatchedBy(func(input *s3.ListObjectsInput) bool {
		return input.Delimiter != nil && *input.Delimiter == "/" && input.Prefix == nil
	})).Return(mockResultFn, nil)

	listingInput := &s3.ListObjectsInput{
		Bucket:    aws.String("foo"),
		Delimiter: aws.String("/"),
	}
	listingOutput, err := mockS3.ListObjects(listingInput)
	if err != nil {
		panic(err)
	}

	for _, x := range listingOutput.CommonPrefixes {
		fmt.Printf("common prefix: %+v\n", *x)
	}

mockS3 := &mocks.S3API{}

mockResultFn := func(input *s3.ListObjectsInput) *s3.ListObjectsOutput {

output := &s3.ListObjectsOutput{}

output.SetCommonPrefixes([]*s3.CommonPrefix{

&s3.CommonPrefix{

Prefix: aws.String("2017-01-01"),

})

return output

}

// NB: .Return(...) must return the same signature as the method being mocked.

// In this case it's (*s3.ListObjectsOutput, error).

mockS3.On("ListObjects", mock.MatchedBy(func(input *s3.ListObjectsInput) bool {

return input.Delimiter != nil && *input.Delimiter == "/" && input.Prefix == nil

})).Return(mockResultFn, nil)

listingInput := &s3.ListObjectsInput{

Bucket: aws.String("foo"),

Delimiter: aws.String("/"),

}

listingOutput, err := mockS3.ListObjects(listingInput)

if err != nil {

panic(err)

}

for _, x := range listingOutput.CommonPrefixes {

fmt.Printf("common prefix: %+v\n", *x)

}

https://github.com/vektra/mockery Mocking SQL database

// a successful case
func TestShouldUpdateStats(t *testing.T) {
	db, mock, err := sqlmock.New()
	if err != nil {
		t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
	}
	defer db.Close()

	mock.ExpectBegin()
	mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))
	mock.ExpectExec("INSERT INTO product_viewers").WithArgs(2, 3).WillReturnResult(sqlmock.NewResult(1, 1))
	mock.ExpectCommit()

	// now we execute our method
	if err = recordStats(db, 2, 3); err != nil {
		t.Errorf("error was not expected while updating stats: %s", err)
	}

	// we make sure that all expectations were met
	if err := mock.ExpectationsWereMet(); err != nil {
		t.Errorf("there were unfulfilled expectations: %s", err)
	}
}

// a successful case

func TestShouldUpdateStats(t *testing.T) {

db, mock, err := sqlmock.New()

if err != nil {

t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)

}

defer db.Close()

mock.ExpectBegin()

mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))

mock.ExpectExec("INSERT INTO product_viewers").WithArgs(2, 3).WillReturnResult(sqlmock.NewResult(1, 1))

mock.ExpectCommit()

// now we execute our method

if err = recordStats(db, 2, 3); err != nil {

t.Errorf("error was not expected while updating stats: %s", err)

}

// we make sure that all expectations were met

if err := mock.ExpectationsWereMet(); err != nil {

t.Errorf("there were unfulfilled expectations: %s", err)

}

https://github.com/DATA-DOG/go-sqlmock Mocking HTTP server for unit-tests

	handler := func(w http.ResponseWriter, r *http.Request) {
		io.WriteString(w, "<html><body>Hello World!</body></html>")
	}

	req := httptest.NewRequest("GET", "http://example.com/foo", nil)
	w := httptest.NewRecorder()
	handler(w, req)

	resp := w.Result()
	body, _ := ioutil.ReadAll(resp.Body)

	fmt.Println(resp.StatusCode)
	fmt.Println(resp.Header.Get("Content-Type"))
	fmt.Println(string(body))

handler := func(w http.ResponseWriter, r *http.Request) {

io.WriteString(w, "<html><body>Hello World!</body></html>")

}

req := httptest.NewRequest("GET", "http://example.com/foo", nil)

w := httptest.NewRecorder()

handler(w, req)

resp := w.Result()

body, _ := ioutil.ReadAll(resp.Body)

fmt.Println(resp.StatusCode)

fmt.Println(resp.Header.Get("Content-Type"))

fmt.Println(string(body))

https://golang.org/pkg/net/http/httptest/ Stubbing HTTP server for e2e-tests

{
	"host": "example.com",
	"method": "GET|POST|PUT|PATCH|... (Mandatory)", 
	"path": "/your/path/:variable (Mandatory)",
	"queryStringParameters": {
		"name": ["value"],
		"name": ["value", "value"]
	},
	"headers": {
		"name": ["value"]
	},
	"cookies": {
		"name": "value"
	},
	"body": "Expected Body"
}

{

"host": "example.com",

"method": "GET|POST|PUT|PATCH|... (Mandatory)",

"path": "/your/path/:variable (Mandatory)",

"queryStringParameters": {

"name": ["value"],

"name": ["value", "value"]

"headers": {

"name": ["value"]

"cookies": {

"name": "value"

"body": "Expected Body"

}

https://github.com/jmartin82/mmock Mocking for e2e-tests with dockertest

var db *sql.DB

func TestMain(m *testing.M) {
	// uses a sensible default on windows (tcp/http) and linux/osx (socket)
	pool, err := dockertest.NewPool("")
	if err != nil {
		log.Fatalf("Could not connect to docker: %s", err)
	}

	// pulls an image, creates a container based on it and runs it
	resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
	if err != nil {
		log.Fatalf("Could not start resource: %s", err)
	}

	// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
	if err := pool.Retry(func() error {
		var err error
		db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
		if err != nil {
			return err
		}
		return db.Ping()
	}); err != nil {
		log.Fatalf("Could not connect to docker: %s", err)
	}

	code := m.Run()
	
	// You can't defer this because os.Exit doesn't care for defer
	if err := pool.Purge(resource); err != nil {
		log.Fatalf("Could not purge resource: %s", err)
	}
	
	os.Exit(code)
}

func TestSomething(t *testing.T) {
	// db.Query()
}

var db *sql.DB

func TestMain(m *testing.M) {

// uses a sensible default on windows (tcp/http) and linux/osx (socket)

pool, err := dockertest.NewPool("")

if err != nil {

log.Fatalf("Could not connect to docker: %s", err)

}

// pulls an image, creates a container based on it and runs it

resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})

if err != nil {

log.Fatalf("Could not start resource: %s", err)

}

// exponential backoff-retry, because the application in the container might not be ready to accept connections yet

if err := pool.Retry(func() error {

var err error

db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))

if err != nil {

return err

}

return db.Ping()

}); err != nil {

log.Fatalf("Could not connect to docker: %s", err)

}

code := m.Run()

// You can't defer this because os.Exit doesn't care for defer

if err := pool.Purge(resource); err != nil {

log.Fatalf("Could not purge resource: %s", err)

}

os.Exit(code)

}

func TestSomething(t *testing.T) {

// db.Query()

}

https://github.com/ory/dockertest