Category Archives: golang

Useful links for proto generator development

https://rotemtam.com/2021/03/22/creating-a-protoc-plugin-to-gen-go-code/ https://medium.com/@tim.r.coulson/writing-a-protoc-plugin-with-google-golang-org-protobuf-cd5aa75f5777 https://github.com/nametake/protoc-gen-gohttp https://github.com/drekle/protoc-gen-goexample/blob/c60883c9711f3cfbd5191fd697651dbd067e6fdd/main.go#L135 https://github.com/grpc-ecosystem/grpc-gateway/blob/24434e22fb9734f1a62c81c4ea246125d7844645/examples/internal/proto/examplepb/echo_service_grpc.pb.go https://pkg.go.dev/google.golang.org/protobuf/compiler/protogen#Options.Run https://github.com/lyft/protoc-gen-star https://go.dev/blog/protobuf-apiv2 https://docs.buf.build/build/usage/#configuration https://github.com/grpc-ecosystem/grpc-gateway https://github.com/googleapis/googleapis/tree/master/google/api https://github.com/bufbuild/buf https://stackoverflow.com/questions/36540427/cannot-find-package-google-protobuf https://pkg.go.dev/google.golang.org/protobuf https://developers.google.com/protocol-buffers/docs/reference/go-generated#package https://stackoverflow.com/questions/61666805/correct-format-of-protoc-go-package https://grpc.io/docs/languages/go/quickstart/ https://github.com/protocolbuffers/protobuf/tree/master/src/google/protobuf https://github.com/gogo/protobuf/blob/b03c65ea87cdc3521ede29f62fe3ce239267c1bc/protobuf/google/protobuf/descriptor.proto

Go study

A list of resources for learning golang https://github.com/ardanlabs/gotraining/tree/master/reading

Nullable fields in json generated from protobuf file

Null for nil value my.proto

import "google/protobuf/wrappers.proto";

message FeeView {
    google.protobuf.StringValue name = 1 [json_name = "name"];
}

import "google/protobuf/wrappers.proto";

message FeeView {

google.protobuf.StringValue name = 1 [json_name = "name"];

}

go mapper

import "github.com/gogo/protobuf/types"

var name *types.StringValue
if fee.Name != nil {
	name = &types.StringValue{
		Value: *fee.Name,
	}
}

fee := &desc.FeeView{
	Name: name,
}

import "github.com/gogo/protobuf/types"

var name *types.StringValue

if fee.Name != nil {

name = &types.StringValue{

Value: *fee.Name,

}

fee := &desc.FeeView{

Name: name,

}

Which will lead to this json response

{"translation_key":"a fee","name":null,"value":5,"style":"body","text_color":"grey","tag":null}

1	{"translation_key":"a fee","name":null,"value":5,"style":"body","text_color":"grey","tag":null}

Omit a field with nil value If you want to omit the field completely, you can use oneof approach proto

message View {
    oneof subtitle {
        string oneof_string_subtitle = 1 [json_name = "subtitle"];
    }
}

message View {

oneof subtitle {

string oneof_string_subtitle = 1 [json_name = "subtitle"];

}

go mapper

var subtitle *desc.View_OneofStringSubtitle
if in.Subtitle != nil {
	subtitle = &desc.View_OneofStringSubtitle{
		OneofStringSubtitle: *in.Subtitle,
	}
}

return &desc.View{
	Subtitle: subtitle,
}

var subtitle *desc.View_OneofStringSubtitle

if in.Subtitle != nil {

subtitle = &desc.View_OneofStringSubtitle{

OneofStringSubtitle: *in.Subtitle,

}

return &desc.View{

Subtitle: subtitle,

}

More info on json options for grpc — https://blog.bullgare.com/2020/07/complete-list-of-swagger-options-to-protobuf-file/

Talk about pros and cons of GraphQL

Pros: Very flexible — clients decide which fields of which entities they need. Documentation and test UI are available by default. Cons: Code looks ugly. N+1 selects problem is hard to solve (most probably — with some ad-hoc). It’s harder to test as you don’t know all the use cases. Not a binary protocol — …

Debugging go app in Jetbrains Goland on m1

First download a proper version of GoLand here — select .dmg (MacOS Apple Silicon) here (direct link — https://www.jetbrains.com/go/download/download-thanks.html?type=eap&platform=macM1&build=211.6556.11&code=GO). Then make sure you are using arm version of go (go version go1.16.2 darwin/arm64). If not, download and install arm version of golang (like «go1.16.2.darwin-arm64.pkg») from the list here — https://golang.org/dl/. Now it should work. With …

Example of using embed in go

Nice article on using go templates with embed introduced in go1.16: https://philipptanlak.com/web-frontends-in-go/. That’s the code:

Embedding assets to go binary in go 1.16

That’s how you can use it:

package server

import "embed"

// content holds our static web server content.
//go:embed image/* template/*
//go:embed html/index.html
var content embed.FS
data, _ := f.ReadFile("html/index.html")
print(string(data))

package server

import "embed"

// content holds our static web server content.

//go:embed image/* template/*

//go:embed html/index.html

var content embed.FS

data, _ := f.ReadFile("html/index.html")

print(string(data))

Or embedding one file directly into a variable:

import _ "embed"

//go:embed hello.txt
var s string
print(s)

import _ "embed"

//go:embed hello.txt

var s string

print(s)

More details — https://golang.org/pkg/embed/

Forward multiple kubernetes pods for local development

kubefwd is a great tool that can forward a lot of pods at once to your local machine. The only downside for me is that it touches your local /etc/hosts. That’s how I use it:

sudo KUBECONFIG=<path to your k8s config> kubefwd svc -l "app in (first-api, second-api, third-service)"

1	sudo KUBECONFIG=<path to your k8s config> kubefwd svc -l "app in (first-api, second-api, third-service)"

It does not have huge documentation, but it is written in golang, and you can check how the source …

Force ipv4 for golang http client

Sometimes you want to prevent your http client from using ipv6/tcp6. That’s how you do it.

dialer := &net.Dialer{
	Timeout:   dialTimeout,
	KeepAlive: keepAliveTimeout,
}

transport := &http.Transport{
	Proxy:             http.ProxyFromEnvironment,
	ForceAttemptHTTP2: false,
	DialContext: func(ctx context.Context, network string, addr string) (net.Conn, error) {
		ipv4, err := resolveIPv4(addr)
		if err != nil {
			return nil, err
		}

		return dialer.DialContext(ctx, network, ipv4)
	},
}

httpClient := &http.Client{
	Transport: transport,
	Timeout:   args.HTTPTimeout,
}

// resolveIPv4 resolves an address to IPv4 address.
func resolveIPv4(addr string) (string, error) {
	url := strings.Split(addr, ":")

	m := new(dns.Msg)
	m.SetQuestion(dns.Fqdn(url[0]), dns.TypeA)
	m.RecursionDesired = true

	config, _ := dns.ClientConfigFromFile("/etc/resolv.conf")
	c := new(dns.Client)
	r, _, err := c.Exchange(m, net.JoinHostPort(config.Servers[0], config.Port))
	if err != nil {
		return "", err
	}
	for _, ans := range r.Answer {
		if a, ok := ans.(*dns.A); ok {
			url[0] = a.A.String()
		}
	}

	return strings.Join(url, ":"), nil
}

dialer := &net.Dialer{

Timeout: dialTimeout,

KeepAlive: keepAliveTimeout,

}

transport := &http.Transport{

Proxy: http.ProxyFromEnvironment,

ForceAttemptHTTP2: false,

DialContext: func(ctx context.Context, network string, addr string) (net.Conn, error) {

ipv4, err := resolveIPv4(addr)

if err != nil {

return nil, err

}

return dialer.DialContext(ctx, network, ipv4)

}

httpClient := &http.Client{

Transport: transport,

Timeout: args.HTTPTimeout,

}

// resolveIPv4 resolves an address to IPv4 address.

func resolveIPv4(addr string) (string, error) {

url := strings.Split(addr, ":")

m := new(dns.Msg)

m.SetQuestion(dns.Fqdn(url[0]), dns.TypeA)

m.RecursionDesired = true

config, _ := dns.ClientConfigFromFile("/etc/resolv.conf")

c := new(dns.Client)

r, _, err := c.Exchange(m, net.JoinHostPort(config.Servers[0], config.Port))

if err != nil {

return "", err

}

for _, ans := range r.Answer {

if a, ok := ans.(*dns.A); ok {

url[0] = a.A.String()

}

return strings.Join(url, ":"), nil

}

Mocking for unit-tests and e2e-tests in golang

Some patterns to test your golang app. Mocking an interface

	mockS3 := &mocks.S3API{}

	mockResultFn := func(input *s3.ListObjectsInput) *s3.ListObjectsOutput {
		output := &s3.ListObjectsOutput{}
		output.SetCommonPrefixes([]*s3.CommonPrefix{
			&s3.CommonPrefix{
				Prefix: aws.String("2017-01-01"),
			},
		})
		return output
	}

	// NB: .Return(...) must return the same signature as the method being mocked.
	//     In this case it's (*s3.ListObjectsOutput, error).
	mockS3.On("ListObjects", mock.MatchedBy(func(input *s3.ListObjectsInput) bool {
		return input.Delimiter != nil && *input.Delimiter == "/" && input.Prefix == nil
	})).Return(mockResultFn, nil)

	listingInput := &s3.ListObjectsInput{
		Bucket:    aws.String("foo"),
		Delimiter: aws.String("/"),
	}
	listingOutput, err := mockS3.ListObjects(listingInput)
	if err != nil {
		panic(err)
	}

	for _, x := range listingOutput.CommonPrefixes {
		fmt.Printf("common prefix: %+v\n", *x)
	}

mockS3 := &mocks.S3API{}

mockResultFn := func(input *s3.ListObjectsInput) *s3.ListObjectsOutput {

output := &s3.ListObjectsOutput{}

output.SetCommonPrefixes([]*s3.CommonPrefix{

&s3.CommonPrefix{

Prefix: aws.String("2017-01-01"),

})

return output

}

// NB: .Return(...) must return the same signature as the method being mocked.

// In this case it's (*s3.ListObjectsOutput, error).

mockS3.On("ListObjects", mock.MatchedBy(func(input *s3.ListObjectsInput) bool {

return input.Delimiter != nil && *input.Delimiter == "/" && input.Prefix == nil

})).Return(mockResultFn, nil)

listingInput := &s3.ListObjectsInput{

Bucket: aws.String("foo"),

Delimiter: aws.String("/"),

}

listingOutput, err := mockS3.ListObjects(listingInput)

if err != nil {

panic(err)

}

for _, x := range listingOutput.CommonPrefixes {

fmt.Printf("common prefix: %+v\n", *x)

}

https://github.com/vektra/mockery Mocking SQL database

// a successful case
func TestShouldUpdateStats(t *testing.T) {
	db, mock, err := sqlmock.New()
	if err != nil {
		t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
	}
	defer db.Close()

	mock.ExpectBegin()
	mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))
	mock.ExpectExec("INSERT INTO product_viewers").WithArgs(2, 3).WillReturnResult(sqlmock.NewResult(1, 1))
	mock.ExpectCommit()

	// now we execute our method
	if err = recordStats(db, 2, 3); err != nil {
		t.Errorf("error was not expected while updating stats: %s", err)
	}

	// we make sure that all expectations were met
	if err := mock.ExpectationsWereMet(); err != nil {
		t.Errorf("there were unfulfilled expectations: %s", err)
	}
}

// a successful case

func TestShouldUpdateStats(t *testing.T) {

db, mock, err := sqlmock.New()

if err != nil {

t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)

}

defer db.Close()

mock.ExpectBegin()

mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))

mock.ExpectExec("INSERT INTO product_viewers").WithArgs(2, 3).WillReturnResult(sqlmock.NewResult(1, 1))

mock.ExpectCommit()

// now we execute our method

if err = recordStats(db, 2, 3); err != nil {

t.Errorf("error was not expected while updating stats: %s", err)

}

// we make sure that all expectations were met

if err := mock.ExpectationsWereMet(); err != nil {

t.Errorf("there were unfulfilled expectations: %s", err)

}

https://github.com/DATA-DOG/go-sqlmock Mocking HTTP server for unit-tests

	handler := func(w http.ResponseWriter, r *http.Request) {
		io.WriteString(w, "<html><body>Hello World!</body></html>")
	}

	req := httptest.NewRequest("GET", "http://example.com/foo", nil)
	w := httptest.NewRecorder()
	handler(w, req)

	resp := w.Result()
	body, _ := ioutil.ReadAll(resp.Body)

	fmt.Println(resp.StatusCode)
	fmt.Println(resp.Header.Get("Content-Type"))
	fmt.Println(string(body))

handler := func(w http.ResponseWriter, r *http.Request) {

io.WriteString(w, "<html><body>Hello World!</body></html>")

}

req := httptest.NewRequest("GET", "http://example.com/foo", nil)

w := httptest.NewRecorder()

handler(w, req)

resp := w.Result()

body, _ := ioutil.ReadAll(resp.Body)

fmt.Println(resp.StatusCode)

fmt.Println(resp.Header.Get("Content-Type"))

fmt.Println(string(body))

https://golang.org/pkg/net/http/httptest/ Stubbing HTTP server for e2e-tests

{
	"host": "example.com",
	"method": "GET|POST|PUT|PATCH|... (Mandatory)", 
	"path": "/your/path/:variable (Mandatory)",
	"queryStringParameters": {
		"name": ["value"],
		"name": ["value", "value"]
	},
	"headers": {
		"name": ["value"]
	},
	"cookies": {
		"name": "value"
	},
	"body": "Expected Body"
}

{

"host": "example.com",

"method": "GET|POST|PUT|PATCH|... (Mandatory)",

"path": "/your/path/:variable (Mandatory)",

"queryStringParameters": {

"name": ["value"],

"name": ["value", "value"]

"headers": {

"name": ["value"]

"cookies": {

"name": "value"

"body": "Expected Body"

}

https://github.com/jmartin82/mmock Mocking for e2e-tests with dockertest

var db *sql.DB

func TestMain(m *testing.M) {
	// uses a sensible default on windows (tcp/http) and linux/osx (socket)
	pool, err := dockertest.NewPool("")
	if err != nil {
		log.Fatalf("Could not connect to docker: %s", err)
	}

	// pulls an image, creates a container based on it and runs it
	resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
	if err != nil {
		log.Fatalf("Could not start resource: %s", err)
	}

	// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
	if err := pool.Retry(func() error {
		var err error
		db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
		if err != nil {
			return err
		}
		return db.Ping()
	}); err != nil {
		log.Fatalf("Could not connect to docker: %s", err)
	}

	code := m.Run()
	
	// You can't defer this because os.Exit doesn't care for defer
	if err := pool.Purge(resource); err != nil {
		log.Fatalf("Could not purge resource: %s", err)
	}
	
	os.Exit(code)
}

func TestSomething(t *testing.T) {
	// db.Query()
}

var db *sql.DB

func TestMain(m *testing.M) {

// uses a sensible default on windows (tcp/http) and linux/osx (socket)

pool, err := dockertest.NewPool("")

if err != nil {

log.Fatalf("Could not connect to docker: %s", err)

}

// pulls an image, creates a container based on it and runs it

resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})

if err != nil {

log.Fatalf("Could not start resource: %s", err)

}

// exponential backoff-retry, because the application in the container might not be ready to accept connections yet

if err := pool.Retry(func() error {

var err error

db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))

if err != nil {

return err

}

return db.Ping()

}); err != nil {

log.Fatalf("Could not connect to docker: %s", err)

}

code := m.Run()

// You can't defer this because os.Exit doesn't care for defer

if err := pool.Purge(resource); err != nil {

log.Fatalf("Could not purge resource: %s", err)

}

os.Exit(code)

}

func TestSomething(t *testing.T) {

// db.Query()

}

https://github.com/ory/dockertest