Historical errata for Distributed Services with Go

The 3rd curl command is missing a quotation mark in the data being POSTed.

$ curl -X POST localhost:8080 -d \\
‘{record“: {”value“: ”TGV0J3MgR28gIzMK"}}’

The missing double quote character should be placed before ‘record’ in the data.

Looks like there’s a missing word here:

“and I’ve found Go to the most pleasant by far.”

I’m guessing it was meant to be: “I’ve found Go to the most”.

In the listing for the segment.Remove() method, a “s.Close()” method is being called, but I’m not finding its definition in the text, and you might want to include it for people who are typing along.

In the listing for the log.Read method, it has a test in the “for” loop that looks like this:
if segment.baseOffset <= off { … break }

Let’s say there are thousands of segments, and I’m looking for off=10000.
In the for loop, it starts at the first segment… let’s say its baseOffset is 0.
No matter what value I set for “off”, the very first segment will always be chosen because 0 is less than any number.
This seems like a bug to me.

This is likely an issue with my own setup, but in case it’s not, on page 19 the reader is asked to run a ‘go get’ command before compiling the log.proto file. This command was insufficient for me to be able to compile the proto file. I received errors about a missing executable.

error:

protoc-gen-gogo: program not found or is not executable
—gogo_out: protoc-gen-gogo: Plugin failed with status code 1.

I needed to run the following in order to be able to compile:
‘go get github.com/gogo/protobuf/protoc-gen-gogo@v1.3.1’

System:
Ubuntu 18.04
golang v1.13.4

I think it would be nice if the tearDown function returned by testSetup would also delete the temporary directory and its contents.

Additionally, at the end of Chapter 4, it would be nice if you could run “go test ./…” and everything passed, but the cmd/server/main.go program has not yet been updated to be a gRPC server, but is instead now outdated and fails to compile due to it still be an HTTP server.

Minor suggestion - instead of calling os.Getenv twice, I like putting assignment in the “if” statement, like:
func configFile(filename string) string {
  if v := os.Getenv(“CONFIG_DIR”); v != “” {
    return filepath.Join(v, filename)
  }
  …
}

The Makefile has a line with “-config=test/ca-config.json” in it, but I looked through the whole chapter and couldn’t find that file.
Also, on page 77 it would be nice to tell us to either run “make init gencert” or tell us that we should hold off on running that until later.
Thanks! I’m enjoying your book a lot! Keep up the great work!

When typing in the code on page 80, it would be less work for the reader and possibly less confusion to either explicitly add the new function signature for “testSetup”, or possibly just changing the name of “config” to “cfg” back on page 66.

The new version of NewGRPCServer doesn’t compile.
As far as I can tell, it needs to be this:
func NewGRPCServer(cfg *Config, opts …grpc.ServerOption) (*grpc.Server, error) {
  gsrv := grpc.NewServer(opts…)
  srv, err := newgrpcServer(cfg)
  if err != nil {
    return nil, err
  }

  api.RegisterLogServer(gsrv, srv)
  return gsrv, nil
}

Also, at this point, it says that “make test” will work, however, it was never said to call “make init gencert”, but when I do that, I get this error:

$ make init gencert
mkdir -p /home/glenn/.proglog/
cfssl gencert \\
  -initca test/ca-csr.json | cfssljson -bare ca
2020/03/27 19:41:00 [INFO] generating a new CA key and certificate from CSR
2020/03/27 19:41:00 [INFO] generate received request
2020/03/27 19:41:00 [INFO] received CSR
2020/03/27 19:41:00 [INFO] generating key: rsa-2048
2020/03/27 19:41:00 [INFO] encoded CSR
2020/03/27 19:41:00 [INFO] signed certificate with serial number 4010-redacted-8909
cfssl gencert \\
  -ca=ca.pem \\
  -ca-key=ca-key.pem \\
  -config=test/ca-config.json \\
  -profile=server \\
  test/server-csr.json | cfssljson -bare server
Failed to load config file: {code not read configuration file"}Failed to parse input: unexpected end of JSON input
Makefile:9: recipe for target ‘gencert’ failed
make: * [gencert] Error 1

I think this is due to test/ca-config.json never being written, and I believe I reported that already. Sorry if that’s a duplicate.

At the bottom of page 82 and top of 83, the code there is identical to the code on page 80, so no changes were necessary.
Additionally, tests have been failing since page 81 and are still failing:

go test ./…
—- FAIL: TestServer (0.01s)
—- FAIL: TestServer/produce/consume_a_message_to/from_the_log_succeeds (0.00s)
server_test.go:99:
  Error Trace:  server_test.go:99
           server_test.go:30
  Error:   Received unexpected error:
     rpc error: code = Unavailable desc = write tcp 127.0.0.1:33552->127.0.0.1:46693: write: broken pipe
  Test:   TestServer/produce/consume_a_message_to/from_the_log_succeeds
—- FAIL: TestServer/produce/consume_stream_succeeds (0.00s)
server_test.go:143:
  Error Trace:  server_test.go:143
           server_test.go:30
  Error:   Received unexpected error:
     rpc error: code = Unavailable desc = write tcp 127.0.0.1:43432->127.0.0.1:38163: write: connection reset by peer
  Test:   TestServer/produce/consume_stream_succeeds
—- FAIL: TestServer/consume_past_log_boundary_fails (0.00s)
server_test.go:116:
  Error Trace:  server_test.go:116
           server_test.go:30
  Error:   Received unexpected error:
     rpc error: code = Unavailable desc = write tcp 127.0.0.1:46394->127.0.0.1:34597: write: broken pipe
  Test:   TestServer/consume_past_log_boundary_fails
FAIL
FAIL  github.com/gmlewis/proglog/internal/server  0.013s
FAIL

Why is “srv *grpc.Server” being added to the scenarios “for” loop when on page 86, the “srv” is not being returned from “testSetup”?
Additionally, function signatures have changed at this point such as the signature for “testSetup” itself, and I don’t think that has been mentioned before saying to run the tests again on page 88. (My tests have been broken since page 81.

Also, it might be useful when you say (for example) “change the client setup code” on the bottom of page 85 to actually show the entire function again, or at least show the “before” and “after”. This one change was non-trivial, and I believe I got it right, but it would be nice to either see the “before&after” or see the whole method typed out again, possibly highlighting the changed section in bold.

Suggestion - if you want to see an extremely well-done and easy-to-follow “How To” book using Go, I highly recommend every book written by Thorsten Ball: compilerbook dot com. These books were impeccably well written and super-fun to code along with.

As I reported earlier, tests fail from page 81-87. I’m rather exited to report that after several changes, tests are now passing.
Here is my version of server_test.go in case it helps (without the imports):

func TestServer(t *testing.T) {
  for scenario, fn := range map[string]func(
    t *testing.T,
    rootClient, nobodyClient api.LogClient,
  ){
    “produce/consume a message to/from the log succeeds”: testProduceConsume,
    “produce/consume stream succeeds”: testProduceConsumeStream,
    “consume past log boundary fails”: testConsumePastBoundary,
  } {
    t.Run(scenario, func(t *testing.T) {
      rootClient, nobodyClient, teardown := testSetup(t, nil)
      defer teardown()
      fn(t, rootClient, nobodyClient)
    })
  }
}

func testSetup(t *testing.T, fn func(*Config)) (
  rootClient, nobodyClient api.LogClient,
  teardown func(),
) {
  t.Helper()

  l, err := net.Listen(“tcp”, “127.0.0.1:0”)
  require.NoError(t, err)

  newClient := func(crtPath, keyPath string) (*grpc.ClientConn, api.LogClient, []grpc.DialOption) {
    tlsConfig, err := config.SetupTLSConfig(config.TLSConfig{
      CertFile: crtPath,
      KeyFile: keyPath,
      CAFile: config.CAFile,
      Server: false,
    })
    require.NoError(t, err)

    tlsCreds := credentials.NewTLS(tlsConfig)
    opts := []grpc.DialOption{grpc.WithTransportCredentials(tlsCreds)}
    conn, err := grpc.Dial(l.Addr().String(), opts…)
    require.NoError(t, err)

    client := api.NewLogClient(conn)
    return conn, client, opts
  }

  rootConn, rootClient, _ := newClient(
    config.RootClientCertFile,
    config.RootClientKeyFile,
  )

  nobodyConn, nobodyClient, _ := newClient(
    config.NobodyClientCertFile,
    config.NobodyClientKeyFile,
  )

  serverTLSConfig, err := config.SetupTLSConfig(config.TLSConfig{
    CertFile: config.ServerCertFile,
    KeyFile: config.ServerKeyFile,
    CAFile: config.CAFile,
    ServerAddress: l.Addr().String(),
    Server: true,
  })
  require.NoError(t, err)

  serverCreds := credentials.NewTLS(serverTLSConfig)

  dir, err := ioutil.TempDir(“”, “server-test”)
  require.NoError(t, err)

  clog, err := log.NewLog(dir, log.Config{})
  require.NoError(t, err)

  cfg := &Config{CommitLog: clog}
  if fn != nil {
    fn(cfg)
  }

  server, err := NewGRPCServer(cfg, grpc.Creds(serverCreds))
  require.NoError(t, err)

  go func() {
    server.Serve(l)
  }()

  return rootClient, nobodyClient, func() {
    server.Stop()
    rootConn.Close()
    nobodyConn.Close()
    l.Close()
  }
}

func testProduceConsume(t *testing.T, client, _ api.LogClient) {
  ctx := context.Background()

  want := &api.Record{Value: []byte(“hello world”)}

  produce, err := client.Produce(
    context.Background(),
    &api.ProduceRequest{Record: want},
  )
  require.NoError(t, err)

  consume, err := client.Consume(ctx, &api.ConsumeRequest{Offset: produce.Offset})
  require.NoError(t, err)
  require.Equal(t, want, consume.Record)
}

func testConsumePastBoundary(t *testing.T, client, _ api.LogClient) {
  ctx := context.Background()

  produce, err := client.Produce(ctx, &api.ProduceRequest{
    Record: &api.Record{Value: []byte(“hello world”)},
  })
  require.NoError(t, err)

  consume, err := client.Consume(ctx, &api.ConsumeRequest{
    Offset: produce.Offset + 1,
  })
  if consume != nil {
    t.Fatal(“consume not nil”)
  }

  got := grpc.Code(err)
  want := grpc.Code(api.ErrOffsetOutOfRange{}.GRPCStatus().Err())
  if got != want {
    t.Fatalf(“got err: %v, want %v”, got, want)
  }
}

func testProduceConsumeStream(t *testing.T, client, _ api.LogClient) {
  ctx := context.Background()

  records := []*api.Record{{Value: []byte(“first message”)}, {Value: []byte(“second message”)}}

  {
    stream, err := client.ProduceStream(ctx)
    require.NoError(t, err)

    for offset, record := range records {
      err = stream.Send(&api.ProduceRequest{Record: record})
      require.NoError(t, err)

      res, err := stream.Recv()
      require.NoError(t, err)

      if res.Offset != uint64(offset) {
        t.Fatalf(“got offset: %v, want %v”, res.Offset, offset)
      }
    }
  }

  {
    stream, err := client.ConsumeStream(
      ctx,
      &api.ConsumeRequest{Offset: 0},
    )
    require.NoError(t, err)

    for _, record := range records {
      res, err := stream.Recv()
      require.NoError(t, err)
      require.Equal(t, res.Record, record)
    }
  }
}

Before suggesting to run “make test”, it should be pointed out that a new line needs to be added to TestServer, around line 27, like this:
“unauthorized client can not produce or consume”: testUnauthorized,

For the update of NewGRPCServer, it would be nice to show the newly-needed import paths. I’m guessing they are:
  grpc_middleware “github.com/grpc-ecosystem/go-grpc-middleware”
  grpc_auth “github.com/grpc-ecosystem/go-grpc-middleware/auth”
however I could be wrong because now my tests are generating errors:

go test ./…
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8d6669]

goroutine 53 [running]:
github.com/gmlewis/proglog/internal/server.(*grpcServer).Produce(0xc0000103a8, 0xabd5e0, 0xc00008e870, 0xc00008e840, 0xc0000103a8, 0x0, 0x3)
  /home/glenn/src/github.com/gmlewis/proglog/internal/server/server.go:70 +0x79
github.com/gmlewis/proglog/api/v1._Log_Produce_Handler.func1(0xabd5e0, 0xc00008e870, 0x9c86c0, 0xc00008e840, 0x0, 0x0, 0x0, 0xc00008e840)
  /home/glenn/src/github.com/gmlewis/proglog/api/v1/log.pb.go:454 +0x86
github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1(0xabd5e0, 0xc00008e810, 0x9c86c0, 0xc00008e840, 0xc0000a4f40, 0xc0000a4f60, 0x8bf2aa, 0x99dba0, 0xc0000a4f80, 0xc0000a4f40)
  /home/glenn/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/auth/auth.go:47 +0x108
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0xabd5e0, 0xc00008e810, 0x9c86c0, 0xc00008e840, 0xc000418000, 0x0, 0xc000201b18, 0x40e648)
  /home/glenn/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0xabd5e0, 0xc00008e810, 0x9c86c0, 0xc00008e840, 0xc0000a4f40, 0xc0000a4f60, 0xc000201b88, 0x4f8f98, 0x9ad720, 0xc00008e810)
  /home/glenn/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.2.0/chain.go:34 +0xd5
github.com/gmlewis/proglog/api/v1._Log_Produce_Handler(0x974920, 0xc0000103a8, 0xabd5e0, 0xc00008e810, 0xc00009e660, 0xc0001ba690, 0xabd5e0, 0xc00008e810, 0xc0000927f0, 0xf)
  /home/glenn/src/github.com/gmlewis/proglog/api/v1/log.pb.go:456 +0x14b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00007da00, 0xac22e0, 0xc0002a6300, 0xc000418000, 0xc0001ba7e0, 0xe55480, 0x0, 0x0, 0x0)
  /home/glenn/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x50a
google.golang.org/grpc.(*Server).handleStream(0xc00007da00, 0xac22e0, 0xc0002a6300, 0xc000418000, 0x0)
  /home/glenn/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xcc9
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00028cbd0, 0xc00007da00, 0xac22e0, 0xc0002a6300, 0xc000418000)
  /home/glenn/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
  /home/glenn/go/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1
FAIL  github.com/gmlewis/proglog/internal/server  0.013s
FAIL

The first code example, for log.go, has an improper path name if compared to the path of the project created in the previous page.

Title of the code block is: LetsGo/internal/server/log.go

Project path created in previous page was: github.com/travisjeffery/proglog

The title of the code block should probably be proglog/internal/server/log.go

Code example file name is LetsGo/internal/server/http.go. The instructions right above say to create a file called server.go.

$ echo ‘export PATH=“$PATH:/usr/local/protobuf/bin”’" >> ~/.zshenv

It should be

$ echo ‘export PATH=“$PATH:/usr/local/protobuf/bin/protoc”’" >> ~/.zshenv

I followed the setup in the book, but cannot generate the proto. When running the proto generation code I get the message: Missing output directives. It’s a blocker I haven’t been able to get past.

Not really sure at this point if is the setup on my machine or what.
I can’t compile the protobuffer. I’m on a MAc, installed go with brew.

When running:
protoc api/v1/*.proto —gogo_out=Mgogoproto/gogo.proto=github.com/gogo/protobuf/proto:. —proto_path=$(go list -f ‘{{ .Dir }}’ -m github.com/gogo/protobuf) —proto_path=.

This is the results I get:

protoc-gen-gogo: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
—gogo_out: protoc-gen-gogo: Plugin failed with status code 1.

The code provided in the book results in the following errors:
internal/server/server.go:9:5: cannot use (*grpcServer)(nil) (type *grpcServer) as type log_v1.LogServer in assignment:
*grpcServer does not implement log_v1.LogServer (wrong type for ProduceStream method)
have ProduceStream(log_v1.Log_ProduceStreamServer) error
want ProduceStream(*log_v1.ProduceRequest, log_v1.Log_ProduceStreamServer) error
internal/server/server.go:43:21: stream.Recv undefined (type log_v1.Log_ProduceStreamServer has no field or method Recv)
internal/server/server.go:51:23: cannot use res (type *log_v1.ProduceResponse) as type *log_v1.ProduceRequest in argument to stream.Send

Placeholder left: (Other frameworks name the same concept middleware.)

The paragraph starts with: “The authenticate(context.Context) function is an interceptor that reads …”

Bottom of the page: there is a code snippet from file titled ServerSideServiceDiscovery/internal/discovery/membership_test.go:
package is named “discovery_test”, “discovery” expected.

Upper third of the page: it says: “Append the following snippet to finish implementing replicate()”, where the method name is meant to be: “join()”

Follow-up on #86525: replicate() method name is used 2 more times on the same page instead of “join()”. The ‘replicate()’ method name makes a lot of sense though :-)

Replicator.Close() is missing on the page (referenced at page 113 (pdf page 117) by Agent.Shutdown())

After completing all code from chapter 8 (Discover Servers and Load Balance From the Client) I ended up getting an unsafe pointer error when running make test. To make sure I didn’t make a typo myself I ran the test on the provided example code, but that results in the same error:

go test -race ./…
?   github.com/travisjeffery/proglog/api/v1  [no test files]
?   github.com/travisjeffery/proglog/cmd/proglog  [no test files]
fatal error: checkptr: unsafe pointer conversion

goroutine 20 [running]:
runtime.throw(0x1023892, 0x23)
  /usr/lib/go/src/runtime/panic.go:1112 +0x72 fp=0xc000070df8 sp=0xc000070dc8 pc=0x463152
runtime.checkptrAlignment(0xc0001f86d0, 0xfaa800, 0x1)
  /usr/lib/go/src/runtime/checkptr.go:18 +0xb7 fp=0xc000070e28 sp=0xc000070df8 pc=0x4347a7
github.com/boltdb/bolt.(*Bucket).write(0xc000070fa0, 0x0, 0x0, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/bucket.go:626 +0x15c fp=0xc000070e90 sp=0xc000070e28 pc=0xca229c
github.com/boltdb/bolt.(*Bucket).CreateBucket(0xc000184638, 0x16b9414, 0x4, 0x4, 0xc0001a63c0, 0xc000184620, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/bucket.go:188 +0x465 fp=0xc000071050 sp=0xc000070e90 pc=0xc9e805
github.com/boltdb/bolt.(*Bucket).CreateBucketIfNotExists(0xc000184638, 0x16b9414, 0x4, 0x4, 0x0, 0xc0001fa940, 0x1719dc0)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/bucket.go:206 +0x5e fp=0xc0000710b8 sp=0xc000071050 pc=0xc9ebae
github.com/boltdb/bolt.(*Tx).CreateBucketIfNotExists(…)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/tx.go:115
github.com/hashicorp/raft-boltdb.(*BoltStore).initialize(0xc0001fa940, 0x0, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/hashicorp/raft-boltdb@v0.0.0-20191021154308-4207f1bf0617/bolt_store.go:98 +0x11c fp=0xc000071170 sp=0xc0000710b8 pc=0xcc397c
github.com/hashicorp/raft-boltdb.New(0xc0001228a0, 0x29, 0x0, 0xc000122800, 0x29, 0x0, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/hashicorp/raft-boltdb@v0.0.0-20191021154308-4207f1bf0617/bolt_store.go:81 +0x1af fp=0xc000071210 sp=0xc000071170 pc=0xcc372f
github.com/hashicorp/raft-boltdb.NewBoltStore(…)
  /home/arjan/Dev/go/pkg/mod/github.com/hashicorp/raft-boltdb@v0.0.0-20191021154308-4207f1bf0617/bolt_store.go:60
github.com/travisjeffery/proglog/internal/log.(*DistributedLog).setupRaft(0xc000184540, 0xc0001f8600, 0x1d, 0x0, 0x0)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/log/distributed.go:65 +0x327 fp=0xc0000716d8 sp=0xc000071210 pc=0xce1f87
github.com/travisjeffery/proglog/internal/log.NewDistributedLog(0xc0001f8600, 0x1d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/log/distributed.go:34 +0x11f fp=0xc000071738 sp=0xc0000716d8 pc=0xce18cf
github.com/travisjeffery/proglog/internal/agent.(*Agent).setupLog(0xc0001be180, 0x0, 0x0)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/agent/agent.go:111 +0x2f3 fp=0xc000071a58 sp=0xc000071738 pc=0xe63f43
github.com/travisjeffery/proglog/internal/agent.(*Agent).setupLog-fm(0xc000071b30, 0x0)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/agent/agent.go:94 +0x42 fp=0xc000071a90 sp=0xc000071a58 pc=0xe65332
github.com/travisjeffery/proglog/internal/agent.New(0xc000102c00, 0xc000102d80, 0xc0001f8600, 0x1d, 0xc0001cf3b0, 0x3f12, 0x16c4b50, 0x1, 0x0, 0x0, …)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/agent/agent.go:71 +0x224 fp=0xc000071b58 sp=0xc000071a90 pc=0xe638c4
github.com/travisjeffery/proglog/internal/agent_test.TestAgent(0xc0001ba240)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/agent/agent_test.go:59 +0x8f3 fp=0xc000071ed0 sp=0xc000071b58 pc=0xe84ca3
testing.tRunner(0xc0001ba240, 0x1040878)
  /usr/lib/go/src/testing/testing.go:992 +0x1ec fp=0xc000071fd0 sp=0xc000071ed0 pc=0x5b73fc
runtime.goexit()
  /usr/lib/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000071fd8 sp=0xc000071fd0 pc=0x4976b1
created by testing.(*T).Run
  /usr/lib/go/src/testing/testing.go:1043 +0x661

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001ba120, 0x101011d, 0x9, 0x1040878, 0x0)
  /usr/lib/go/src/testing/testing.go:1044 +0x699
testing.runTests.func1(0xc0001ba120)
  /usr/lib/go/src/testing/testing.go:1285 +0xa7
testing.tRunner(0xc0001ba120, 0xc000075d50)
  /usr/lib/go/src/testing/testing.go:992 +0x1ec
testing.runTests(0xc00013d8c0, 0x1706c60, 0x1, 0x1, 0x0)
  /usr/lib/go/src/testing/testing.go:1283 +0x528
testing.(*M).Run(0xc00018e200, 0x0)
  /usr/lib/go/src/testing/testing.go:1200 +0x300
main.main()
  _testmain.go:46 +0x224
FAIL  github.com/travisjeffery/proglog/internal/agent  0.018s
?   github.com/travisjeffery/proglog/internal/auth  [no test files]
ok   github.com/travisjeffery/proglog/internal/discovery  (cached)
ok   github.com/travisjeffery/proglog/internal/loadbalance  (cached)
fatal error: checkptr: unsafe pointer conversion

goroutine 8 [running]:
runtime.throw(0xdd8df8, 0x23)
  /usr/lib/go/src/runtime/panic.go:1112 +0x72 fp=0xc0000ad140 sp=0xc0000ad110 pc=0x462f92
runtime.checkptrAlignment(0xc000028d30, 0xd72ae0, 0x1)
  /usr/lib/go/src/runtime/checkptr.go:18 +0xb7 fp=0xc0000ad170 sp=0xc0000ad140 pc=0x434747
github.com/boltdb/bolt.(*Bucket).write(0xc0000ad2e8, 0x0, 0x0, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/bucket.go:626 +0x15c fp=0xc0000ad1d8 sp=0xc0000ad170 pc=0xaf094c
github.com/boltdb/bolt.(*Bucket).CreateBucket(0xc00026c0f8, 0x1338410, 0x4, 0x4, 0xc00000c5a0, 0xc00026c0e0, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/bucket.go:188 +0x465 fp=0xc0000ad398 sp=0xc0000ad1d8 pc=0xaeceb5
github.com/boltdb/bolt.(*Bucket).CreateBucketIfNotExists(0xc00026c0f8, 0x1338410, 0x4, 0x4, 0x0, 0xc00000ef20, 0x13967c0)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/bucket.go:206 +0x5e fp=0xc0000ad400 sp=0xc0000ad398 pc=0xaed25e
github.com/boltdb/bolt.(*Tx).CreateBucketIfNotExists(…)
  /home/arjan/Dev/go/pkg/mod/github.com/boltdb/bolt@v1.3.1/tx.go:115
github.com/hashicorp/raft-boltdb.(*BoltStore).initialize(0xc00000ef20, 0x0, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/hashicorp/raft-boltdb@v0.0.0-20191021154308-4207f1bf0617/bolt_store.go:98 +0x11c fp=0xc0000ad4b8 sp=0xc0000ad400 pc=0xb1202c
github.com/hashicorp/raft-boltdb.New(0xc00002ea50, 0x2e, 0x0, 0xc00002ea00, 0x2e, 0x4c4b40, 0x0)
  /home/arjan/Dev/go/pkg/mod/github.com/hashicorp/raft-boltdb@v0.0.0-20191021154308-4207f1bf0617/bolt_store.go:81 +0x1af fp=0xc0000ad558 sp=0xc0000ad4b8 pc=0xb11ddf
github.com/hashicorp/raft-boltdb.NewBoltStore(…)
  /home/arjan/Dev/go/pkg/mod/github.com/hashicorp/raft-boltdb@v0.0.0-20191021154308-4207f1bf0617/bolt_store.go:60
github.com/travisjeffery/proglog/internal/log.(*DistributedLog).setupRaft(0xc00026c000, 0xc00002e660, 0x22, 0x0, 0x0)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/log/distributed.go:65 +0x327 fp=0xc0000ada20 sp=0xc0000ad558 pc=0xc6f887
github.com/travisjeffery/proglog/internal/log.NewDistributedLog(0xc00002e660, 0x22, 0x0, 0x2faf080, 0x2faf080, 0x4c4b40, 0x0, 0x0, 0x0, 0x0, …)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/log/distributed.go:34 +0x11f fp=0xc0000ada80 sp=0xc0000ada20 pc=0xc6f1cf
github.com/travisjeffery/proglog/internal/log_test.TestMultipleNodes(0xc00007b680)
  /home/arjan/Dev/go/src/github.com/arjanvaneersel/distributedgo/ClientSideServiceDiscovery/internal/log/distributed_test.go:47 +0x4f3 fp=0xc0000aded0 sp=0xc0000ada80 pc=0xc7ec93
testing.tRunner(0xc00007b680, 0xdf0bf8)
  /usr/lib/go/src/testing/testing.go:992 +0x1ec fp=0xc0000adfd0 sp=0xc0000aded0 pc=0x5af53c
runtime.goexit()
  /usr/lib/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000adfd8 sp=0xc0000adfd0 pc=0x497201
created by testing.(*T).Run
  /usr/lib/go/src/testing/testing.go:1043 +0x661

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00007b0e0, 0xdcce2f, 0x11, 0xdf0bf8, 0x1)
  /usr/lib/go/src/testing/testing.go:1044 +0x699
testing.runTests.func1(0xc00007b0e0)
  /usr/lib/go/src/testing/testing.go:1285 +0xa7
testing.tRunner(0xc00007b0e0, 0xc00018fd50)
  /usr/lib/go/src/testing/testing.go:992 +0x1ec
testing.runTests(0xc00000eb80, 0x1389e20, 0x5, 0x5, 0x0)
  /usr/lib/go/src/testing/testing.go:1283 +0x528
testing.(*M).Run(0xc000228100, 0x0)
  /usr/lib/go/src/testing/testing.go:1200 +0x300
main.main()
  _testmain.go:54 +0x224
FAIL  github.com/travisjeffery/proglog/internal/log  0.018s
?   github.com/travisjeffery/proglog/internal/network  [no test files]
ok   github.com/travisjeffery/proglog/internal/server  (cached)
?   github.com/travisjeffery/proglog/internal/testutil  [no test files]
FAIL
make: * [Makefile:67: test] Error 1

Following up on #86500. It was because $GOPATH/bin was not on my load path.

in this line (helper testAppend function)
n, pos, err := s.Append(write)
write - is undefined

In the code example for /log/index.go, you’ve declared offWidth as a uint64 but later on mention that it’s defined as a uint32 so that it takes up 4 bytes of space.

By the moment of agent_test implementation, package github.com/travisjeffery/proglog/internal/network is never mentioned.

In the description paragraph for Log’s Truncate method contains this sentence, “Because we don’t have disks with infinite space, we’ll periodically call Truncate() to remove old segments whose data we (hopefully) by then and don’t need anymore.” In the latter half, it seems like maybe the word, “saved” or something similar is missing!

“newSegment(off int64) creates a new segment, appends that segment to the log’s slice of segments, and makes the new segment the active segment so that subsequent appends calls write to it.”

Toward the end of the sentence appears “subsequent appends calls write to it.” It feels like append should be singular here?

In the code snippet for testInitExisting, you changed the variable name of log *log.Log to o *log.Log. While the code is still correct, it’s an noticeable difference from the other two log tests. Was this intentional? If so, fair enough!

Correction to #86545, I see why the difference was made! Sorry!

Last paragraph, second line contains “… as if the designers had Go had taken …” probably should read “… as if the designers of Go had taken …”

fsm code listings are missing 2 method definitions: Snapshot() and Restore(1) so the resulting chapter tests wouldn’t pass.

In logStore.GetLog(2) definition line: out.Type = raft.LogType(in.Type) introduces a new field in api.v1.Record: type. To my best knowledge, the protobuf entry change was not mentioned before. I think it might make sense to indicate it.

In the chapter “Test a gRPC Server and Client” there are 3 unit tests. I can’t get the last one “testProduceConsumeStream” to run successfully. There is a line ‘res, err := stream.Recv()’, but at this point there is no method Recv() at the Log_ProduceStreamClient returned from some lines above ‘stream, err := client.ProduceStream(ctx)’
This depends on the generated code. I used exactly the versions from the book to generate, even if there are now more recent versions. If I use the existing method ‘CloseAndRecv()’ the test fails with:
  Error:   Received unexpected error:
     rpc error: code = Unknown desc = EOF

Related to #86596

Problem was in the log.proto
rpc ProduceStream(stream ProduceRequest) returns (ProduceResponse) {}

instead of

rpc ProduceStream(stream ProduceRequest) returns (stream ProduceResponse) {}

Everything’s fine. Keep going. Very good book!

In the paragraph at the top of page 37, describing the newSegment function, it says “Next, we call Truncate() on the index file to ensure it’s the proper size for memory mapping.” However, I believe that logic was moved into the index itself, and so doesn’t happen there (it’s also not in the code above the paragraph).

After finishing up the consensus section, I ended up getting test failures in the agent and distributed log tests due to the initial leader election not being successful:
[WARN] raft: no known peers, aborting election

It seems there’s a missing first line in segment.go for forcing the offset when appending:
func (s *segment) Append(record *api.Record) (offset uint64, err error) {
  record.Offset = s.nextOffset

.. which is not mentioned in the original segment section of the book. Once I added that, server_test also needed to be updated to correctly validate the records consumed from the stream:

require.Equal(t, res.Record, &api.Record{
        Value: record.Value,
        Offset: uint64(i),
      })

instead of the original from page 69:
require.Equal(t, res.Record, record)

I hope that helps!

I’ve tried to follow your suggestions on how to install protobuf and plugins, but I get this error when I run `make`.
Instead of catering for all sorts of operating systems, perhaps you could describe how to install the required tooling in a Docker container?

This it the error I get when I run `make`:

protoc api/v1/*.proto \\
—gogo_out=\\ Mgogoproto/gogo.proto=github.com/gogo/protobuf/proto:. \\
—proto_path=\\
$(go list -f ‘{{ .Dir }}’ -m github.com/gogo/protobuf) \\
—proto_path=.
protoc-gen-gogo: program not found or is not executable
—gogo_out: protoc-gen-gogo: Plugin failed with status code 1.
make: * [compile] Error 1

Cheers,
Martin

The problem is in using []byte instead of string for Value in the Record struct.
The encoding/json package will expect a base64 encoded string when decoding to []byte, this leads to throw an illegal base64 when is POSTed the payload from page 11

the snippet seems to require the client cert authentication:
tlsConfig.ClientAuth = tls.RequireAndVerifyClientCert

At this point of the book, we are trying only to set up the server tls authentication.
The test will fail with this setting at this moment, if following the book precisely.

func NewGRPCServer(log logger, opts …grpc.ServerOption) *grpc.Server {

there is no logger type at this point of the book. The parameter should be probably replaced with
config *Config

maybe i’m missing something but
func nearestMultiple(j, k uint64) uint64{
  if j>= 0{
    return (j/k) * k
  }
  return ((j-k+1) / k) * k
}

j is uint64 so can only be positive… why the check for >= 0?