Code Generation Tool

kitex is a command line tool for code generation provided by the RPC framework Kitex. At present, kitex accepts both thrift and protobuf IDLs, and supports generating a skeleton of a server side project.

Installation

If you want to use it in windows, please make sure the version of Kitex >= v0.5.2.

Kitex code generation relies on thriftgo and protoc compilers. You need to install the corresponding compilers first: thriftgo or protoc.

After installing the aforementioned tools, you can install the Kitex command-line tool itself using the following go command:

go install github.com/cloudwego/kitex/tool/cmd/kitex@latest

Alternatively, you can download the Kitex source code and navigate to the tool/cmd/kitex directory. From there, you can execute go install to install the Kitex command-line tool:

After installation, you can check the version of the tool by executing kitex -version or view the usage help by executing kitex -help.

Generate Code

Generating code involves two parts: one is the serialization and deserialization code for struct, which is generated by the underlying compilers, thriftgo or protoc. The other part is the stub code used for creating and initiating RPC calls, which is built on the top of the former by the Kitex tool. Users only need to run the Kitex code generation tool, and all the code generation process will be automatically completed by the underlying tools.

The syntax for generating code with the Kitex tool is kitex [options] xx.thrift/xxx.proto. For information about the usage of options, please refer to the end of this document.

Taking the thrift scenario as an example, there are two IDL files as follows:

// File1：example.thrift
namespace go test
include "base.thrift"

struct MyReq{
    1:required string input
    2:required base.BaseReq baseReq
}

service MyService{
    string Hello(1:required MyReq req)
}

// File2：base.thrift
struct BaseReq{
    1:required string name
}

• Execute outside GOPATH
• Execute under GOPATH

kitex -module xxx example.thrift

kitex example.thrift

After execution, a directory named kitex_gen will be generated in the current directory with the following contents:

kitex_gen/
├── base					// The generated content of base.thrift, using lowercase IDL file name as the package name when there is no 'go namespace'
│   ├── base.go				// Go code generated by thriftgo, containing the definitions from base.thrift
│   ├── k-base.go			// Generated by Kitex and includes additional serialization optimization implementations provided by Kitex
│   └── k-consts.go			// Avoiding placeholder files for 'import not used'
└── test					// The generated content of example.thrift, use go namespace as package name
    ├── example.go			// The Go code generated by Thriftgo, which includes the content defined in example.thrift
    ├── k-consts.go			// Placeholder file to avoid 'import not used'
    ├── k-example.go		// Generated by Kitex, which includes additional serialization optimization implementations provided by Kitex
    └── myservice			// The code generated by kitex for the myservice defined in example.thrift
        ├── client.go		// Provide NewClient API
        ├── invoker.go		// Provide API in the form of a Server SDK.
        ├── myservice.go	// Provide some definitions that are shared by both client.go and server.go
        └── server.go		// Provide NewServer API

Generate Code with Scaffolding

The code example mentioned earlier cannot be directly executed and requires building NewClient and NewServer on your own. The kitex command-line tool provides the -service parameter to generate scaffolded code directly. Execute the following command:

kitex -service mydemoservice demo.thrift

The generated results are as follows:

├── build.sh			// Script for quickly building services
├── handler.go		    // Generate a handler scaffold for the server
├── kitex_info.yaml  	// Record meta information for integration with cwgo tools
├── main.go		 	 // Quickly start the main function of the server
└── script			 // Build service related scripts
│    └── bootstrap.sh
├── kitex_gen
     └── ....
    

After filling in the business code in the generated function of handler. go, execute the main function of main. go to quickly start the Kitex Server.

Library Dependencies

The codes generated by kitex may depends on certain Go libraries:

• For thrift IDLs, it is github.com/apache/thrift v0.13.0
• For protobuf IDLs, it is google.golang.org/protobuf, whose version depends on Kitex’s version and no need to be specified manually by user;

Do note that, since github.com/apache/thrift/lib/go/thrift v0.14.0, the APIs are incompatible with previous versions. If you specify the -u flag when using go get to update dependencies, this library will be updated to an incompatible version and you may encounter compilation errors like:

not enough arguments in call to iprot.ReadStructBegin

Therefore, for versions >= v0.4.5, Kitex tool will automatically add a replace directive in go.mod to specify version v0.13.0. If by any chance the replace directive is missing, you can fix it by:

go mod edit -replace github.com/apache/thrift=github.com/apache/thrift@v0.13.0

Notes for Using Protobuf IDLs

The kitex only supports the proto3 version of the protocol buffers language.

The go_package option in the IDL is required. Its value is a package name sequence separated by dots (.) or by slashes (/). It determines the suffix of the result import path. For instance,

option go_package = "hello.world"; // or hello/world

will generates an import path ${imoprt path of the current directory}/kitex_gen/hello/world.

If you assign a complete import path to go_package, then kitex will generate codes for this IDL only when the import path matches the kitex_gen directory. For example:

• go_package="${import path of current module}/kitex_gen/hello/world";: OK. Kitex will generate codes for this IDL;
• go_package="${import path of current module}/hello/world";: Kitex will not generate codes for this IDL;
• go_package="any.other.domain/some/module/kitex_gen/hello/world";: Kitex will not generate codes for this IDL;

You can overwrite the go_package value in a proto file with a command line option --protobuf Msome.proto=your.package.name/kitex_gen/wherever/you/like. For the usage of the option, please refer to the official document of Protocol Buffers.

Options

The option description here may be outdated. Run kitex -h or kitex --help to get all usable options of kitex.

-service service_name

When this option is specified, kitex will generate a scaffold to build a server. Parameter service_name gives the name of the server itself when launched. Its value is usually up to the service registry and service disccovery components when using the Kitex framework.

-module module_name

This option is used to specify the Go module the generated codes belongs to. It affects import paths.

1. If the current directory is a child path of $GOPATH/src, this option can be omitted; kitex will use a path relative to $GOPATH/src as the prefix of import path in generated codes. For example, if you run kitex under $GOPATH/src/example.com/hello/world, the import path of kitex_gen/example_package/example_package.go for other package will be example.com/hello/world/kitex_gen/example_package.

2. If the current directory is not a child path of $GOPATH/src, this option must be specified.

3. If -module is specified, then kitex searches for go.mod from the current directory to the root.

   • If go.mod is not found, kitex will generated one with go mod init.
   • If go.mod is found, then kitex will check if the module name in go.mod is identical with the argument of -module; if they diffs, kitex will report an error and exit.
   • Finally, the position of go.mod and its module names determines the import path in generated codes.

-I path

Add a search path for IDL. Support adding multiple files. When searching for IDL (including other files included in IDL), it will search in the order of the added path.

Path input can also support git pull. When the current suffix is git @, http://, https://, the remote git repository will be pulled to the local location and included in the search path. The usage method is as follows:

kitex -module xx -I xxx.git abc/xxx.thrift

Or use @ xxx to specify branch pull:

kitex -module xx -I xxx.git@branch abc/xxx.thrift

During execution, the git repository will be pulled first and stored in~/. kitex/cache/xxx/xxx/ xxx@branch Directory, then search for abc/xxx.thrift in this directory and generate code

-v or -verbose

Output more logs.

-use path

When generating server codes (with -service), the -use option stops kitex from generating the kitex_gen and uses the import path given by the argument instead.

-combine-service

For thrift IDLs, when generating codes for a server, kitex only generates methods for the last service definition in the IDL. If there are multiple services in the IDL and you want to export all their abilities, the -combine-service option is for that.

This option creates a CombineService that assembles all methods of services in the target IDL and uses it in the main package. Notice that no two methods of services can have the same name.

-protobuf value

Pass an argument to protoc. The argument will be appended to the -go_out for protoc. See the documentation of protoc for available values.

-thrift value

Pass an argument to thriftgo. The argument will be appended to the -g go: for thriftgo. See the documentation of thriftgo for available values.

Kitex by default passes naming_style=golint,ignore_initialisms,gen_setter,gen_deep_equal to thriftgo and it can be overridden if you specify the same parameter in addition.

-record

In some scenarios, it may be necessary to run the Kitex command multiple times to generate code for multiple IDLs. -record parameter is used to automatically record each Kitex command executed and generate a script file for batch re execution during updates.

Usage:

kitex -module xxx -service xxx -record xxx.thrift

After executing with the - record parameter, generate the kitex-all.sh file in the execution directory and record the current command If the - record parameter is used multiple times, it will be recorded multiple times The content of kitex-all.sh is as follows:

#!/bin/bash

kitex -module xxx -service xxx xxx.thrift
kitex -module xxx xxxa.thrift
kitex -module xxx xxxb.thrift
kitex -module xxx xxxc.thrift
kitex -module xxx xxxd.thrift

....Continue Recording newly executed commands

Command records are not always appended backwards, as follows:

• Only one command with - service will be recorded
• If the idl path of the recorded command is new, append the record at the end
• If the idl path already exists, overwrite the original record

To regenerate the code, execute kitex-all.sh. If you want to manually adjust, simply open the script file and edit the command directly

-gen-path

Currently, it only takes effect in the Thrift scenario, and the protobuf side needs to be further improved and implemented.

Default scenario, kitex will generate code in kitex_ Under the gen directory, adjustments can be made through - gen path

-gen-frugal

Use case: Gray-scale access to verify the stability of Frugal, specify certain structures, and bridge them via FastCodec to be encoded and decoded through Frugal.

Add (go.codec=‘frugal’) annotation to struct in IDL, such as:

struct FrugalStruct{
    1: required string st
    2: required bool bl
}(go.codec='frugal')

When generating code, adding the “-gen-frugal” parameter will transform the FastCodec function implementations of these structures into Frugal implementations.

-frugal_struct

Use case: Gray-scale access to verify the stability of Frugal, specify certain structures, and bridge them via FastCodec to be encoded and decoded through Frugal.

Usage: The “-gen-frugal” mentioned above needs to be combined with modifications to the IDL content in order to specify the structure. This parameter provides a way to configure specifying structures directly during generation. Use “-frugal_struct $structName” to specify the structure during generation. You can append this parameter multiple times to specify multiple structure names, for example:

kitex -frugal_struct FrualStruct -frugal-struct MyStruct xxx.thrift

When specified using “-frugal_struct”, the priority is higher, and Frugal encoding and decoding will no longer be generated for the annotated scenarios of “-gen-frugal”.

-protobuf-plugin

Plugins that support extending protocs can be integrated into a rich protoc plugin ecosystem, providing convenience for expanding code generation

The usage method is as follows:

kitex -protobuf-plugin {plugin_name:options:out_dir} idl/myservice.proto

Among them:

• plugin_name: Indicates the name of the plugin to be executed; For example, ‘protoc gen go’, then its plugin name is ‘go’
• options: Represents the options passed to the plugin; Usually, some information such as ‘go module’ is passed on
• out_dir: Represents the path where the plugin generates code; If there are no special needs, it is generally specified as “.”

The above three options can be mapped to the following protoc commands, which can be automatically concatenated and executed by kitex:

protoc
    --{$plugin_name}_out={$out_dir}
    --{$plugin_name}_opt={$options}
    idl/myservice.proto

For example, if you want to use protoc-gen-validator plugin, you can execute the following command:

kitex
    -protobuf-plugin=validator:module=toutiao/middleware/kitex,recurse=true:.
    idl/myservice.proto