Skip to content

Feat/go table model api - #949

Merged
ColinLeeo merged 13 commits into
apache:developfrom
ColinLeeo:feat/go-table-model-api
Sep 10, 2026
Merged

Feat/go table model api#949
ColinLeeo merged 13 commits into
apache:developfrom
ColinLeeo:feat/go-table-model-api

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Summary

Add a table-model-only Go API backed by the public C ABI.

The Go writer now binds and retains a single TableSchema during construction. Callers can write targetless Tablets or Arrow Record/Table batches, then query table data through one configurable Reader.Query entry point.

Changes

  • Add NewWriter(path, schema, options...) with schema ownership and validation.
  • Change NewTablet to accept only columns and row capacity.
  • Add Tablet and Arrow batch writes for all supported table data types.
  • Add unified query options for time range, Tag filters, pagination, and Arrow batch size.
  • Add typed row access with one-based ResultSet column indexes.
  • Add Arrow RecordBatch/Table result consumption with io.EOF handling.
  • Add table schema lookup and enumeration.
  • Preserve BLOB length, embedded NUL bytes, empty values, and NULL values.
  • Normalize table and column identifiers consistently with the native table model.
  • Truncate existing files when creating a writer.
  • Remove tree-model symbols and examples from the public Go API.
  • Strengthen C ABI argument validation and resource ownership handling.

Validation

  • go test ./...
  • go test -race ./tsfile
  • go vet ./...
  • C++ test suite: 934 passed, 3 skipped because external compatibility fixtures were not configured
  • ./mvnw -P with-cpp -DskipTests -Dxml-format.skip=true package

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

superpowers?

@ColinLeeo
ColinLeeo requested review from HTHou and jt2594838 and a lite review from Copilot September 10, 2026 02:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

This PR introduces a table-model-only Go API backed by the public C ABI, adding schema-bound writers, unified querying with composable options, and Arrow batch interoperability while removing tree-model Go symbols/examples.

Changes:

  • Add schema-bound NewWriter(path, schema, ...), targetless Tablet, and Arrow batch write support.
  • Replace multiple query entry points with Reader.Query(...) + QueryOptions (time range, tag filters, pagination, batch mode).
  • Add table schema lookup/enumeration, one-based ResultSet column indexing, BLOB support, and strengthen C ABI validation/ownership.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
pom.xml Excludes go/go.sum from Maven-related checks.
go/tsfile/writer.go Schema-bound writer; adds Tablet/Arrow batch write paths and Arrow schema validation.
go/tsfile/tag_filter.go Introduces composable TAG filter AST + validation.
go/tsfile/tablet.go Makes Tablets targetless; adds BLOB setter; tracks row/timestamp presence.
go/tsfile/table_api_contract_test.go Adds contract tests for schema/queries/tag filters/Arrow schema validation.
go/tsfile/schema.go Removes tree-model schema types; adds table schema normalization/validation/copy.
go/tsfile/result_set.go Switches to one-based columns; adds batch-mode Arrow reading + BLOB getter.
go/tsfile/reader.go Adds unified Query API with composable options; adds schema lookup APIs.
go/tsfile/integration_test.go Reworks integration coverage for new table-only API + Arrow round trips.
go/tsfile/errors.go Removes tree-model sentinels; adds ErrWrongResultMode.
go/tsfile/doc.go Updates package docs to table-only API and indexing rules.
go/tsfile/constants_test.go Re-pins constants to new table-model-focused ABI surface.
go/tsfile/constants.go Removes tree-model constants/types (encoding/compression/etc.); keeps table types.
go/tsfile/cgo_bridge_test.go Updates bridge tests for new tablet/writer constructors.
go/tsfile/cgo_bridge.go Refactors cgo bridge to table writer + schema binding; adds tag-filter lifetime mgmt; adds Arrow/BLOB plumbing.
go/tsfile/api_test.go Updates public API tests for new NewWriter/NewTablet contracts.
go/go.sum Adds Go module checksums for Arrow and dependencies.
go/go.mod Adds Arrow dependency set; updates Go version directive format.
go/examples/tree_read_write/main.go Removes tree-model example.
go/examples/table_read_write/main_test.go Adds runnable example test harness for new table example.
go/examples/table_read_write/main.go Updates example to write with Tablet + Arrow batch, then query rows.
go/README.md Updates README to table-only API semantics and Arrow ownership.
cpp/test/file/utf8_path_test.cc Updates expectations: writer/file creation truncates existing UTF-8 paths.
cpp/test/cwrapper/cwrapper_public_writer_test.cc Adds C ABI tests for table writer schema validation and null args.
cpp/test/cwrapper/c_release_test.cc Updates C release tests to reflect truncate-on-create behavior.
cpp/src/writer/tsfile_table_writer.h Adds init error propagation + schema/table introspection methods.
cpp/src/writer/tsfile_table_writer.cc Allows targetless tablet binding per write and exposes bound schema.
cpp/src/reader/tsfile_reader.h Adds table query overload with offset/limit.
cpp/src/reader/tsfile_reader.cc Implements pagination validation and safe schema lookup.
cpp/src/cwrapper/tsfile_cwrapper.h Adds flush, Arrow write, unified table query, binary getter, checked schema APIs.
cpp/src/cwrapper/tsfile_cwrapper.cc Strengthens validation/ownership; implements new C ABI entry points.
cpp/src/cwrapper/arrow_c.cc Strengthens Arrow C Data Interface validation and schema matching.
Suppressed comments (1)

go/tsfile/writer.go:1

  • cdata.ExportArrowRecordBatch typically returns an error in Arrow Go; ignoring it can either fail compilation (if the signature returns error) or allow an uninitialized/invalid exported C struct to be passed into the C ABI. Capture and handle the export error and return a wrapped ErrInvalidArgument (or similar) before calling into writeArrow.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread go/tsfile/result_set.go
Comment on lines +205 to +208
var nativeArray cdata.CArrowArray
var nativeSchema cdata.CArrowSchema
defer cdata.ReleaseCArrowArray(&nativeArray)
defer cdata.ReleaseCArrowSchema(&nativeSchema)
Comment thread go/tsfile/result_set.go
}
return nil, err
}
record, err := cdata.ImportCRecordBatch(&nativeArray, &nativeSchema)
Comment thread go/tsfile/cgo_bridge.go Outdated
Comment on lines 424 to 427
h := C.tablet_new(&names[0], &types[0], C.uint32_t(len(dataTypes)), C.uint32_t(maxRows))
if h == nil {
return nil, newError("new tablet", C.RET_OOM)
}
Comment thread go/tsfile/result_set.go Outdated
Comment on lines +216 to +218
if nativeError, ok := err.(*Error); ok && nativeError.Code == 21 {
return nil, io.EOF
}
Comment thread go/tsfile/schema.go
return copy, nil
}

func normalizeIdentifier(value string) string { return strings.ToLower(value) }
Comment thread go/go.mod
module github.com/apache/tsfile/go

go 1.22
go 1.22.0

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 5 comments.

Suppressed comments (1)

go/tsfile/writer.go:1

  • array.TableReader can stop iteration due to an underlying error as well as normal exhaustion. The current code ignores reader.Err() after the loop, which can silently drop failures. Check reader.Err() after for reader.Next() and return it when non-nil.

Comment thread go/go.mod
module github.com/apache/tsfile/go

go 1.22
go 1.22.0
Comment thread go/tsfile/reader.go
Comment on lines +172 to +174
normalizedColumns := make([]string, len(columns))
for i, column := range columns {
normalizedColumns[i] = normalizeIdentifier(column)
Comment thread go/tsfile/result_set.go Outdated
Comment on lines +216 to +218
if nativeError, ok := err.(*Error); ok && nativeError.Code == 21 {
return nil, io.EOF
}
Comment thread go/tsfile/schema.go
Comment on lines +71 to 73
if err := validateCString("validate table schema", "table name", schema.Table); err != nil {
return TableSchema{}, fmt.Errorf("%w: %v", ErrInvalidSchema, err)
}
Comment thread go/tsfile/schema.go
return copy, nil
}

func normalizeIdentifier(value string) string { return strings.ToLower(value) }
Comment thread cpp/src/cwrapper/tsfile_cwrapper.cc
Comment thread go/tsfile/cgo_bridge.go
Comment thread go/tsfile/schema.go
Comment thread go/tsfile/tablet.go Outdated
Comment thread go/tsfile/writer.go Outdated
@ColinLeeo
ColinLeeo merged commit c9b9243 into apache:develop Sep 10, 2026
51 checks passed
@ColinLeeo
ColinLeeo deleted the feat/go-table-model-api branch September 10, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants