Reach for Go and Rust methods: receiver types from dependency code, and targets inside units

docs/projectgraph.md

@@ -63,6 +63,22 @@ and no source text past a short call text.
 files of dependency code, and the report of the run. `Result.Unit` returns
 the indexes and the files of one unit of dependency code.
 
+A generated Go file of a unit, with a `// Code generated ... DO NOT EDIT.`
+line or a name that starts with `tables`, can hit a limit of the index,
+as a table of thousands of entries. Its index then comes from the file
+with each top-level var value that holds only data blanked. Function bodies
+and values with a call or a function literal stay, so the calls of a
+generated file, as a `.pb.go` file, stay in the graph. The index keeps the
+hash of the file. If the skeleton still hits a limit, the file stays
+truncated, and reach counts it as budget.
+
+The index leaves out a Go file of a unit only when no Go build reads it by
+default: a `_test.go` file, a file with `//go:build ignore`, and a file
+whose constraint needs a tag that the go command never sets, as `purego`.
+It keeps the files of every GOOS and GOARCH, so a reach answer holds for
+each platform that a project builds for. `Result.Excluded` lists the files
+that it leaves out.
+
 When the context ends, `Run` stops, marks each file that it did not index
 as `missing` and returns the result so far with the error of the context.
 The link step takes the partial result, and the reach query reports each
@@ -147,8 +163,70 @@ of the highest confidence.
 
 A call that the link step cannot resolve is an unresolved site with a
 reason: `computed-name`, `dynamic-import`, `unknown-receiver`,
-`no-definition`, `ambiguous` or `limit`. The site keeps the method name of
-the call, so the reach query can tell which targets it concerns.
+`no-definition`, `ambiguous`, `limit`, `interface-receiver`,
+`trait-receiver` or `no-dependency-source`. The site keeps the method name of the call, so the
+reach query can tell which targets it concerns.
+
+A call on the result of another call, as `html.NewTokenizer(r).Next()`,
+needs the type of that result. When the link step does not know it, the
+site is a record with the method name, so a target of that method is
+`uncertain` and not `no-such-target`.
+
+### Result types
+
+The Go index records the type name of each result of a function, by
+position, and each name that an assignment or a short variable declaration
+binds to a result, as `z` in `z := html.NewTokenizer(r)` and `err` in
+`n, err := html.Parse(r)`. The blank name `_` binds nothing. The link step
+reads the definition of the function, in the project or in the unit of
+dependency code that gives its package, and binds the type of the result
+to the name. A call on the name then gets the external key of the method
+of the type, as `golang.org//x//net//html//Tokenizer//Next`, and a package
+edge joins the key to the definition in the unit. A chained call, as
+`pkg.F().M()` or `a.B().C()`, takes the type of each result in turn.
+
+| Result | What the call gets |
+| --- | --- |
+| A type of a unit, or of the project | The edge to the method of the type |
+| An interface type | The `cha` edges to the implementations in the graph. An implementation in a unit gets its edge through its external key. With no implementation, the site is a record with the reason `interface-receiver` |
+| A function of a package that no unit gives | A record with the reason `no-dependency-source` |
+| A result with no named type, or a function that the unit does not define | A record with the reason `unknown-receiver` |
+| Two definitions of the function, as in two platform files of a Go package, with two result types | A record with the reason `unknown-receiver`. The walk follows one type, and a method of the other type would be `not-found` |
+
+Rust follows the same rules, with these additions.
+
+- The link step links the code of a `cargo:` unit, as the crate directory
+  in the cargo registry or in `vendor/`. The root module of the crate is
+  the unit, so two versions of one crate stay apart. The external key of
+  an item is the path that another crate writes, and an item of a private
+  module that the root gives on with `pub use` gets the shortest path, as
+  `smallvec//SmallVec`.
+- A Cargo.toml can rename a dependency, as `helper = { package = "kit" }`.
+  A path of the crate writes `helper`, and the link step reads it as the
+  crate `kit` of the unit. The external node keeps the written name, as
+  `helper//serve`, and the package edge goes on to the definition in the
+  unit, whose external key is `kit//serve`.
+- The index records the return type of each function and method, with
+  `Self` as the type of its `impl`. A type drops `&`, `&mut`, `Box<..>`,
+  `Rc<..>`, `Arc<..>` and its type arguments, so `&mut SmallVec<[u8; 4]>`
+  is `SmallVec`. An `Option<T>` or a `Result<T, E>` result keeps `T` in
+  `Returns` and its wrapper in `Wraps`. `unwrap` and `expect` give `T`,
+  and so does a `?`, which the index does not record. A call of another
+  method of the wrapper, as `map`, has no type.
+- A declared type of a parameter, a field or a local that a unit gives
+  binds the receiver. An associated function, as `Type::from_vec(v)`,
+  binds the type that it returns, and `clone`, `to_owned`, `default` and
+  `from` on a type of a unit keep the type also when the crate derives
+  them.
+- A trait object or a generic bound `T: Trait` of a unit gets the `cha`
+  edges to the implementations in the graph, through their external keys.
+  With no implementation, the site is a record with the reason
+  `trait-receiver`.
+
+The standard library is a unit too. Pass `GOROOT/src` as the unit
+`go:std@<version>`, as `-deps $(go env GOROOT)/src=go:std@go1.25.1`. It gives
+each import path whose first element has no dot, as `net/http`. The index
+step leaves out the `cmd` module of it, which no program imports.
 
 ## Overlay
 
@@ -263,6 +341,14 @@ the key, so a query never holds a key that no node can have.
 | `reach.TargetByKey("yaml//load")` | The key, as a configuration or a flag writes it | `yaml//load` |
 | `reach.TargetByImport(langcode.Go, "os/exec", "Command")` | The module as the source of the language writes it, and the names under it | `os//exec//Command` |
 
+A target also matches each function, method and constructor of a unit of
+dependency code whose `Node.External` is the key, as the unexported
+`go:golang.org/x/net@v0.61.0//html/render.go//render1` for
+`golang.org//x//net//html//render1`. A call inside a unit goes to the
+definition and not to the external key, so a target that only the code of
+its unit calls is found through the definition. The call sites of a
+target leave out the package edges, which join a key to its definition.
+
 `reach.TargetsByKey` reads a list of keys. `Target.Key` returns the key,
 and the JSON form of a target is its key. One breadth-first search from
 the roots, and from the sources of the query, serves every target of a
@@ -277,7 +363,7 @@ what the search did.
 | `found` | A path from a root reaches the target. `Path` holds the shortest one |
 | `not-found` | No path reaches the target, and no uncertainty concerns it |
 | `uncertain` | No path reaches the target, and an uncertainty concerns it |
-| `no-such-target` | The key names no external node of the graph, and no uncertainty could name it |
+| `no-such-target` | The key names no external node and no definition of a unit of the graph, and no uncertainty could name it |
 
 `reach.Result.String` writes a result as lines, and `reach.PathString`
 writes a path as `main.py:3 -> handlers.py:12 -> yaml//load`.

examples/projectgraph/main.go

@@ -13,8 +13,10 @@
 // -import-dir venv/site-packages -deps venv/site-packages/yaml=pypi:PyYAML@6.0.1,
 // keeps its path under the import directory, as yaml/__init__.py. Any
 // other -deps directory is an import directory of its own, and a pypi
-// package given that way is its package. With -summary it prints the
-// summary of each such unit as JSON.
+// package given that way is its package. The Go standard library is a unit
+// too, as -deps $(go env GOROOT)/src=go:std@go1.25.1, and so is a Rust
+// crate, as -deps vendor/smallvec=cargo:smallvec@1.13.2. With -summary it
+// prints the summary of each such unit as JSON.
 package main
 
 import (

fileindex/index.go

@@ -19,7 +19,7 @@ import (
 // Version is the version of the index records and of the rules that fill
 // them. Bump it when a field, a kind, an order or a fill rule changes. A
 // store keys an index by Version, so the bump makes every older index miss.
-const Version = "11"
+const Version = "14"
 
 // DefaultTestPaths are the globs that mark the test files of the common
 // layouts, as projectgraph.Options.TestPaths takes them. A glob with no "/"
@@ -88,6 +88,16 @@ const (
 	UnresolvedNoDefinition    UnresolvedReason = "no-definition"
 	UnresolvedAmbiguous       UnresolvedReason = "ambiguous"
 	UnresolvedLimit           UnresolvedReason = "limit"
+	// UnresolvedInterfaceReceiver is a call on a value of an interface
+	// type that no type of the graph implements, so its code is elsewhere.
+	UnresolvedInterfaceReceiver UnresolvedReason = "interface-receiver"
+	// UnresolvedTraitReceiver is a call on a Rust trait object or a value
+	// of a generic bound that no type of the graph implements.
+	UnresolvedTraitReceiver UnresolvedReason = "trait-receiver"
+	// UnresolvedNoDependencySource is a call on a value whose type comes
+	// from a package with no code in the graph, as the result of a
+	// function of a module that no unit gives.
+	UnresolvedNoDependencySource UnresolvedReason = "no-dependency-source"
 )
 
 type FlagKind string
@@ -204,8 +214,12 @@ type Export struct {
 //
 // Arity, Variadic and Returns hold the signature of a Go function or
 // method: the count of its parameters, a last parameter of the form ...T,
-// and the type name of its first result. Static marks a static method of
-// Java.
+// and the type name of its first result. Results holds the type name of
+// each result of a Go function with two or more results, by position, and
+// "" for a result with no named type. Wraps names the Option or the Result
+// that holds the result of a Rust function, and Returns then names the
+// type inside, as Thing for -> Option<Thing>. Static marks a static method
+// of Java.
 type Definition struct {
 	Key        string   `json:"key"`
 	Kind       DefKind  `json:"kind"`
@@ -221,6 +235,8 @@ type Definition struct {
 	Arity      int      `json:"arity,omitempty"`
 	Variadic   bool     `json:"variadic,omitempty"`
 	Returns    string   `json:"returns,omitempty"`
+	Results    []string `json:"results,omitempty"`
+	Wraps      string   `json:"wraps,omitempty"`
 	Static     bool     `json:"static,omitempty"`
 	Flags      []Flag   `json:"flags,omitempty"`
 }
@@ -250,12 +266,14 @@ type Signature struct {
 // Typed is a parameter, a field or a local variable with the type that it
 // declares, as parser in run(parser: Parser). Type is the type name as
 // written, with no type arguments and no pointer, as Parser or ns.Parser.
-// Result marks a name that holds the first result of a call. Type then
-// names the function, as NewParser in p := NewParser().
+// Result marks a name that holds a result of a call. Type then names the
+// function, as NewParser in p := NewParser(), and Index is the position of
+// the result, as 1 for err in p, err := NewParser().
 type Typed struct {
 	Name   string `json:"name"`
 	Type   string `json:"type"`
 	Result bool   `json:"result,omitempty"`
+	Index  int    `json:"index,omitempty"`
 }
 
 // Call is one call site. Caller is the definition key that holds the site.
@@ -600,6 +618,8 @@ func (ix *Index) EachString(fn func(string) string) {
 		typed(def.Params)
 		typed(def.Locals)
 		kind(&def.Returns)
+		strs(def.Results)
+		kind(&def.Wraps)
 		flags(def.Flags)
 	}
 	for i := range ix.Classes {

fileindex/testdata/index.json

@@ -1,5 +1,5 @@
 {
-  "version": "11",
+  "version": "14",
   "path": "src/app.py",
   "language": "python",
   "content_hash": "a172cedcae47474b615c54d510a5d84a8dea3032e958587430b413538be3f333",

plugin/indexer/golang.go

@@ -184,7 +184,13 @@ func goSignature(p *projection, def *fileindex.Definition, node *sitter.Node) {
 	var params []fileindex.Typed
 	params, def.Arity, def.Variadic = p.goParams(node.ChildByFieldName("parameters"), p.goConstraints(node))
 	def.Params = append(def.Params, params...)
-	def.Returns = p.goResult(node.ChildByFieldName("result"))
+	results := p.goResults(node.ChildByFieldName("result"))
+	if len(results) > 0 {
+		def.Returns = results[0]
+	}
+	if len(results) > 1 {
+		def.Results = results
+	}
 	def.Locals = p.goLocals(node.ChildByFieldName("body"))
 }
 
@@ -253,20 +259,26 @@ func (p *projection) goParams(list *sitter.Node, constraints map[string]string)
 	return typed, arity, variadic
 }
 
-// goResult returns the type name of the first result of a function.
-func (p *projection) goResult(result *sitter.Node) string {
+// goResults returns the type name of each result of a function, by
+// position. A result with no named type, as int or func(), gives "".
+func (p *projection) goResults(result *sitter.Node) []string {
 	if result == nil {
-		return ""
+		return nil
 	}
 	if result.Type() != "parameter_list" {
-		return p.goTypeName(result)
+		return []string{p.goTypeName(result)}
 	}
+	var out []string
 	for _, param := range ts.NamedChildren(result) {
-		if param.Type() == "parameter_declaration" {
-			return p.goTypeName(param.ChildByFieldName("type"))
+		if param.Type() != "parameter_declaration" {
+			continue
+		}
+		typ := p.goTypeName(param.ChildByFieldName("type"))
+		for range max(len(childrenByField(param, "name")), 1) {
+			out = append(out, typ)
 		}
 	}
-	return ""
+	return out
 }
 
 // goLocals returns the local variables of a function body whose type the
@@ -297,21 +309,32 @@ func (p *projection) goLocals(body *sitter.Node) []fileindex.Typed {
 }
 
 // goAssigned returns the typed names of an assignment. One call on the
-// right of more names gives its first result to the first name.
+// right of more names gives each name the result at its position, and the
+// blank name _ takes none.
 func (p *projection) goAssigned(names []*sitter.Node, right *sitter.Node) []fileindex.Typed {
 	if right == nil {
 		return nil
 	}
 	values := ts.NamedChildren(right)
 	if len(values) == 1 && len(names) > 1 && values[0].Type() == "call_expression" {
-		names = names[:1]
+		t, ok := p.goValueType(values[0])
+		if !ok || !t.Result {
+			return nil
+		}
+		var typed []fileindex.Typed
+		for i, name := range names {
+			if name.Type() == "identifier" && p.text(name) != "_" {
+				typed = append(typed, fileindex.Typed{Name: p.text(name), Type: t.Type, Result: true, Index: i})
+			}
+		}
+		return typed
 	}
 	if len(values) != len(names) {
 		return nil
 	}
 	var typed []fileindex.Typed
 	for i, name := range names {
-		if name.Type() != "identifier" {
+		if name.Type() != "identifier" || p.text(name) == "_" {
 			continue
 		}
 		if t, ok := p.goValueType(values[i]); ok {

plugin/indexer/golang_test.go

@@ -29,6 +29,7 @@ func TestFileIndexGo(t *testing.T) {
 		"  run(c.Add, k, f)",
 		"}",
 		"func Apply[T Saver, U any](t T, u U) { b := NewBox[int](1) }",
+		"func Pair() (_ *store.DB, ok, done bool) { _, x := New(\"y\"); return nil, x == nil, true }",
 	}, "\n") + "\n"
 	ix := indexSource(t, core.LanguageCodeGo, "shop.go", source)
 
@@ -51,6 +52,7 @@ func TestFileIndexGo(t *testing.T) {
 		assert.Equal(t, 2, def.Arity)
 		assert.True(t, def.Variadic)
 		assert.Equal(t, "Cart", def.Returns)
+		assert.Equal(t, []string{"Cart", ""}, def.Results, "a result of a predeclared type has no name")
 	})
 	t.Run("a method has its receiver first and its typed locals", func(t *testing.T) {
 		def := definitionOf(t, ix, "shop.go//Cart//Add")
@@ -60,12 +62,19 @@ func TestFileIndexGo(t *testing.T) {
 		assert.Equal(t, []fileindex.Typed{
 			{Name: "s", Type: "Saver"},
 			{Name: "d", Type: "New", Result: true},
+			{Name: "err", Type: "New", Result: true, Index: 1},
 			{Name: "e", Type: "store.DB"},
 			{Name: "f", Type: "Cart"},
 			{Name: "g", Type: "Cart"},
 			{Name: "h", Type: "store.Open", Result: true},
 		}, def.Locals)
 	})
+	t.Run("each result has its position and the blank name binds none", func(t *testing.T) {
+		def := definitionOf(t, ix, "shop.go//Pair")
+		assert.Equal(t, "store.DB", def.Returns)
+		assert.Equal(t, []string{"store.DB", "", ""}, def.Results)
+		assert.Equal(t, []fileindex.Typed{{Name: "x", Type: "New", Result: true, Index: 1}}, def.Locals)
+	})
 	t.Run("a type parameter stands for its constraint", func(t *testing.T) {
 		def := definitionOf(t, ix, "shop.go//Apply")
 		assert.Equal(t, []fileindex.Typed{{Name: "t", Type: "Saver"}}, def.Params)