feat(modgraph): the module graph of a Go module, with no Go toolchain

README.md

@@ -13,3 +13,6 @@ See [docs/detection.md](./docs/detection.md) for the APIs that security detectio
 See [docs/projectgraph.md](./docs/projectgraph.md) for the project graph: the call graph of a
 whole project across its files and into its dependencies, with the index, link, roots, reach and
 summary steps.
+
+See [docs/modgraph.md](./docs/modgraph.md) for the module graph of a Go module: which module
+imports which, from the code on the disk, with no Go toolchain.

docs/modgraph.md

@@ -0,0 +1,145 @@
+# Module graph
+
+`modgraph` builds the module graph of a Go module: which module of the build
+list imports which. A `go.mod` lists the modules of a build, but not the
+edges between them. A caller needs the edges to tell that no code imports a
+module, or that no other module can call one.
+
+The package reads the code on the disk. It runs no `go` command, opens no
+network connection and builds with `CGO_ENABLED=0`. It parses each Go file
+with `go/parser` in `ImportsOnly` mode. It reads the first 64 KiB of a file,
+and reads the whole file when that head does not show the end of the import
+block: a token after the last import that is not `import`, and that ends
+inside the head.
+
+## The rule
+
+The graph may hold an edge that a build does not have. It never misses an
+edge that a build has. A caller can then trust the absence of a path: no
+build of any platform or tag set can reach a module that the graph does not
+reach.
+
+So a module whose imports `Build` cannot read is **open**: it may import any
+module, and a path that meets it reaches each module.
+
+| State | The module | Edges |
+| --- | --- | --- |
+| `imports` | Has source. `Build` read each package of it that the build reaches | The imports of those packages |
+| `requires` | Has no source, only its `go.mod` | Its require lines, as evidence only. A `go.mod` need not list each module that its code imports, because the main `go.mod` can give it as `// indirect`. So the node is open |
+| `open` | Has nothing that `Build` can read, or an import that `Build` cannot resolve | None |
+
+## The walk
+
+1. `Build` reads each package of the main module, its test files too, as
+   `./...` names them. It skips `testdata`, `vendor`, a directory or file
+   whose name starts with `_` or `.`, and a directory with a `go.mod` of its
+   own. A link to a file is a file of its package.
+2. For each import, the resolver takes the modules of the build list whose
+   path is a prefix of the import path, longest first. A module with source
+   holds the package when its directory has an entry whose name ends in
+   `.go` and that is a file or a link to a file, as the go command decides
+   it. A module with no source may hold it, so it gets an edge too, and the
+   test goes on to a shorter prefix.
+   A path that no module holds is the standard library when its first
+   element has no dot. Else its importer is open.
+3. The walk reads each package that an import reaches, as `go list -deps`
+   does, a `testdata` package or a package behind a link to a directory too.
+   A package of a dependency that no import reaches adds nothing. Test files
+   of a dependency never count.
+4. A file counts unless its build constraint can never hold, with every tag
+   free except `ignore`. `Build` reads the constraint as `go/build` does: a
+   `//go:build` line in the leading comments decides, else each `// +build`
+   line before the last blank line of the leading `//` comments. A file of
+   `//go:build purego` counts, because a project can build with
+   `-tags purego`. A GOOS or GOARCH suffix of the file name never drops a
+   file.
+5. Once a reached node is open, it can import any package of the build
+   list. So each node counts as reached, and `Build` reads each directory
+   of each module with source that an import can name, except `vendor` and
+   a nested module. Else a module of one read package would hide the edges
+   of its other packages. A module with a link to a directory is then open,
+   because the walk does not follow the link. A module zip holds no links.
+   This read skips a package that does not parse, or that imports a path
+   that no module of the build list holds. No build of the project reads
+   such a package, because `go mod tidy` lists in the main `go.mod` each
+   module that a package of the build imports. In the walk from the main
+   module, such a package makes its module open.
+
+A main `go.mod` with a `go` directive below 1.17, or with none, gives
+`ErrUnpruned`. Before module graph pruning, `go.mod` need not list each
+module of the build.
+
+## Use
+
+```go
+sources, err := modgraph.Discover(modgraph.DiscoverOptions{
+    Dir:         projectDir,  // the directory of the main go.mod
+    ModuleCache: moduleCache, // GOMODCACHE; the caller finds it
+    LocalRoot:   scanRoot,    // a local replace directive is read only inside it
+})
+if err != nil {
+    return err
+}
+g, err := modgraph.Build(ctx, modgraph.Input{
+    MainFS:  os.DirFS(projectDir),
+    Modules: sources,
+    Store:   store, // optional cache of the packages of the module cache
+})
+if err != nil {
+    return err
+}
+path := g.Path([]string{g.Main.Path}, "github.com/x/y") // nil when no path reaches it
+```
+
+`Graph.Path` and `Graph.Reaches` take a set of start modules. A path that
+meets an open node, or a module that `Build` did not walk (`Node.Walked` is
+false), can reach each module: the path ends at that node.
+
+`Discover` finds each module as the go command builds it: `vendor` when
+`vendor/modules.txt` lists the module at its version, then the local
+directory of a `replace` directive, inside `LocalRoot` after links are
+resolved, then the module cache, then the `.mod` file of the module cache.
+Each read of a local replacement resolves its links again, and a link that
+leads out of `LocalRoot` makes the module open.
+A caller that finds the sources in another way can pass its own
+`[]Source`. `Module.Path` is the path that import statements use, also for
+a replaced module, and `Source.Origin` is the module that the source holds.
+
+`Discover` parses the main `go.mod` in full, because a lax parse drops the
+`replace` directives. A directive that `golang.org/x/mod` does not know
+fails the parse, and the caller keeps no graph.
+
+`examples/modgraph` prints the graph of a module. Run `go mod download`
+first: a module with no source on the disk is open, and one open node that
+the build reaches makes each module reached.
+
+## Cache
+
+`Input.Store` caches the imports of each package of an immutable source,
+the module cache. The key is the module that the source holds
+(`Source.Origin` for a replacement), its version, the directory and the
+`go.sum` hash. A
+`vendor` directory and a local replacement can change, so they are read on
+each run. `Build` stores no entry for a package that it could not read in
+full.
+
+## Oracle test
+
+`TestBuildHoldsEachEdgeOfGoList` runs `go list -deps -test` on this
+repository for four platforms and checks that the graph holds each module
+edge, or that the edge leaves an open node. `MODGRAPH_ORACLE_DIRS` adds more
+projects, and a project with `vendor/modules.txt` reads its dependencies
+from `vendor`. The test needs the `go` command and a module cache, and skips
+with neither. It sets `GOPROXY=off`, so it downloads nothing.
+
+## Not covered
+
+- `go.work` workspaces. A caller keeps no graph for a workspace.
+- A file larger than 32 MiB whose first 64 KiB does not parse makes its
+  module open.
+- A file budget, 200,000 files by default, caps a run. Past it the module
+  that is next is open, and `Stats.Truncated` is set.
+- `Build` tests a build constraint one tag set at a time. A constraint of
+  more than 16 tags counts as one that can hold. After 4,194,304 tested
+  sets in one run, each constraint counts as one that can hold. So a
+  dependency of many such files cannot make a run slow.

examples/modgraph/main.go

@@ -0,0 +1,81 @@
+// Command modgraph prints the module graph of a Go module: each module of
+// the build list, what Build knows of its imports, and the edges between
+// them. It finds the source of each module with modgraph.Discover in the
+// vendor directory and in the module cache, which -modcache names, or
+// $GOMODCACHE, or $GOPATH/pkg/mod. It runs no go command and downloads
+// nothing. Run go mod download first for a graph with no open node.
+//
+//	go run ./examples/modgraph -dir ../vet
+//	go run ./examples/modgraph -dir ../vet -json
+package main
+
+import (
+	"cmp"
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+
+	"github.com/safedep/code/modgraph"
+)
+
+func main() {
+	dir := flag.String("dir", ".", "the directory of the main go.mod")
+	modcache := flag.String("modcache", "", "the module cache, else $GOMODCACHE, else $GOPATH/pkg/mod, else $HOME/go/pkg/mod")
+	asJSON := flag.Bool("json", false, "print the graph as JSON")
+	flag.Parse()
+	if err := run(os.Stdout, *dir, *modcache, *asJSON); err != nil {
+		fmt.Fprintln(os.Stderr, "modgraph:", err)
+		os.Exit(1)
+	}
+}
+
+func run(w io.Writer, dir, modcache string, asJSON bool) error {
+	sources, err := modgraph.Discover(modgraph.DiscoverOptions{Dir: dir, ModuleCache: moduleCache(modcache), LocalRoot: dir})
+	if err != nil {
+		return err
+	}
+	g, err := modgraph.Build(context.Background(), modgraph.Input{MainFS: os.DirFS(dir), Modules: sources})
+	if err != nil {
+		return err
+	}
+	if asJSON {
+		enc := json.NewEncoder(w)
+		enc.SetIndent("", "  ")
+		return enc.Encode(g)
+	}
+	for _, n := range g.Nodes {
+		reached := "unreached"
+		if n.Reached {
+			reached = "reached"
+		}
+		if _, err := fmt.Fprintf(w, "%-9s %-9s %s %s\n", reached, n.State, n.Module, n.Reason); err != nil {
+			return err
+		}
+	}
+	for _, e := range g.Edges {
+		if _, err := fmt.Fprintf(w, "%s -> %s (%s, %s:%d)\n", e.From.Path, e.To.Path, e.Kind, e.Evidence.File, e.Evidence.Line); err != nil {
+			return err
+		}
+	}
+	_, err = fmt.Fprintf(w, "%d files, %d packages\n", g.Stats.Files, g.Stats.Packages)
+	return err
+}
+
+// moduleCache finds the module cache as the go command does, with no go
+// command.
+func moduleCache(flagValue string) string {
+	if c := cmp.Or(flagValue, os.Getenv("GOMODCACHE")); c != "" {
+		return c
+	}
+	if p := filepath.SplitList(os.Getenv("GOPATH")); len(p) > 0 && p[0] != "" {
+		return filepath.Join(p[0], "pkg", "mod")
+	}
+	if home, err := os.UserHomeDir(); err == nil {
+		return filepath.Join(home, "go", "pkg", "mod")
+	}
+	return ""
+}

examples/modgraph/main_test.go

@@ -0,0 +1,44 @@
+package main
+
+import (
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/safedep/code/modgraph"
+)
+
+func TestRunPrintsTheGraph(t *testing.T) {
+	dir := t.TempDir()
+	cache := t.TempDir()
+	for name, data := range map[string]string{
+		filepath.Join(dir, "go.mod"):                        "module example.com/app\n\ngo 1.22\n\nrequire example.com/dep v1.0.0\n",
+		filepath.Join(dir, "main.go"):                       "package main\n\nimport _ \"example.com/dep\"\n",
+		filepath.Join(cache, "example.com/dep@v1.0.0/d.go"): "package dep\n\nimport _ \"fmt\"\n",
+	} {
+		require.NoError(t, os.MkdirAll(filepath.Dir(name), 0o755))
+		require.NoError(t, os.WriteFile(name, []byte(data), 0o600))
+	}
+
+	var out strings.Builder
+	require.NoError(t, run(&out, dir, cache, false))
+	assert.Contains(t, out.String(), "reached   imports   example.com/dep@v1.0.0")
+	assert.Contains(t, out.String(), "example.com/app -> example.com/dep (imports, main.go:3)")
+	assert.Contains(t, out.String(), "example.com/dep -> std (imports, d.go:3)")
+
+	out.Reset()
+	require.NoError(t, run(&out, dir, cache, true))
+	var g modgraph.Graph
+	require.NoError(t, json.Unmarshal([]byte(out.String()), &g))
+	dep, ok := g.Node("example.com/dep")
+	require.True(t, ok)
+	assert.True(t, dep.Reached)
+	assert.Equal(t, modgraph.StateImports, dep.State)
+	assert.True(t, g.Reaches([]string{"example.com/app"}, modgraph.StdPath))
+	assert.Equal(t, []string{"example.com/app", "example.com/dep"}, g.Path([]string{"example.com/app"}, "example.com/dep"))
+}

modgraph/cache.go

@@ -0,0 +1,98 @@
+package modgraph
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/safedep/code/fileindex"
+	"github.com/safedep/code/fileindex/store"
+)
+
+// cachePlugin is the plugin name of the artifacts of Build.
+const cachePlugin = "modgraph"
+
+// errCache marks a failure of the store. It ends Build, because it says
+// nothing about the module.
+var errCache = errors.New("modgraph: cache")
+
+// cacheEntry is the artifact of one package: the imports of its Go files
+// that count. Build stores an entry only after it read each file of the
+// package with no error, so an entry never stands for a part of a package.
+type cacheEntry struct {
+	// Imports is a pointer, so an entry with no imports field is a miss, not
+	// a package with no imports.
+	Imports *[]importRef `json:"imports"`
+}
+
+// cacheKey is the key of a package of an immutable source. Its path is the
+// unit and the directory, as "go:golang.org/x/net@v0.30.0//html". The go.sum
+// hash, when the caller has it, separates two copies of one version with
+// other bytes.
+func cacheKey(ref pkgRef) store.ArtifactKey {
+	m := ref.mod.src.Origin
+	if m.Path == "" {
+		m = ref.mod.src.Module
+	}
+	unit := fileindex.PackageUnit{Ecosystem: "go", Name: m.Path, Version: m.Version}.String()
+	return store.ArtifactKey{
+		Key:     fileindex.Key{Path: fileindex.UnitPath(unit, ref.dir), ContentHash: ref.mod.src.Sum},
+		Plugin:  cachePlugin,
+		Version: Version,
+	}
+}
+
+// cacheable reports whether the imports of a package can come from, and go
+// to, the cache.
+func (b *builder) cacheable(ref pkgRef) bool {
+	if b.in.Store == nil || ref.mod.main || !ref.mod.src.Immutable {
+		return false
+	}
+	origin := ref.mod.src.Origin
+	if origin.Path == "" {
+		origin = ref.mod.src.Module
+	}
+	return origin.Version != ""
+}
+
+// cached returns the imports of a package from the cache.
+func (b *builder) cached(ref pkgRef) ([]importRef, bool, error) {
+	if !b.cacheable(ref) {
+		return nil, false, nil
+	}
+	k := cacheKey(ref)
+	found, err := b.in.Store.GetArtifacts(b.ctx, []store.ArtifactKey{k})
+	if err != nil {
+		return nil, false, fmt.Errorf("%w: read %s: %w", errCache, k.Path, err)
+	}
+	data, ok := found[k]
+	if !ok {
+		return nil, false, nil
+	}
+	var e cacheEntry
+	if err := json.Unmarshal(data, &e); err != nil || e.Imports == nil {
+		// A bad entry is a miss. The read that follows writes a good one.
+		return nil, false, nil
+	}
+	b.stats.Packages++
+	b.stats.CachedPackages++
+	return *e.Imports, true, nil
+}
+
+// cache stores the imports of a package that Build read whole.
+func (b *builder) cache(ref pkgRef, imps []importRef) error {
+	if !b.cacheable(ref) {
+		return nil
+	}
+	if imps == nil {
+		imps = []importRef{}
+	}
+	data, err := json.Marshal(cacheEntry{Imports: &imps})
+	if err != nil {
+		return fmt.Errorf("encode the cache entry of %s: %w", ref.dir, err)
+	}
+	if err := b.in.Store.PutArtifacts(b.ctx, []store.Artifact{{Key: cacheKey(ref), Data: data}}); err != nil {
+		return fmt.Errorf("%w: write %s: %w", errCache, ref.dir, err)
+	}
+	return nil
+}