Skip to content

Rewrite the CSV file through a private copy next to it and never replace it with a fragment - #131

Merged
vharseko merged 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:csv-temp-files-and-locks
Oct 5, 2026
Merged

vharseko merged 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:csv-temp-files-and-locks

Conversation

@vharseko

@vharseko vharseko commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Closes the medium CodeQL alerts java/local-temp-file-or-directory-information-disclosure #20 #21 #22 and java/unreleased-lock #15, and fixes a data-loss bug found on the way.

CSV connector: how update and delete rewrite the file

doUpdate and doDelete rebuild the whole CSV — accounts, attributes, password hashes — in a File.createTempFile("csvfile", "tmp"), i.e. in the system temp directory with the umask permissions (0644 as a rule), where any local user can read it while it is written. Then the finally block unconditionally deleted the CSV and moved the temp file over it. Two consequences, both reproduced by the new tests against the old code:

  • a rewrite that fails half-way (the tests use a row with the wrong number of columns after the row being updated) replaced the data file with the rows written so far — everything after the failure was gone;
  • the CSV came back with the temp file's permissions: a file kept at 0600 was 0644 after every update or delete.

Now:

  • createRewriteFile() creates the copy next to the CSV (<csv>.<random>.tmp, outside the <csv>.<13 digits> pattern of the sync copies) with Files.createTempFile: owner-only on POSIX, same file system, so the replacement is a rename.
  • The writer is closed inside the try, before the rewrite counts as complete, so a failing final flush (ENOSPC, EIO, NFS close) discards the copy instead of putting a truncated one in place. The copy's FileOutputStream is also closed on its own: a FileWriter whose final flush fails leaves it open on JDK 8–21, and Windows can not delete an open file.
  • finishRewrite(tmp, rewritten) replaces the CSV only when the rewrite completed: the copy is fsynced, takes over the CSV's group if the process may set it (otherwise it keeps the process's group and gets no group permissions) and the CSV's mode bits (a chmod refused by a mount that fixes the modes is logged, and the copy keeps the mode it was created with; all of this is skipped on non-POSIX file systems; the owner becomes the process's user), and moves into place with REPLACE_EXISTING + ATOMIC_MOVE (plain move as fallback); the directory is synced after the rename, best effort. A copy that is not put in place — of a failed rewrite or of a failed replacement — is deleted. A failed replacement is now a ConnectorIOException instead of a logged-and-swallowed error that left the caller believing the operation succeeded.
  • doDelete decrements the cached row count only after the replacement succeeded. Create and delete leave the count alone while it is still -1 (not counted yet), so the next paged search counts the rows; before, a delete made it -2 (no totals and no paging cookie for the life of the JVM) and a create made it 0 (paging stopped after the first page).
  • A copy left behind by a JVM killed while writing it is not swept automatically (the write lock is per JVM, another one may be rewriting the same CSV); the Javadoc and a new section of the installation chapter (chap-install-csvfile-connector.xml) say to delete such .tmp files by hand while no connector runs against the CSV.
  • lock.unlock() sits in its own inner finally in doCreate, doDelete, doUpdate (Bump commons-io:commons-io from 2.2 to 2.7 in /OpenICF-java-framework/connector-test-common #15): a RuntimeException from a close() can no longer keep the write lock forever and hang every later operation on the file. The three copies of the close-and-log code became closeQuietly.
  • FileUtils.moveFile was the only use of commons-io; the dependency is removed.

Bundle temp directory (#22)

LocalConnectorInfoManagerImpl created java.io.tmpdir/bundle-<random> with mkdir() (umask permissions) for the expanded lib/ and native/ entries of a connector bundle. It now uses Files.createTempDirectory, owner-only on POSIX; the name-probing loop goes with it.

Tests

  • RewriteSafetyTest (new, 8): update/delete keep a 0640 data file at 0640 — a mode neither a new temporary file nor a 022 umask produces (POSIX, otherwise skipped) — and leave no copy behind; a failed update/delete leaves the file byte-for-byte as it was and no copy behind; the copy is created rw------- next to the CSV; a failed delete keeps the paged-search total, and a create or delete before the first search does not corrupt it. Each test fails on the code it guards against (old code or a mutant of the fix).
  • LocalConnectorInfoManagerTests.testBundleTempDirectoryIsPrivate (new): a freshly expanded bundle directory is rwx------ (POSIX, otherwise skipped). Uses its own LocalConnectorInfoManagerImpl so the factory cache and connector pools of the other tests are untouched.
  • The unlock() ordering, a failing writer close, the separately closed stream, a failed replacement and the group copy (including the dropped group bits) have no tests: they need a close() that throws inside a private method, a failing disk, an immutable file or a group the test user is not in.

Local runs: connector-framework-internal 500 tests (2 skipped, unrelated), csvfile-connector 88 tests, all green.

Not changed

java/unreleased-lock #13 (sync: the read lock taken while downgrading from the write lock is released by the following try/finally; nothing can throw in between) and #14 (findObjectInFile: if (rwLock != null) lock … finally { if (rwLock != null) unlock }) are false positives of the heuristic and will be dismissed as such.

@vharseko vharseko added security Security fix / CVE remediation java Pull requests that update java code tests Test additions or fixes framework OpenICF-java-framework connector:csvfile CSV file connector labels Sep 18, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The rewrite now goes through a private copy next to the CSV, and a failed read-side rewrite leaves the data file alone.

  • createRewriteFile() creates the copy with Files.createTempFile in the CSV's own directory, so the copy is owner-only while it is written and the replacement is a same-file-system rename.
  • finishRewrite(tmp, rewritten) discards the copy of a half-done rewrite, and a failed replacement is now a ConnectorIOException instead of a logged error behind a reported success.
  • lock.unlock() sits in its own inner finally in doCreate/doDelete/doUpdate, and getBundleTempDir uses Files.createTempDirectory.

issue (blocking): If writer.close() fails, update and delete still replace the CSV with a truncated copy.

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1057-1058, :1068-1069, :1121, :1131-1132

rewritten = true is set while the last rows are still in the writer's buffers (the FileWriter encoder buffer, plus super-csv's BufferedWriter). Those rows reach the copy only in closeQuietly(writer, "writer"), which catches and logs any exception, and then finishRewrite(tmp, true) moves the copy over the CSV. If the final flush fails (ENOSPC, EDQUOT, EIO, or close(2) on NFS), the CSV loses its tail, or all of it when the file is smaller than the buffer, and the operation returns success. This is the "fragment" case the PR title says is fixed. RewriteSafetyTest only injects a read-side failure.

// doDelete, after the `!found` check
writer.close(); // the last rows reach the copy here; a failure must not replace the CSV
totalRowCount.put(csvFilePath, totalRowCount.get(csvFilePath) - 1);
rewritten = true;

// doUpdate, after the `updated == null` check
writer.close();
rewritten = true;

A close failure then goes to the catch (IOException e) and the copy is discarded. The second close in closeQuietly does nothing on a writer that is already closed.


issue (non-blocking): updateKeepsFilePermissions/deleteKeepsFilePermissions cannot fail when finishRewrite stops copying the CSV's permissions.

OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:117-124, CSVFileConnector.java:1171

The fixture sets the CSV to rw-------, which is exactly the mode Files.createTempFile gives the copy on POSIX. With Files.setPosixFilePermissions(tmp.toPath(), Files.getPosixFilePermissions(csv)) removed, RewriteSafetyTest stays green (4 run, 0 failures, 0 skipped, macOS). So with this mutant a group-readable CSV comes back rw------- after every update or delete, and CI would not notice. These tests are the only pin on the permission fix.

// neither Files.createTempFile (rw-------) nor a 022 umask (rw-r--r--) yields this mode
Set<PosixFilePermission> mode = PosixFilePermissions.fromString("rw-r-----");
Files.setPosixFilePermissions(csv.toPath(), mode);
return mode;

Pin: with restrictToOwner() returning rw-r-----, both tests fail against the mutant and still fail against the old /tmp + umask code.


issue (non-blocking): doDelete decrements the cached row count before the replacement, which can now fail.

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1057, :1069

When finishRewrite throws, the CSV still has the row, but the static totalRowCount has already gone down by one. It stays wrong for the life of the JVM, because the count is recomputed (:347) only while it is -1, and it is set to -1 once, at :224. Paged searches then report a wrong totalPagedResults/remainingPagedResults (:406-413). The decrement-first order is older than this PR, but the PR now reports the failure to the caller while the count records a delete. Probe: count 3 → 2 and the CSV unchanged after a failed move.

try {
    closeQuietly(reader, "reader");
    closeQuietly(writer, "writer");
    finishRewrite(tmp, rewritten);
    if (rewritten) {
        totalRowCount.put(csvFilePath, totalRowCount.get(csvFilePath) - 1);
    }
} finally {
    lock.unlock();
}

The decrement moves out of the try body (:1057).


question (non-blocking): Is the rewritten copy kept on purpose when the replacement fails?

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1181-1183

The IOException catch rethrows with the copy's name but never deletes it. Each failed replacement leaves behind a <csv>.<random>.tmp: a full copy of the accounts and password hashes, holding the change the caller was told had failed. Causes include an immutable or locked CSV, a Windows sharing violation, or a CSV removed outside the lock. The copy has the CSV's own mode, so nothing is exposed more widely than the CSV itself, but the copies pile up (probe: chflags uchg on the CSV → ConnectorIOException, copy left). If the copy is meant for recovery, say so in the Javadoc. Otherwise delete it:

} catch (IOException e) {
    if (!tmp.delete() && tmp.exists()) {
        log.warn("Could not delete {0}", tmp);
    }
    throw new ConnectorIOException("Failed to replace " + csv + " with its rewritten copy", e);
}

suggestion (non-blocking): The failed-rewrite tests do not check that the partial copy is deleted.

OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:93-115

failedUpdateLeavesFileUntouched/failedDeleteLeavesFileUntouched assert only the CSV's content. Replacing if (!tmp.delete() && tmp.exists()) with if (false) in finishRewrite keeps all 4 tests green and leaves rewrite*.csv.<n>.tmp partial copies next to the CSV.

// in both catch blocks, after the content check (static import org.testng.Assert.assertFalse)
for (String name : csv.getParentFile().list()) {
    assertFalse(name.startsWith(csv.getName() + ".") && name.endsWith(".tmp"),
            "the copy of the failed rewrite was left behind: " + name);
}

suggestion (non-blocking): finishRewrite keeps the CSV's mode bits but not its group.

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1170-1174

The copy is a new file, so its group is the process's (or the directory's), and the copied group bits apply to that group. A CSV shared as rw-rw---- with group idm comes back rw-rw---- under the process's group: members of idm lose access, and the process's group gains rw (probe on macOS: group everyone → staff). The old code also replaced the file, but the Javadoc now promises "keeping the CSV's permissions".

try {
    try {
        Files.setAttribute(tmp.toPath(), "posix:group", Files.getAttribute(csv, "posix:group"));
    } catch (IOException e) {
        // not a member of the CSV's group: the copy keeps the process's group
    }
    Files.setPosixFilePermissions(tmp.toPath(), Files.getPosixFilePermissions(csv));
} catch (UnsupportedOperationException e) {
    // not a POSIX file system: the copy inherits the directory's ACL
}

Or: narrow the Javadoc to "the CSV's mode bits".

@vharseko
vharseko force-pushed the csv-temp-files-and-locks branch from 0c1847d to b224469 Compare October 2, 2026 18:54
@vharseko

vharseko commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

The branch is rebased onto current master (the conflict was in the imports and test methods of LocalConnectorInfoManagerTests, next to the zip-slip test from #126; both tests kept). All six points are taken.

writer.close() failure (blocking) — doDelete and doUpdate now close the writer explicitly in the try body, before rewritten = true. A failing final flush goes to the catch (IOException e), the copy is discarded, and the caller gets a ConnectorIOException; the closeQuietly(writer, …) in finally is a no-op on the closed writer.

Beyond the review: the copy is now also fsynced (FileChannel.force(true)) before the rename, so a crash right after the rename can not leave the CSV name on a copy whose data never reached the disk (the empty-file-after-rename case on delayed-allocation file systems). A failing sync discards the copy like any other replacement failure. The directory is synced after the rename as well, best effort: Windows can not open a directory, and a lost rename only undoes the operation.

Permission tests — the fixture mode is now rw-r-----. Checked against the mutant without setPosixFilePermissions: both *KeepsFilePermissions tests fail (expected [OWNER_READ, OWNER_WRITE, GROUP_READ] but got [OWNER_READ, OWNER_WRITE]).

totalRowCount — the decrement moved behind finishRewrite, under if (rewritten), so a failed close, sync or move leaves the count as it was.

Copy left after a failed replacement — not intended. The IOException catch in finishRewrite now deletes the copy before throwing, and the exception message no longer names it.

Partial copy check — both failed*LeavesFileUntouched tests now assert that no <csv>.*.tmp is left next to the CSV. Checked against the mutant that skips the delete: both fail in assertNoRewriteFileLeft.

Group — finishRewrite copies posix:group from the CSV before the mode bits; when the process is not a member of that group the copy keeps the process's group. The Javadoc now says what is kept: the mode bits, the group if the process may set it, and that the owner becomes the process's user.

Not covered by tests: the close failure, a failed replacement and the group copy — each needs a failing disk, an immutable file or a second group membership, none of which a unit test can set up portably.

Local runs: csvfile-connector 84 tests, connector-framework-internal 500 tests (2 skipped: testConfigurationUpdate, unrelated), all green.

@vharseko
vharseko requested a review from maximthomas October 2, 2026 18:55
@vharseko vharseko added bug Something isn't working data-loss Fixes or risks loss or corruption of stored data dependencies Pull requests that update a dependency file concurrency Races, locking and thread-safety fixes labels Oct 2, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: every round-1 point landed where the bug was, and the rewrite now fails closed end to end.

  • writer.close() runs before rewritten = true in doDelete (CSVFileConnector.java:1060) and doUpdate (:1128), so a failing final flush discards the copy instead of replacing the CSV with it.
  • finishRewrite syncs the copy (FileChannel.force(true), :1181-1183) before the rename and deletes it on any failed replacement (:1201-1203).
  • setDistinctMode() (RewriteSafetyTest.java:120) uses rw-r-----, a mode Files.createTempFile never produces, so the *KeepsFilePermissions tests now fail when the mode copy is dropped.

question (non-blocking): when the group cannot be copied, should the CSV's group bits still be applied to the copy?

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1186-1191

The posix:group copy fails with EPERM when the process user is not in the CSV's group, and the empty catch swallows it. setPosixFilePermissions then still copies the CSV's group bits, so they now apply to the process's group (Linux: the egid; BSD/macOS: the directory's group). Example: a CSV svc:hr rw-r----- whose owner svc is not in hr comes back svc:<svc's group> rw-r-----, and members of svc's group can read the account data and password hashes. Probe on macOS: CSV wheel rw-r-----, chown EPERM swallowed, copy ends staff rw-r-----. This is still narrower than master (0644). If granting these bits is intended, it is worth one sentence in the Javadoc. If not, drop the group bits:

// import java.nio.file.attribute.PosixFilePermission;
Set<PosixFilePermission> mode = Files.getPosixFilePermissions(csv);
try {
    Files.setAttribute(tmp.toPath(), "posix:group", Files.getAttribute(csv, "posix:group"));
} catch (IOException e) {
    // not a member of the CSV's group: the copy keeps the process's group,
    // which must not get the access the CSV grants to its own group
    mode.remove(PosixFilePermission.GROUP_READ);
    mode.remove(PosixFilePermission.GROUP_WRITE);
    mode.remove(PosixFilePermission.GROUP_EXECUTE);
}
Files.setPosixFilePermissions(tmp.toPath(), mode);

suggestion (non-blocking): no test pins the CodeQL #20/#21 fix itself: that the copy is owner-only and next to the CSV while it is written.

OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:57, CSVFileConnector.java:1155-1158

The fixture CSV is in java.io.tmpdir. The mode tests read only the final mode, which finishRewrite copies onto any copy, and assertNoRewriteFileLeft matches only <csv>.*.tmp. A mutant that turns createRewriteFile back into BASE's File.createTempFile("csvfile", "tmp") (system temp dir, umask 0644) keeps the whole csvfile module green: 15 classes, RewriteSafetyTest 4/4. With createRewriteFile() made package-private:

@Test
public void rewriteCopyIsPrivateAndNextToTheFile() throws Exception {
    setDistinctMode(); // skips where POSIX permissions are not supported
    File tmp = connector.createRewriteFile();
    try {
        assertEquals(tmp.getParentFile(), csv.getAbsoluteFile().getParentFile());
        assertEquals(Files.getPosixFilePermissions(tmp.toPath()),
                PosixFilePermissions.fromString("rw-------"));
    } finally {
        tmp.delete();
    }
}

Pin: the mode assertion kills the BASE-creator mutant. The parent assertion only bites once the fixture CSV is created in its own Files.createTempDirectory and not directly in java.io.tmpdir.


suggestion (non-blocking): the success-path tests do not check that the copy is gone, and nothing pins the guarded row-count decrement.

OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:77-92, CSVFileConnector.java:1072-1075, :1196-1199

Two mutants keep the whole csvfile module green (15 classes). In the first, both Files.move calls in finishRewrite become Files.copy(tmp.toPath(), csv, REPLACE_EXISTING): every successful update and delete then leaves a full <csv>.<n>.tmp next to the CSV. In the second, the decrement becomes unconditional and runs before finishRewrite: an unknown-uid or failed delete lowers the paging count, and no test reads it.

connector.update(ObjectClass.ACCOUNT, new Uid("vilo"), lastName("updated"), null);

assertEquals(Files.getPosixFilePermissions(csv.toPath()), mode);
assertNoRewriteFileLeft(); // same line in deleteKeepsFilePermissions: kills the copy-not-move mutant

Pin: for the decrement, run a paged search so that the count is known, then a failed delete, then a second paged search, and assert that the total in its SearchResult is unchanged.


issue (non-blocking): the row-count updates still ignore the -1 "not counted yet" sentinel.

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1072-1075, :1008

init() stores -1 (:226). The recount runs only on == -1 (:349), and the paging totals are reported only on > -1 (:408). A delete before the first counting search stores -2: from then on, for the life of the JVM, nothing recounts and no SearchResult with totals or a cookie is returned. A create first stores 0, which reads as a known count of zero, so the total is reported as 0 and paging stops after the first page. This predates the PR (BASE :1052, :1002), but the PR already touches the delete line.

if (rewritten) {
    Integer count = totalRowCount.get(csvFilePath);
    if (count > -1) {
        totalRowCount.put(csvFilePath, count - 1);
    }
}
// and the same `> -1` guard around the increment in doCreate (:1008)

issue (non-blocking): on JDK 8-21, a failing final flush leaves the copy's file handle open, so on Windows the partial copy cannot be deleted.

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1042, :1102, :1060, :1128, :1208-1212

writer.close() closes the FileWriter. On JDK 8/11/17/19/21, StreamEncoder.implClose() calls out.close() inside its try after the last writeBytes(), so when that write throws (ENOSPC, EIO) the FileOutputStream stays open. JDK 24+ uses try (out) (checked in each JDK's src.zip). The second closeQuietly(writer, …) is a no-op. On Windows, deleteRewriteFile then fails on the open handle, logs "Could not delete", and leaves a partial copy of the accounts next to the CSV. Not run: needs a Windows host with a failing disk. On POSIX only the descriptor leaks until GC.

// import java.io.FileOutputStream; import java.io.OutputStreamWriter;
FileOutputStream copy = null;
...
copy = new FileOutputStream(tmp);
writer = new CsvMapWriter(new OutputStreamWriter(copy), csvPreference); // same default charset as FileWriter
...
} finally {
    try {
        closeQuietly(reader, "reader");
        closeQuietly(writer, "writer");
        closeQuietly(copy, "copy");
        finishRewrite(tmp, rewritten);

question (non-blocking): is a refused chmod on the copy meant to fail every update and delete?

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1191, :1201-1204

The group copy is best effort, but the mode copy tolerates only UnsupportedOperationException. Its IOException reaches the outer catch, which deletes the copy and throws ConnectorIOException. Consider a mount where every file belongs to one fixed uid that is not the connector's (vfat/exfat uid=…,umask=000 without quiet, or CIFS uid=… without noperm). There the copy is not owned by the process, chmod gets EPERM, and every rewrite fails; master worked on such mounts. Not run: needs such a mount. EPERM for a non-owner chmod surfaces as FileSystemException (probed). If the copy is left at rw-------, nothing is exposed. If failing closed is not intended:

try {
    Files.setPosixFilePermissions(tmp.toPath(), Files.getPosixFilePermissions(csv));
} catch (IOException e) {
    log.warn(e, "Could not copy the permissions of {0}: the rewritten file is owner-only", csv);
}

suggestion (non-blocking): say that a copy left by a killed JVM is not cleaned up.

OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1157

If the JVM dies (SIGKILL, OOM kill, power loss) between createRewriteFile and Files.move, the copy, which holds every account and password hash, stays next to the CSV permanently. It is owner-only. scrubSyncFiles and getLatestSyncToken match only <csv>.<13 digits>, and nothing scans for .tmp. On master, such leftovers went to java.io.tmpdir, which OS cleaners empty. A startup sweep would race another JVM rewriting the same CSV (the write lock is per JVM), so documenting it is the safer fix:

 * A copy left behind by a process that died while writing it is not removed
 * automatically: delete {@code <csv>.<digits>.tmp} files by hand while no
 * connector is running against the CSV.

vharseko added a commit that referenced this pull request Oct 4, 2026
…tors (#133)

Closes 35 of the 37 open CodeQL alerts of severity `warning` without a
security rating. Not in here: `java/constant-comparison` #1541
(`ObjectPool`, already fixed in #126) and `java/unsafe-get-resource`
#1741 (`ScriptedConfiguration.getClass().getResource(...)`: the default
customizer path is built from `getClass().getPackage()` on purpose, so
that a subclass ships its own `CustomizerScript.groovy` next to itself —
to be dismissed as intended).

## Real defects

- **`ADLdapUtil`** (`dereferenced-value-may-be-null` #1567 #1568): both
GUID conversions swallowed the `NamingException` of `attr.get()` and
then indexed the still-null array — the caller saw a
`NullPointerException` with no clue about the LDAP error. They throw a
`ConnectorException` carrying it now.
- **`AbstractRemoteConnection.convert`** (`missing-case-in-switch`
#1538, #1562): no case for `RequestType.API`, so the method went on to
call `setHeader` on a null request. `default` throws
`NotSupportedException`.
- **`ObjectClassRunner.isObjectClassSupported`** (#1563): a contract
test with no required operation iterated a null set; it now falls back
to every object class of the schema.
- **`DocBookResourceMojo`** (`output-resource-leak` #1559): the
`FileWriter` of the remote-resources manifest was never closed, i.e. not
even reliably flushed. try-with-resources.
- **`OperationalContext.getConfiguration`**
(`unsafe-double-checked-locking-init-order` #1553): the configuration
bean was stored in the volatile field before its change callback was
registered, so another thread could use it without the callback; it is
built completely and published last.
- **`ScriptedConfiguration.getGroovyScriptEngine`** (#1552): the same
pattern, but it can not be fixed by reordering —
`initializeCustomizer()` → `getCustomizerClass()` →
`getGroovyScriptEngine()` re-enters the getter, and the customizer
script gets `this`. The getter is `synchronized` for the whole
initialisation instead of double-checked, so other threads wait for the
customizer rather than see the engine before it ran. Uncontended
monitor, not a hot path.

## Guards after the dereference / redundant guards

`SchemaParser` (`type.getName()` before `if (type != null)` — now a
`continue` guard at the top, the block dedented; review with `-w`; a
top-level element of a simple type, which threw an NPE before, is
skipped with a warning), `AttributeTypeUtil` (`attrInfo.getType()`
before `attrInfo != null`), `ObjectPool.borrowObject` (`if (null != rv)`
after `rv.getPooledObject()`; `borrowObjectNoTest` never returns null),
`MultiOpTests` (`coBeforeTest` starts as an empty map),
`TstAbstractConnector` (null-safe paged-results cookie),
`CSVFileConnector.generateSyncDelta` (explicit
`IllegalArgumentException` when both objects are null).

## Housekeeping

- `field-masks-super-field` #1739 #1740: the `OperationMessageListener`
queues of `ICFWebSocket` and `OpenICFWebSocket` shadowed the `listeners`
fields of their superclasses; renamed to `messageListeners` (the
`WebSocketListener...` constructor parameter is untouched).
- `input-resource-leak` #1557 / `output-resource-leak` #1558:
`IOUtil.getResourceAsString` closes the reader (which closes the stream;
a null `charset` is rejected before the stream opens),
`IOUtil.writeFileUTF8` uses try-with-resources with
`StandardCharsets.UTF_8`.
- `non-sync-override` #1548 #1547 #1783: `getCause()` overrides and
`CompletionListener.start()` are `synchronized` like the methods they
override.
- `reference-equality-on-strings` #1539: `SQLParam.equals` via
`Objects.equals`.
- `non-null-boxed-variable` ×9: `Boolean`/`Integer` locals that never
hold null are primitives (`getColumnType` never returns null —
`Types.NULL` fallback).
- `constant-comparison` #1540 #1542 #1543: always-true conditions
removed.

## Compatibility

`OpenICFWebSocketApplication.OpenICFWebSocket` is public and non-final,
and its `protected` queue is renamed `listeners` → `messageListeners`. A
subclass compiled against the old name still links: its `getfield
listeners` resolves to Grizzly's `SimpleWebSocket.listeners`, a
`Queue<WebSocketListener>`, and fails with a `ClassCastException` on
dispatch. No such subclass is known — inside this repository there is
none, and a code search finds it elsewhere only in a fork that carries
its own copy of the class — but a custom connector server that
subclasses it has to be rebuilt.

## Tests

- `ADLdapUtilTests`: an `Attribute` whose `get()` throws — both GUID
conversions throw a `ConnectorException` carrying the `NamingException`
(an NPE with the old empty `catch`).
- `AbstractRemoteConnectionTest`: a read request reporting
`RequestType.API` fails the promise with `NotSupportedException` (an
`InternalServerErrorException` wrapping the NPE without the `default`
branch).
- `SchemaParserTests.parseSchemaShouldSkipSimpleTypedElement`: the test
schema plus a simple-typed top-level element parses to the same schema
(an NPE without the guard).

The rest are dead conditions, types, monitors and ordering with no
behaviour a test can observe.

Local runs of all 13 touched modules, all green: framework 186, dbcommon
86, internal 469, contract 43, server 29, grizzly 34, csvfile 78,
databasetable 43, groovy 125, ldap 159 (embedded OpenDJ), xml 81. After
the review round, rebased on `master`: framework 192, internal 457
(without `RemoteConnectorInfoManager*Tests`, whose fixed port was taken
by a parallel run), groovy 126, ldap 161, xml 82.

`CSVFileConnector`'s header gets the same line #131 adds, so the two
merge cleanly.
…ace it with a fragment

Update and delete rebuilt the CSV in a File.createTempFile in the system
temp directory - readable by other local users while the whole file,
passwords included, was written there - and moved it over the CSV
unconditionally, so a rewrite that failed half-way replaced the data
with the rows written so far, and the CSV came back with the temp file's
permissions instead of its own.

The copy is now created next to the CSV with owner-only permissions,
takes over the CSV's permissions and replaces it atomically, and only
when the rewrite completed; otherwise it is deleted. The write lock is
released in its own finally so that a failing close() can not keep it.

The directory a connector bundle is expanded into is created with
Files.createTempDirectory, owner-only on POSIX file systems.
A failing final flush of the writer was logged by closeQuietly and the
truncated copy still replaced the CSV; the writer is now closed before
the rewrite counts as complete. The copy is synced before the rename and
the directory after it, takes over the CSV's group as well as its mode
bits, and is deleted when the replacement fails. doDelete decrements the
cached row count only after the replacement succeeded.

The permission tests use a mode that neither a new temporary file nor a
022 umask produces, and the failed-rewrite tests check that no copy is
left behind.
…nly once counted

- finishRewrite drops the group bits when the CSV's group can not be copied,
  and logs a refused chmod instead of failing the operation
- doDelete/doUpdate close the copy's FileOutputStream on their own, so a
  failing final flush can not leave it open (JDK 8-21, undeletable on Windows)
- create/delete leave the cached row count alone while it is still -1
- document that a copy left by a killed JVM must be deleted by hand
- RewriteSafetyTest: copy private and next to the CSV, no copy left on
  success, row count after a failed delete and before the first search
@vharseko
vharseko force-pushed the csv-temp-files-and-locks branch from b224469 to abde58a Compare October 4, 2026 11:32
@vharseko

vharseko commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

@maximthomas all seven points are taken in abde58a, on a branch rebased onto current master (#133, #134, #136, #141, #143; no conflicts).

Group bits when the group can not be copied (question). Not intended. When posix:group fails, finishRewrite now drops GROUP_READ/GROUP_WRITE/GROUP_EXECUTE from the mode before applying it, so the process's group gets nothing the CSV granted to its own group. The mode is collected with EnumSet.noneOf + addAll, not EnumSet.copyOf, which would throw on the empty mode of a 000 file. The Javadoc says what is kept in that case. No test: it needs a file whose group the test user is not in, which only root can create.

Copy private and next to the CSV (suggestion). createRewriteFile() is package-private, and rewriteCopyIsPrivateAndNextToTheFile is your test. The fixture now lives in its own Files.createTempDirectory, so the parent assertion bites, and assertNoRewriteFileLeft scans that directory instead of java.io.tmpdir. The mutant that goes back to File.createTempFile("csvfile", "tmp") fails it.

Success path and the guarded decrement (suggestion). Both *KeepsFilePermissions tests now call assertNoRewriteFileLeft(): the Files.copy-instead-of-move mutant fails both. failedDeleteKeepsRowCount takes the count with a paged search (total 2), deletes an unknown uid, checks the total is still 2, then deletes a real row and checks 1. The mutant with the decrement made unconditional and moved before finishRewrite fails it.

-1 sentinel (issue). doCreate and doDelete go through a new adjustRowCount(delta), which leaves the count alone while it is still -1, so the next paged search counts the rows. createBeforeFirstSearchKeepsRowCountUncounted (total 3, was 0) and deleteBeforeFirstSearchKeepsRowCountUncounted (total 1, was no SearchResult at all, from -2) both fail without the guard.

File handle left open after a failing final flush (issue). Confirmed in StreamEncoder.implClose() of JDK 11/17: out.close() is reached only after a successful writeBytes(), and close() marks the encoder closed either way. doDelete/doUpdate now keep the FileOutputStream of the copy and close it on its own after the writer (closeQuietly(copy, "copy")). The writer is an OutputStreamWriter on that stream, with the same default charset FileWriter used. No test: it needs a disk that fails on the last write.

Refused chmod on the copy (question). Failing closed was not intended. The process owns the copy, so on a POSIX file system its chmod does not fail; it fails only where the mount fixes the modes of every file, and there the mode is the mount's anyway. setPosixFilePermissions now has its own catch (IOException) that logs a warning, and the copy keeps the mode it was created with. A failure to read the CSV's own attributes still fails the operation.

Copy left by a killed JVM (suggestion). Documented, no sweep, for the reason you gave: the write lock is per JVM. The note is in the Javadoc of createRewriteFile(), and, since that is private API, also in a new section "How Updates and Deletes Rewrite the CSV File" of the connector's installation chapter (chap-install-csvfile-connector.xml), which also describes the permissions the rewritten file keeps.

Mutation check, one mutant at a time, RewriteSafetyTest 8 tests: each of the four mutants above fails at least one test, and the restored source passes all eight.

Local run: csvfile-connector 88 tests, all green.

@vharseko
vharseko requested a review from maximthomas October 4, 2026 11:33
@vharseko vharseko added the documentation README, docs, license headers label Oct 4, 2026

@maximthomas maximthomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: The rewrite path now handles its failure cases on every platform.

  • finishRewrite drops GROUP_* when the posix:group copy is refused, and treats a refused chmod as a warning (CSVFileConnector.java:1251-1264); EnumSet.noneOf keeps a 000 file from breaking it.
  • The copy's FileOutputStream is closed on its own before finishRewrite (:1107, :1180), so the JDK 8-21 StreamEncoder leak can no longer keep a partial copy undeletable on Windows.
  • adjustRowCount (:1211) leaves an uncounted -1 alone, so a create or delete before the first paged search no longer leaves a wrong total for the life of the JVM.

@vharseko
vharseko merged commit 54d54b1 into OpenIdentityPlatform:master Oct 5, 2026
14 checks passed
@vharseko
vharseko deleted the csv-temp-files-and-locks branch October 5, 2026 12:49
vharseko added a commit that referenced this pull request Oct 6, 2026
…nterned-string locks, racy lazy init (#132)

Closes the remaining 10 open CodeQL alerts of severity `error` that
carry no security rating: `java/contradictory-type-checks` #1536 #1537,
`java/dereferenced-value-is-always-null` #1560,
`java/sync-on-boxed-types` #1554 #1555 #1556,
`java/unsafe-double-checked-locking` #1550 #1551,
`java/unsynchronized-getter` #1544 #1545. The eleventh, #1549
(`ScriptedConfiguration`), was closed by #133.

Fixes #148
Fixes #149

| Where | Finding | Change |
|---|---|---|
| `SQLUtil.setParam` | `else if (val instanceof Integer)` appears twice;
the second branch can never run | second branch removed |
| `ContractITCase.createInstances` | `IObjectFactory objectFactory =
null` is dereferenced for any test class with a `(String)` constructor.
None of the default classes has one today, so the code only worked
because the `NoSuchMethodException` path was always taken | the
`(String)` constructor is called through plain reflection, so the method
no longer depends on TestNG's internal `ObjectFactoryImpl`, which TestNG
7.5 moved and whose `IObjectFactory` 7.10.2 removed. The unused nested
`ContractTestFactory` with its three never-read fields is gone |
| `BatchRemoteCache` (testbundlev1) | `synchronized (resultLock)` on the
interned literal `"resultLock"`, a monitor shared with any other code in
the JVM that synchronises on the same string | `new Object()` |
| `WebSocketConnectionGroup.operationContext`,
`TstStatefulConnectorConfig.executorService` | double-checked lazy
initialisation reads a non-volatile field outside the lock, so a thread
may see a partially constructed object | fields declared `volatile`,
which makes the pattern correct under the JMM |
| `ScriptedConfiguration.getGroovyScriptEngine()` | #133 made the getter
`synchronized` for the whole initialisation. That closed the
double-checked-locking alert and #149 (the getter and `release()` share
the monitor now), and concurrent callers wait for the customizer. But
the getter still stored the engine before its customizer ran, so a
failed customizer left an uncustomized engine in place for good (#148) |
on top of #133's synchronised getter, the engine is published only after
`initializeCustomizer()` succeeds. The customizer's own re-entrant calls
get the engine under construction from a second lock-guarded field. If
the customizer fails, nothing is published and the next call retries;
the failed customizer class is removed with `InvokerHelper.removeClass`,
so the metaclass the REST, CREST and SSH configurations register on it
does not keep one class loader alive per retry. The customizer runs
under the configuration's monitor, so it must not wait for another
thread that uses the engine; the comment on `initializeCustomizer()`
says so |
| `customize` in `SSHConfiguration`, `ScriptedCRESTConfiguration` | with
the retry above, a failed attempt that registered `release {}` left its
closure for `release()` to run after a successful retry that registered
none: the SSH `customize` never reset it, and CREST's `release = null`
wrote a private field nothing reads | both reset it with
`setReleaseClosure(null)`, as REST's `customize` already does; CREST's
unread field is removed |
| `FrameworkUtil.getFrameworkVersion()` | unsynchronised lazy init while
`setFrameworkVersion` is synchronised | `static synchronized` (not a hot
path). The `(ClassLoader)` overload reads
`connectors-framework.properties` rather than the field and is only used
by `FrameworkUtilTests`, so it is renamed to `readFrameworkVersion` and
the getter/setter pairing no longer applies to it |

`ScriptedConfigurationTest` covers #148: a customizer that fails once is
retried on the next call, a second thread blocks until the customizer
has finished, and a failed customizer class has no metaclass left
registered. Each test fails when its part of the fix is reverted (engine
published before the customizer, no `removeClass`, getter not
synchronised). `ScriptedConfigurationTest` (CREST) and
`SSHConfigurationCustomizerTest` (SSH) cover the release closure: a
first attempt registers `release {}` and fails, the retry registers
none, and `release()` must not run the stale closure; both fail without
the `customize` reset. #149 is fixed on master by #133; it was a window
between two adjacent field reads that no test can hold open, so it has
no test. The other changes have no behaviour a unit test can observe:
dead branches, `volatile`, a private monitor and a synchronised getter.
`createInstances` takes its classes from a fixed list with no `(String)`
constructors, so its reflective path cannot be reached from a test.

The class of a customizer that **succeeded** is still never
unregistered, so `release()` keeps its loader alive; master behaves the
same, and that is tracked separately in #154.

Local reactor run (rebased on current master, after #131 #133 #134 #135
#136 #139 #141 #142 #143 #145) of all touched modules and everything
they depend on: connector-framework 200, dbcommon 89, framework-internal
524 (2 skipped), contract 43, connector-framework-server 33,
groovy-connector 130 (42 skipped as on master, 4 new), ssh-connector 97
(82 skipped, 1 new), testbundlev1 compiles — all green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working concurrency Races, locking and thread-safety fixes connector:csvfile CSV file connector data-loss Fixes or risks loss or corruption of stored data dependencies Pull requests that update a dependency file documentation README, docs, license headers framework OpenICF-java-framework java Pull requests that update java code security Security fix / CVE remediation tests Test additions or fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants