Rewrite the CSV file through a private copy next to it and never replace it with a fragment - #131
Conversation
maximthomas
left a comment
There was a problem hiding this comment.
praise: The rewrite now goes through a private copy next to the CSV, and a failed read-side rewrite leaves the data file alone.
createRewriteFile()creates the copy withFiles.createTempFilein the CSV's own directory, so the copy is owner-only while it is written and the replacement is a same-file-system rename.finishRewrite(tmp, rewritten)discards the copy of a half-done rewrite, and a failed replacement is now aConnectorIOExceptioninstead of a logged error behind a reported success.lock.unlock()sits in its own innerfinallyindoCreate/doDelete/doUpdate, andgetBundleTempDirusesFiles.createTempDirectory.
issue (blocking): If writer.close() fails, update and delete still replace the CSV with a truncated copy.
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1057-1058, :1068-1069, :1121, :1131-1132
rewritten = true is set while the last rows are still in the writer's buffers (the FileWriter encoder buffer, plus super-csv's BufferedWriter). Those rows reach the copy only in closeQuietly(writer, "writer"), which catches and logs any exception, and then finishRewrite(tmp, true) moves the copy over the CSV. If the final flush fails (ENOSPC, EDQUOT, EIO, or close(2) on NFS), the CSV loses its tail, or all of it when the file is smaller than the buffer, and the operation returns success. This is the "fragment" case the PR title says is fixed. RewriteSafetyTest only injects a read-side failure.
// doDelete, after the `!found` check
writer.close(); // the last rows reach the copy here; a failure must not replace the CSV
totalRowCount.put(csvFilePath, totalRowCount.get(csvFilePath) - 1);
rewritten = true;
// doUpdate, after the `updated == null` check
writer.close();
rewritten = true;A close failure then goes to the catch (IOException e) and the copy is discarded. The second close in closeQuietly does nothing on a writer that is already closed.
issue (non-blocking): updateKeepsFilePermissions/deleteKeepsFilePermissions cannot fail when finishRewrite stops copying the CSV's permissions.
OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:117-124, CSVFileConnector.java:1171
The fixture sets the CSV to rw-------, which is exactly the mode Files.createTempFile gives the copy on POSIX. With Files.setPosixFilePermissions(tmp.toPath(), Files.getPosixFilePermissions(csv)) removed, RewriteSafetyTest stays green (4 run, 0 failures, 0 skipped, macOS). So with this mutant a group-readable CSV comes back rw------- after every update or delete, and CI would not notice. These tests are the only pin on the permission fix.
// neither Files.createTempFile (rw-------) nor a 022 umask (rw-r--r--) yields this mode
Set<PosixFilePermission> mode = PosixFilePermissions.fromString("rw-r-----");
Files.setPosixFilePermissions(csv.toPath(), mode);
return mode;Pin: with restrictToOwner() returning rw-r-----, both tests fail against the mutant and still fail against the old /tmp + umask code.
issue (non-blocking): doDelete decrements the cached row count before the replacement, which can now fail.
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1057, :1069
When finishRewrite throws, the CSV still has the row, but the static totalRowCount has already gone down by one. It stays wrong for the life of the JVM, because the count is recomputed (:347) only while it is -1, and it is set to -1 once, at :224. Paged searches then report a wrong totalPagedResults/remainingPagedResults (:406-413). The decrement-first order is older than this PR, but the PR now reports the failure to the caller while the count records a delete. Probe: count 3 → 2 and the CSV unchanged after a failed move.
try {
closeQuietly(reader, "reader");
closeQuietly(writer, "writer");
finishRewrite(tmp, rewritten);
if (rewritten) {
totalRowCount.put(csvFilePath, totalRowCount.get(csvFilePath) - 1);
}
} finally {
lock.unlock();
}The decrement moves out of the try body (:1057).
question (non-blocking): Is the rewritten copy kept on purpose when the replacement fails?
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1181-1183
The IOException catch rethrows with the copy's name but never deletes it. Each failed replacement leaves behind a <csv>.<random>.tmp: a full copy of the accounts and password hashes, holding the change the caller was told had failed. Causes include an immutable or locked CSV, a Windows sharing violation, or a CSV removed outside the lock. The copy has the CSV's own mode, so nothing is exposed more widely than the CSV itself, but the copies pile up (probe: chflags uchg on the CSV → ConnectorIOException, copy left). If the copy is meant for recovery, say so in the Javadoc. Otherwise delete it:
} catch (IOException e) {
if (!tmp.delete() && tmp.exists()) {
log.warn("Could not delete {0}", tmp);
}
throw new ConnectorIOException("Failed to replace " + csv + " with its rewritten copy", e);
}suggestion (non-blocking): The failed-rewrite tests do not check that the partial copy is deleted.
OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:93-115
failedUpdateLeavesFileUntouched/failedDeleteLeavesFileUntouched assert only the CSV's content. Replacing if (!tmp.delete() && tmp.exists()) with if (false) in finishRewrite keeps all 4 tests green and leaves rewrite*.csv.<n>.tmp partial copies next to the CSV.
// in both catch blocks, after the content check (static import org.testng.Assert.assertFalse)
for (String name : csv.getParentFile().list()) {
assertFalse(name.startsWith(csv.getName() + ".") && name.endsWith(".tmp"),
"the copy of the failed rewrite was left behind: " + name);
}suggestion (non-blocking): finishRewrite keeps the CSV's mode bits but not its group.
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1170-1174
The copy is a new file, so its group is the process's (or the directory's), and the copied group bits apply to that group. A CSV shared as rw-rw---- with group idm comes back rw-rw---- under the process's group: members of idm lose access, and the process's group gains rw (probe on macOS: group everyone → staff). The old code also replaced the file, but the Javadoc now promises "keeping the CSV's permissions".
try {
try {
Files.setAttribute(tmp.toPath(), "posix:group", Files.getAttribute(csv, "posix:group"));
} catch (IOException e) {
// not a member of the CSV's group: the copy keeps the process's group
}
Files.setPosixFilePermissions(tmp.toPath(), Files.getPosixFilePermissions(csv));
} catch (UnsupportedOperationException e) {
// not a POSIX file system: the copy inherits the directory's ACL
}Or: narrow the Javadoc to "the CSV's mode bits".
0c1847d to
b224469
Compare
|
The branch is rebased onto current
Beyond the review: the copy is now also Permission tests — the fixture mode is now
Copy left after a failed replacement — not intended. The Partial copy check — both Group — Not covered by tests: the close failure, a failed replacement and the group copy — each needs a failing disk, an immutable file or a second group membership, none of which a unit test can set up portably. Local runs: csvfile-connector 84 tests, connector-framework-internal 500 tests (2 skipped: |
maximthomas
left a comment
There was a problem hiding this comment.
praise: every round-1 point landed where the bug was, and the rewrite now fails closed end to end.
writer.close()runs beforerewritten = trueindoDelete(CSVFileConnector.java:1060) anddoUpdate(:1128), so a failing final flush discards the copy instead of replacing the CSV with it.finishRewritesyncs the copy (FileChannel.force(true),:1181-1183) before the rename and deletes it on any failed replacement (:1201-1203).setDistinctMode()(RewriteSafetyTest.java:120) usesrw-r-----, a modeFiles.createTempFilenever produces, so the*KeepsFilePermissionstests now fail when the mode copy is dropped.
question (non-blocking): when the group cannot be copied, should the CSV's group bits still be applied to the copy?
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1186-1191
The posix:group copy fails with EPERM when the process user is not in the CSV's group, and the empty catch swallows it. setPosixFilePermissions then still copies the CSV's group bits, so they now apply to the process's group (Linux: the egid; BSD/macOS: the directory's group). Example: a CSV svc:hr rw-r----- whose owner svc is not in hr comes back svc:<svc's group> rw-r-----, and members of svc's group can read the account data and password hashes. Probe on macOS: CSV wheel rw-r-----, chown EPERM swallowed, copy ends staff rw-r-----. This is still narrower than master (0644). If granting these bits is intended, it is worth one sentence in the Javadoc. If not, drop the group bits:
// import java.nio.file.attribute.PosixFilePermission;
Set<PosixFilePermission> mode = Files.getPosixFilePermissions(csv);
try {
Files.setAttribute(tmp.toPath(), "posix:group", Files.getAttribute(csv, "posix:group"));
} catch (IOException e) {
// not a member of the CSV's group: the copy keeps the process's group,
// which must not get the access the CSV grants to its own group
mode.remove(PosixFilePermission.GROUP_READ);
mode.remove(PosixFilePermission.GROUP_WRITE);
mode.remove(PosixFilePermission.GROUP_EXECUTE);
}
Files.setPosixFilePermissions(tmp.toPath(), mode);suggestion (non-blocking): no test pins the CodeQL #20/#21 fix itself: that the copy is owner-only and next to the CSV while it is written.
OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:57, CSVFileConnector.java:1155-1158
The fixture CSV is in java.io.tmpdir. The mode tests read only the final mode, which finishRewrite copies onto any copy, and assertNoRewriteFileLeft matches only <csv>.*.tmp. A mutant that turns createRewriteFile back into BASE's File.createTempFile("csvfile", "tmp") (system temp dir, umask 0644) keeps the whole csvfile module green: 15 classes, RewriteSafetyTest 4/4. With createRewriteFile() made package-private:
@Test
public void rewriteCopyIsPrivateAndNextToTheFile() throws Exception {
setDistinctMode(); // skips where POSIX permissions are not supported
File tmp = connector.createRewriteFile();
try {
assertEquals(tmp.getParentFile(), csv.getAbsoluteFile().getParentFile());
assertEquals(Files.getPosixFilePermissions(tmp.toPath()),
PosixFilePermissions.fromString("rw-------"));
} finally {
tmp.delete();
}
}Pin: the mode assertion kills the BASE-creator mutant. The parent assertion only bites once the fixture CSV is created in its own Files.createTempDirectory and not directly in java.io.tmpdir.
suggestion (non-blocking): the success-path tests do not check that the copy is gone, and nothing pins the guarded row-count decrement.
OpenICF-csvfile-connector/src/test/java/org/forgerock/openicf/csvfile/RewriteSafetyTest.java:77-92, CSVFileConnector.java:1072-1075, :1196-1199
Two mutants keep the whole csvfile module green (15 classes). In the first, both Files.move calls in finishRewrite become Files.copy(tmp.toPath(), csv, REPLACE_EXISTING): every successful update and delete then leaves a full <csv>.<n>.tmp next to the CSV. In the second, the decrement becomes unconditional and runs before finishRewrite: an unknown-uid or failed delete lowers the paging count, and no test reads it.
connector.update(ObjectClass.ACCOUNT, new Uid("vilo"), lastName("updated"), null);
assertEquals(Files.getPosixFilePermissions(csv.toPath()), mode);
assertNoRewriteFileLeft(); // same line in deleteKeepsFilePermissions: kills the copy-not-move mutantPin: for the decrement, run a paged search so that the count is known, then a failed delete, then a second paged search, and assert that the total in its SearchResult is unchanged.
issue (non-blocking): the row-count updates still ignore the -1 "not counted yet" sentinel.
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1072-1075, :1008
init() stores -1 (:226). The recount runs only on == -1 (:349), and the paging totals are reported only on > -1 (:408). A delete before the first counting search stores -2: from then on, for the life of the JVM, nothing recounts and no SearchResult with totals or a cookie is returned. A create first stores 0, which reads as a known count of zero, so the total is reported as 0 and paging stops after the first page. This predates the PR (BASE :1052, :1002), but the PR already touches the delete line.
if (rewritten) {
Integer count = totalRowCount.get(csvFilePath);
if (count > -1) {
totalRowCount.put(csvFilePath, count - 1);
}
}
// and the same `> -1` guard around the increment in doCreate (:1008)issue (non-blocking): on JDK 8-21, a failing final flush leaves the copy's file handle open, so on Windows the partial copy cannot be deleted.
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1042, :1102, :1060, :1128, :1208-1212
writer.close() closes the FileWriter. On JDK 8/11/17/19/21, StreamEncoder.implClose() calls out.close() inside its try after the last writeBytes(), so when that write throws (ENOSPC, EIO) the FileOutputStream stays open. JDK 24+ uses try (out) (checked in each JDK's src.zip). The second closeQuietly(writer, …) is a no-op. On Windows, deleteRewriteFile then fails on the open handle, logs "Could not delete", and leaves a partial copy of the accounts next to the CSV. Not run: needs a Windows host with a failing disk. On POSIX only the descriptor leaks until GC.
// import java.io.FileOutputStream; import java.io.OutputStreamWriter;
FileOutputStream copy = null;
...
copy = new FileOutputStream(tmp);
writer = new CsvMapWriter(new OutputStreamWriter(copy), csvPreference); // same default charset as FileWriter
...
} finally {
try {
closeQuietly(reader, "reader");
closeQuietly(writer, "writer");
closeQuietly(copy, "copy");
finishRewrite(tmp, rewritten);question (non-blocking): is a refused chmod on the copy meant to fail every update and delete?
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1191, :1201-1204
The group copy is best effort, but the mode copy tolerates only UnsupportedOperationException. Its IOException reaches the outer catch, which deletes the copy and throws ConnectorIOException. Consider a mount where every file belongs to one fixed uid that is not the connector's (vfat/exfat uid=…,umask=000 without quiet, or CIFS uid=… without noperm). There the copy is not owned by the process, chmod gets EPERM, and every rewrite fails; master worked on such mounts. Not run: needs such a mount. EPERM for a non-owner chmod surfaces as FileSystemException (probed). If the copy is left at rw-------, nothing is exposed. If failing closed is not intended:
try {
Files.setPosixFilePermissions(tmp.toPath(), Files.getPosixFilePermissions(csv));
} catch (IOException e) {
log.warn(e, "Could not copy the permissions of {0}: the rewritten file is owner-only", csv);
}suggestion (non-blocking): say that a copy left by a killed JVM is not cleaned up.
OpenICF-csvfile-connector/src/main/java/org/forgerock/openicf/csvfile/CSVFileConnector.java:1157
If the JVM dies (SIGKILL, OOM kill, power loss) between createRewriteFile and Files.move, the copy, which holds every account and password hash, stays next to the CSV permanently. It is owner-only. scrubSyncFiles and getLatestSyncToken match only <csv>.<13 digits>, and nothing scans for .tmp. On master, such leftovers went to java.io.tmpdir, which OS cleaners empty. A startup sweep would race another JVM rewriting the same CSV (the write lock is per JVM), so documenting it is the safer fix:
* A copy left behind by a process that died while writing it is not removed
* automatically: delete {@code <csv>.<digits>.tmp} files by hand while no
* connector is running against the CSV.…tors (#133) Closes 35 of the 37 open CodeQL alerts of severity `warning` without a security rating. Not in here: `java/constant-comparison` #1541 (`ObjectPool`, already fixed in #126) and `java/unsafe-get-resource` #1741 (`ScriptedConfiguration.getClass().getResource(...)`: the default customizer path is built from `getClass().getPackage()` on purpose, so that a subclass ships its own `CustomizerScript.groovy` next to itself — to be dismissed as intended). ## Real defects - **`ADLdapUtil`** (`dereferenced-value-may-be-null` #1567 #1568): both GUID conversions swallowed the `NamingException` of `attr.get()` and then indexed the still-null array — the caller saw a `NullPointerException` with no clue about the LDAP error. They throw a `ConnectorException` carrying it now. - **`AbstractRemoteConnection.convert`** (`missing-case-in-switch` #1538, #1562): no case for `RequestType.API`, so the method went on to call `setHeader` on a null request. `default` throws `NotSupportedException`. - **`ObjectClassRunner.isObjectClassSupported`** (#1563): a contract test with no required operation iterated a null set; it now falls back to every object class of the schema. - **`DocBookResourceMojo`** (`output-resource-leak` #1559): the `FileWriter` of the remote-resources manifest was never closed, i.e. not even reliably flushed. try-with-resources. - **`OperationalContext.getConfiguration`** (`unsafe-double-checked-locking-init-order` #1553): the configuration bean was stored in the volatile field before its change callback was registered, so another thread could use it without the callback; it is built completely and published last. - **`ScriptedConfiguration.getGroovyScriptEngine`** (#1552): the same pattern, but it can not be fixed by reordering — `initializeCustomizer()` → `getCustomizerClass()` → `getGroovyScriptEngine()` re-enters the getter, and the customizer script gets `this`. The getter is `synchronized` for the whole initialisation instead of double-checked, so other threads wait for the customizer rather than see the engine before it ran. Uncontended monitor, not a hot path. ## Guards after the dereference / redundant guards `SchemaParser` (`type.getName()` before `if (type != null)` — now a `continue` guard at the top, the block dedented; review with `-w`; a top-level element of a simple type, which threw an NPE before, is skipped with a warning), `AttributeTypeUtil` (`attrInfo.getType()` before `attrInfo != null`), `ObjectPool.borrowObject` (`if (null != rv)` after `rv.getPooledObject()`; `borrowObjectNoTest` never returns null), `MultiOpTests` (`coBeforeTest` starts as an empty map), `TstAbstractConnector` (null-safe paged-results cookie), `CSVFileConnector.generateSyncDelta` (explicit `IllegalArgumentException` when both objects are null). ## Housekeeping - `field-masks-super-field` #1739 #1740: the `OperationMessageListener` queues of `ICFWebSocket` and `OpenICFWebSocket` shadowed the `listeners` fields of their superclasses; renamed to `messageListeners` (the `WebSocketListener...` constructor parameter is untouched). - `input-resource-leak` #1557 / `output-resource-leak` #1558: `IOUtil.getResourceAsString` closes the reader (which closes the stream; a null `charset` is rejected before the stream opens), `IOUtil.writeFileUTF8` uses try-with-resources with `StandardCharsets.UTF_8`. - `non-sync-override` #1548 #1547 #1783: `getCause()` overrides and `CompletionListener.start()` are `synchronized` like the methods they override. - `reference-equality-on-strings` #1539: `SQLParam.equals` via `Objects.equals`. - `non-null-boxed-variable` ×9: `Boolean`/`Integer` locals that never hold null are primitives (`getColumnType` never returns null — `Types.NULL` fallback). - `constant-comparison` #1540 #1542 #1543: always-true conditions removed. ## Compatibility `OpenICFWebSocketApplication.OpenICFWebSocket` is public and non-final, and its `protected` queue is renamed `listeners` → `messageListeners`. A subclass compiled against the old name still links: its `getfield listeners` resolves to Grizzly's `SimpleWebSocket.listeners`, a `Queue<WebSocketListener>`, and fails with a `ClassCastException` on dispatch. No such subclass is known — inside this repository there is none, and a code search finds it elsewhere only in a fork that carries its own copy of the class — but a custom connector server that subclasses it has to be rebuilt. ## Tests - `ADLdapUtilTests`: an `Attribute` whose `get()` throws — both GUID conversions throw a `ConnectorException` carrying the `NamingException` (an NPE with the old empty `catch`). - `AbstractRemoteConnectionTest`: a read request reporting `RequestType.API` fails the promise with `NotSupportedException` (an `InternalServerErrorException` wrapping the NPE without the `default` branch). - `SchemaParserTests.parseSchemaShouldSkipSimpleTypedElement`: the test schema plus a simple-typed top-level element parses to the same schema (an NPE without the guard). The rest are dead conditions, types, monitors and ordering with no behaviour a test can observe. Local runs of all 13 touched modules, all green: framework 186, dbcommon 86, internal 469, contract 43, server 29, grizzly 34, csvfile 78, databasetable 43, groovy 125, ldap 159 (embedded OpenDJ), xml 81. After the review round, rebased on `master`: framework 192, internal 457 (without `RemoteConnectorInfoManager*Tests`, whose fixed port was taken by a parallel run), groovy 126, ldap 161, xml 82. `CSVFileConnector`'s header gets the same line #131 adds, so the two merge cleanly.
…ace it with a fragment Update and delete rebuilt the CSV in a File.createTempFile in the system temp directory - readable by other local users while the whole file, passwords included, was written there - and moved it over the CSV unconditionally, so a rewrite that failed half-way replaced the data with the rows written so far, and the CSV came back with the temp file's permissions instead of its own. The copy is now created next to the CSV with owner-only permissions, takes over the CSV's permissions and replaces it atomically, and only when the rewrite completed; otherwise it is deleted. The write lock is released in its own finally so that a failing close() can not keep it. The directory a connector bundle is expanded into is created with Files.createTempDirectory, owner-only on POSIX file systems.
A failing final flush of the writer was logged by closeQuietly and the truncated copy still replaced the CSV; the writer is now closed before the rewrite counts as complete. The copy is synced before the rename and the directory after it, takes over the CSV's group as well as its mode bits, and is deleted when the replacement fails. doDelete decrements the cached row count only after the replacement succeeded. The permission tests use a mode that neither a new temporary file nor a 022 umask produces, and the failed-rewrite tests check that no copy is left behind.
…nly once counted - finishRewrite drops the group bits when the CSV's group can not be copied, and logs a refused chmod instead of failing the operation - doDelete/doUpdate close the copy's FileOutputStream on their own, so a failing final flush can not leave it open (JDK 8-21, undeletable on Windows) - create/delete leave the cached row count alone while it is still -1 - document that a copy left by a killed JVM must be deleted by hand - RewriteSafetyTest: copy private and next to the CSV, no copy left on success, row count after a failed delete and before the first search
b224469 to
abde58a
Compare
|
@maximthomas all seven points are taken in abde58a, on a branch rebased onto current Group bits when the group can not be copied (question). Not intended. When Copy private and next to the CSV (suggestion). Success path and the guarded decrement (suggestion). Both
File handle left open after a failing final flush (issue). Confirmed in Refused chmod on the copy (question). Failing closed was not intended. The process owns the copy, so on a POSIX file system its chmod does not fail; it fails only where the mount fixes the modes of every file, and there the mode is the mount's anyway. Copy left by a killed JVM (suggestion). Documented, no sweep, for the reason you gave: the write lock is per JVM. The note is in the Javadoc of Mutation check, one mutant at a time, Local run: csvfile-connector 88 tests, all green. |
maximthomas
left a comment
There was a problem hiding this comment.
praise: The rewrite path now handles its failure cases on every platform.
finishRewritedropsGROUP_*when theposix:groupcopy is refused, and treats a refused chmod as a warning (CSVFileConnector.java:1251-1264);EnumSet.noneOfkeeps a000file from breaking it.- The copy's
FileOutputStreamis closed on its own beforefinishRewrite(:1107,:1180), so the JDK 8-21StreamEncoderleak can no longer keep a partial copy undeletable on Windows. adjustRowCount(:1211) leaves an uncounted-1alone, so a create or delete before the first paged search no longer leaves a wrong total for the life of the JVM.
…nterned-string locks, racy lazy init (#132) Closes the remaining 10 open CodeQL alerts of severity `error` that carry no security rating: `java/contradictory-type-checks` #1536 #1537, `java/dereferenced-value-is-always-null` #1560, `java/sync-on-boxed-types` #1554 #1555 #1556, `java/unsafe-double-checked-locking` #1550 #1551, `java/unsynchronized-getter` #1544 #1545. The eleventh, #1549 (`ScriptedConfiguration`), was closed by #133. Fixes #148 Fixes #149 | Where | Finding | Change | |---|---|---| | `SQLUtil.setParam` | `else if (val instanceof Integer)` appears twice; the second branch can never run | second branch removed | | `ContractITCase.createInstances` | `IObjectFactory objectFactory = null` is dereferenced for any test class with a `(String)` constructor. None of the default classes has one today, so the code only worked because the `NoSuchMethodException` path was always taken | the `(String)` constructor is called through plain reflection, so the method no longer depends on TestNG's internal `ObjectFactoryImpl`, which TestNG 7.5 moved and whose `IObjectFactory` 7.10.2 removed. The unused nested `ContractTestFactory` with its three never-read fields is gone | | `BatchRemoteCache` (testbundlev1) | `synchronized (resultLock)` on the interned literal `"resultLock"`, a monitor shared with any other code in the JVM that synchronises on the same string | `new Object()` | | `WebSocketConnectionGroup.operationContext`, `TstStatefulConnectorConfig.executorService` | double-checked lazy initialisation reads a non-volatile field outside the lock, so a thread may see a partially constructed object | fields declared `volatile`, which makes the pattern correct under the JMM | | `ScriptedConfiguration.getGroovyScriptEngine()` | #133 made the getter `synchronized` for the whole initialisation. That closed the double-checked-locking alert and #149 (the getter and `release()` share the monitor now), and concurrent callers wait for the customizer. But the getter still stored the engine before its customizer ran, so a failed customizer left an uncustomized engine in place for good (#148) | on top of #133's synchronised getter, the engine is published only after `initializeCustomizer()` succeeds. The customizer's own re-entrant calls get the engine under construction from a second lock-guarded field. If the customizer fails, nothing is published and the next call retries; the failed customizer class is removed with `InvokerHelper.removeClass`, so the metaclass the REST, CREST and SSH configurations register on it does not keep one class loader alive per retry. The customizer runs under the configuration's monitor, so it must not wait for another thread that uses the engine; the comment on `initializeCustomizer()` says so | | `customize` in `SSHConfiguration`, `ScriptedCRESTConfiguration` | with the retry above, a failed attempt that registered `release {}` left its closure for `release()` to run after a successful retry that registered none: the SSH `customize` never reset it, and CREST's `release = null` wrote a private field nothing reads | both reset it with `setReleaseClosure(null)`, as REST's `customize` already does; CREST's unread field is removed | | `FrameworkUtil.getFrameworkVersion()` | unsynchronised lazy init while `setFrameworkVersion` is synchronised | `static synchronized` (not a hot path). The `(ClassLoader)` overload reads `connectors-framework.properties` rather than the field and is only used by `FrameworkUtilTests`, so it is renamed to `readFrameworkVersion` and the getter/setter pairing no longer applies to it | `ScriptedConfigurationTest` covers #148: a customizer that fails once is retried on the next call, a second thread blocks until the customizer has finished, and a failed customizer class has no metaclass left registered. Each test fails when its part of the fix is reverted (engine published before the customizer, no `removeClass`, getter not synchronised). `ScriptedConfigurationTest` (CREST) and `SSHConfigurationCustomizerTest` (SSH) cover the release closure: a first attempt registers `release {}` and fails, the retry registers none, and `release()` must not run the stale closure; both fail without the `customize` reset. #149 is fixed on master by #133; it was a window between two adjacent field reads that no test can hold open, so it has no test. The other changes have no behaviour a unit test can observe: dead branches, `volatile`, a private monitor and a synchronised getter. `createInstances` takes its classes from a fixed list with no `(String)` constructors, so its reflective path cannot be reached from a test. The class of a customizer that **succeeded** is still never unregistered, so `release()` keeps its loader alive; master behaves the same, and that is tracked separately in #154. Local reactor run (rebased on current master, after #131 #133 #134 #135 #136 #139 #141 #142 #143 #145) of all touched modules and everything they depend on: connector-framework 200, dbcommon 89, framework-internal 524 (2 skipped), contract 43, connector-framework-server 33, groovy-connector 130 (42 skipped as on master, 4 new), ssh-connector 97 (82 skipped, 1 new), testbundlev1 compiles — all green.
Closes the medium CodeQL alerts
java/local-temp-file-or-directory-information-disclosure#20 #21 #22 andjava/unreleased-lock#15, and fixes a data-loss bug found on the way.CSV connector: how update and delete rewrite the file
doUpdateanddoDeleterebuild the whole CSV — accounts, attributes, password hashes — in aFile.createTempFile("csvfile", "tmp"), i.e. in the system temp directory with the umask permissions (0644 as a rule), where any local user can read it while it is written. Then thefinallyblock unconditionally deleted the CSV and moved the temp file over it. Two consequences, both reproduced by the new tests against the old code:Now:
createRewriteFile()creates the copy next to the CSV (<csv>.<random>.tmp, outside the<csv>.<13 digits>pattern of the sync copies) withFiles.createTempFile: owner-only on POSIX, same file system, so the replacement is a rename.try, before the rewrite counts as complete, so a failing final flush (ENOSPC, EIO, NFSclose) discards the copy instead of putting a truncated one in place. The copy'sFileOutputStreamis also closed on its own: aFileWriterwhose final flush fails leaves it open on JDK 8–21, and Windows can not delete an open file.finishRewrite(tmp, rewritten)replaces the CSV only when the rewrite completed: the copy isfsynced, takes over the CSV's group if the process may set it (otherwise it keeps the process's group and gets no group permissions) and the CSV's mode bits (a chmod refused by a mount that fixes the modes is logged, and the copy keeps the mode it was created with; all of this is skipped on non-POSIX file systems; the owner becomes the process's user), and moves into place withREPLACE_EXISTING+ATOMIC_MOVE(plain move as fallback); the directory is synced after the rename, best effort. A copy that is not put in place — of a failed rewrite or of a failed replacement — is deleted. A failed replacement is now aConnectorIOExceptioninstead of a logged-and-swallowed error that left the caller believing the operation succeeded.doDeletedecrements the cached row count only after the replacement succeeded. Create and delete leave the count alone while it is still-1(not counted yet), so the next paged search counts the rows; before, a delete made it-2(no totals and no paging cookie for the life of the JVM) and a create made it0(paging stopped after the first page).chap-install-csvfile-connector.xml) say to delete such.tmpfiles by hand while no connector runs against the CSV.lock.unlock()sits in its own innerfinallyindoCreate,doDelete,doUpdate(Bump commons-io:commons-io from 2.2 to 2.7 in /OpenICF-java-framework/connector-test-common #15): aRuntimeExceptionfrom aclose()can no longer keep the write lock forever and hang every later operation on the file. The three copies of the close-and-log code becamecloseQuietly.FileUtils.moveFilewas the only use of commons-io; the dependency is removed.Bundle temp directory (#22)
LocalConnectorInfoManagerImplcreatedjava.io.tmpdir/bundle-<random>withmkdir()(umask permissions) for the expandedlib/andnative/entries of a connector bundle. It now usesFiles.createTempDirectory, owner-only on POSIX; the name-probing loop goes with it.Tests
RewriteSafetyTest(new, 8): update/delete keep a 0640 data file at 0640 — a mode neither a new temporary file nor a 022 umask produces (POSIX, otherwise skipped) — and leave no copy behind; a failed update/delete leaves the file byte-for-byte as it was and no copy behind; the copy is createdrw-------next to the CSV; a failed delete keeps the paged-search total, and a create or delete before the first search does not corrupt it. Each test fails on the code it guards against (old code or a mutant of the fix).LocalConnectorInfoManagerTests.testBundleTempDirectoryIsPrivate(new): a freshly expanded bundle directory isrwx------(POSIX, otherwise skipped). Uses its ownLocalConnectorInfoManagerImplso the factory cache and connector pools of the other tests are untouched.unlock()ordering, a failing writer close, the separately closed stream, a failed replacement and the group copy (including the dropped group bits) have no tests: they need aclose()that throws inside a private method, a failing disk, an immutable file or a group the test user is not in.Local runs: connector-framework-internal 500 tests (2 skipped, unrelated), csvfile-connector 88 tests, all green.
Not changed
java/unreleased-lock#13 (sync: the read lock taken while downgrading from the write lock is released by the followingtry/finally; nothing can throw in between) and #14 (findObjectInFile:if (rwLock != null) lock … finally { if (rwLock != null) unlock }) are false positives of the heuristic and will be dismissed as such.