Repository navigation
Merge changes for 1.10 into master branch - #115
Merged
Merged
Conversation
Previously, the real IP of the client was not properly handled by certstream-server-go. This commit takes the IP address from the header and stores them in the request context and the client struct. Additionally the user agent is now also preserved. Also each client now gets a generated ID to identify it even if another client from the same IP connects to the server. Also some log statements were changed so that it fits better when printing the client name.
Optimize tile processing by delaying the fetch of partial tiles. Certstream will wait up to a minute before forcefully fetching the partial tile to ensure slow-growing logs are still processed. Certstream will now: - Track when a partial tile is first seen - Wait up to one minute before forcing a partial fetch - Skip fetching partial tiles within the deferral window - Reset tracking when a full tile was downloaded fixes #104
references #97 Despite having worked fine on my machine, the report stated that they had issues downloading all certificates in time. This value might become configurable in the future.
fix(static-ct): defer partial tile fetching
Handle gosec G115 warnings.
Thought this was the build tag.
Sends messages to the broker in batches to improve throughput. Messages are collected and flushed when a maximum batch size is reached or after a configured wait time. This significantly reduces individual write calls. Messages are also dropped if the client isn't connected.
The code for reading and validating the config grew quite big, so it made sense to split it up into multiple files and have a method called Valid() that checks for config validity on each individual struct. In a further iteration we should split the config_test.go into multiple test files each testing only the respective individual files.
Made stream type configurable so that users can decide which stream they want to feed into kafka.
This should work well with a default kafka configuration. In the future this could become a config option.
Support for stream processing tools such as Kafka and NSQ
StaticCTClient.ctIndex was initialized to the checkpoint tree size (the number of entries already seen) but then updated to the index of the last processed entry, and the skip check treated it as the latter. The two conventions are off by one, with three visible effects: - Once caught up, currentTreeSize is always ctIndex+1, so the "no new entries" early return never fires. Every poll falls through to the partial tile logic, re-fetching a tile whose entries were all already processed. Before the deferral logic this happened on every poll, keeping the backoff reset; with the deferral logic it still happens once per maxPartialWait, forever, on idle logs. - The first entry after a startup at the checkpoint size (index N when ctIndex was set to N) is skipped, since N <= N. - Likewise, starting from a saved index of 0 skips entry 0. Make ctIndex mean "index of the next entry to process" throughout: skip entries with index < ctIndex, advance to entryIndex+1 after processing, and start the full tile loop at ctIndex/TileSize. With that, the tree size comparison is exact, a caught-up client does only the checkpoint fetch per poll, and no entry is dropped at startup. A saved recovery index written by the metrics package holds the last processed entry index, so after this change resuming from it re-emits that one entry, matching the behavior of the RFC 6962 scanner path (which also uses the saved index as StartIndex). Add tests that drive the client against an in-process static log and check that entries are emitted exactly once, that an idle log causes no tile fetches, and that the partial tile deferral still works. Updates #104
Fix off-by-one is Static CT path causing refetches of non-growing logs
Enhances client robustness by: - Making client resource cleanup safe from panics when internal channels are nil or closed repeatedly. - Updating the unregistration process to use client objects directly for more reliable removal. - Ensuring client internal channels are consistently initialized.
Without this, the worker would start at the last known ctIndex from when the worker was initially created as the ctIndex value does not get updated at runtime, since we're using the scanner implementation and our callbacks don't increment the ctIndex. This fix prevents processing already processed certificates again, after a worker crashed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge current changes from dev into master.