Repository navigation
Make aten::logaddexp, logaddexp2 and log_sigmoid numerically stable - #3068
Open
Raashish Aggarwal (raashish1601) wants to merge 1 commit into
Open
Raashish Aggarwal (raashish1601) wants to merge 1 commit into
Raashish Aggarwal (raashish1601) wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aten_logaddexp,aten_logaddexp2andaten_log_sigmoidused the direct formulas, which overflow or underflow well inside the float32 range:logaddexp(a, b) = log(exp(a) + exp(b)): for a, b around 100 or moreexpoverflows and the result isinf(PyTorch returns e.g.1000.69forlogaddexp(1000, 1000)).logaddexp2has the same problem with2**a + 2**b.log_sigmoid(x) = log(sigmoid(x)): for x below about -100,sigmoidunderflows to 0 and the result is-infinstead of aboutx.They now use the usual stable forms (the ones PyTorch uses):
logaddexp:max(a, b) + log(1 + exp(-|a - b|))logaddexp2:max(a, b) + log2(1 + 2**-|a - b|)log_sigmoid:min(x, 0) - log(1 + exp(-|x|))For
a == b == +/-inf,a - bis NaN, sologaddexp/logaddexp2returna + log(2)/a + 1when the inputs are equal, which gives the right infinity (and the same value as the formula for finite inputs).Testing:
ops.aten.logaddexp.large,ops.aten.logaddexp2.largeandops.aten.log_sigmoid.largeOpInfos with values from -1000 to 1000 and +/-inf (all pairs for the binary ops). All three fail onmainand pass with the change.pytest tests/function_libs/torch_lib/ops_test.py -k "logaddexp or log_sigmoid or logsigmoid"passes locally, and an exported model with these ops matches PyTorch on large inputs through onnxruntime. ruff check / format are clean on the changed files.