So much trouble to avoid introducing the optional monad.
woodruffw 9 hours ago [-]
Python’s type system has union types (including Option), and using them is pretty ergonomic. As TFA points out the problem is that `None` gets contextually reused in all kinds of places where a proper sentinel type is more appropriate.
aw1621107 8 hours ago [-]
Pretty sure Python's union types don't nest, though. If you could nest optionals then I feel that reduces the need for separate sentinel types since e.g., you can distinguish between "value not present" (None) and "null value present" (Some(None))
amluto 8 hours ago [-]
Indeed. Option[Foo] really just means Foo | None - you can’t have an Option[Foo] at runtime. Fundamentally, Python’s static typing system describes a language that is a sort of mediocre match for Python.
Another fun one is List[Foo]. Runtime Python has lists but you can’t ask a list what type of list it is, and you can’t write isinstance(a, List[Foo]).
ozyschmozy 8 hours ago [-]
Can you explain or give an example of nested optional types? I don't quite see how that makes sense
aw1621107 8 hours ago [-]
The idea is that optional values are either some value or no value, but sometimes the inner "some value" bit can itself be an optional value (in Rust this would be Option<Option<T>>, in C++ it'd be std::optional<std::optional<T>>, in Python it'd hypothetically be Option[Option[T]], etc.). Such a situation gives you three possible states:
- You have an inner value and the inner value is something non-null (an integer, a string, another optional, etc.)
- You have an inner value and the inner value is null
- You do not have an inner value
The problem, from what I understand, is that Option[T] in Python is equivalent to the union type T | None, and union types are flattened (e.g., (T | U) | V is equivalent to T | U | V) and duplicates are ignored (e.g., T | U | T is equivalent to T | U). As a result, Option[Option[T]] is equivalent to (T | None) | None, which is equivalent to T | None | None, which is finally equivalent to T | None, which means you lose the distinction between "No inner value" and "Inner value is present but is None".
As a more concrete (albeit somewhat contrived) example, say there's some API you can query for the owner of a particular document ID where the document ID may or may not refer to an actual document (e.g., maybe you got it from a cache and the document has since been deleted). One way to express this would be (using Rust syntax)
So if you get None, the document does not exist; if you get Some(None), the document exists but it has no owner; and if you get Some(<id>) the document exists and it has an owner.
(This can probably be done better by creating new enums, but it's an example so give me a bit of slack)
So now you're in the pickle the OP describes: you can end up with a None, but you no longer knonw whether it means "no document" or "no owner". The Python solution is to basically create a different name for either the outer None (e.g., MISSING as in the OP) or the inner one (e.g., NO_OWNER or the like).
FreakLegion 1 hours ago [-]
Most languages can do this if it's what you want. In Python it would be:
class Some[T]:
value: T
type Option[T] = Some[T] | None
def get_owner(id: DocumentId) -> Option[Option[UserId]]: ...
match get_owner(document_id):
case None:
...
case Some(None):
...
case Some(Some(x)):
...
aw1621107 11 minutes ago [-]
Yeah, that's more like what you would do if you wanted to translate the entire concept to Python instead of just the signature, and I probably should have mentioned that alternative as well (or should have said "A Python solution"). Can't say I've seen this approach used in practice, interestingly, though I'm probably not nearly as familiar with the Python that exists out there as most other commenters here.
The name clash is unfortunate, but luckily naming things is famously one of the easiest problems in CS so that should be trivial to solve :P
what 8 hours ago [-]
This seems really silly to me. Why does get_owner need to differentiate between the document exists and maybe has an owner or the document doesn’t exist and thus has no owner. In both cases getting no owner should suffice? Why are you using get_owner to try and figure out if the document exists? Can you give an example where this distinction actually matters?
setr 2 hours ago [-]
You run into this issue any time you do a LEFT JOIN in sql and the field you care about is nullable.
The right side value could be null because it was missing, or because it was literally null. You resolve by checking the join key, and if that is itself nullable, you’re forced to introduce a sentinel
Akronymus 4 hours ago [-]
Maybe a different example works better for your mental model:
A user can set custom settings, and those settings may or may not have a value assigned to them. For example, one setting is optimization level, another option is a debug flag.
You want to have a unified interface to get information about settings, queried through some intermediate system. (lets just say it is limited to querying specific settings, rather than getting the entire list, due to legacy constraints in this case) Because for some settings you just care about them existing and for others you care about the actual value, you have to differentiate between a setting that contains something, a setting that contains nothing, and one that doesnt exist.
aw1621107 7 hours ago [-]
> In both cases getting no owner should suffice?
Depends on what exactly you plan on doing with the information, I'd imagine. I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).
> Why are you using get_owner to try and figure out if the document exists?
Potential TOCTOU issues, perhaps? As I mentioned the document could have vanished out from under you so detecting this situation could let handle things appropriately (though to be fair that argument could probably extend to future operations as well).
Maybe a versioned filesystem would have made a bit more sense, where None would indicate a new version is available? Another example I've seen mentioned is trying to get a key-value pair from a JSON where the key may be missing.
But in any case, I did say it was a bit of a contrived example, so cut me a bit of slack :P It was just the first thing that came to mind.
The examples sibling comments give are probably more practical.
what 6 hours ago [-]
> I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).
If you care whether a document exists or not, you should use an API that exclusively tells you that instead of expecting everything related to documents telling you if the document exists or not. Otherwise you’re overloading the meaning. A non existent document has no owner. An API to fetch the owner of the document should not care if it exists or not, it does not matter.
aw1621107 6 hours ago [-]
As I said, that sounds like a recipe for potential TOCTOU errors.
But again, this was an off-the-cuff example that I explicitly described as contrived from the start. I could probably flesh it out to make more sense, but I think it'd be easier just to refer to sibling comments rather than bikeshed this to death.
what 4 hours ago [-]
Not really? It returns no owner if the document is deleted or just does not exist, which is accurate. Why do you need get_owner to tell you if the document exists or not? If anything it should be an error if the document doesn’t exist because you’ve violated a precondition.
aw1621107 25 minutes ago [-]
> It returns no owner if the document is deleted or just does not exist, which is accurate.
"I don't know" is not the same as "definitely no owner". You may handle the two answers the same for a particular scenario, but then again you may not.
Again, consider the case where get_owner is working on some kind of cache (in which case sure, the function name could stand to be improved). If the document is no longer in the cache it could be because it was actually deleted or because it was simply evicted to make room for other entries. In general you cannot assume that a missing cache entry is equivalent to the document existing and having no owner.
Or similarly (and also yet again), consider the case where get_owner is working on a versioned filesystem, in which case the outer optional could indicate that the (generational) document ID is out of date. In such a case treating "I don't know" and "no owner" the same is incorrect because the owner could have been updated from "no owner" to something else (such a case would likely be more realistically handled by returning a Result<Option<UserId>, SomeErrorKind>, though, given the myriad other ways a file system handle can go stale)
> Why do you need get_owner to tell you if the document exists or not?
Because sometimes there is no good alternative. Asking a cache whether it has an entry and then trying to get the entry value in a separate operation (particularly in a concurrent environment) is a classic setup for TOCTOU errors. You can say something similar for the versioned file system example.
> If anything it should be an error if the document doesn’t exist because you’ve violated a precondition.
I mean, enabling that kind of handling is precisely the point of the outer optional?
lanyard-textile 3 hours ago [-]
It's analogous to how we use undefined and null in JS.
Undefined = No value known. Think 0 rows found, the 1st row's column is undefined.
Null = Value known. It is void of value. The 1st row is populated, and this column has no value.
__s 8 hours ago [-]
Say you select nullable column into vec, Vec<Option<T>>. vec.get returns Option<Option<T>>, which distinguishes null from out of bounds
I'd like nested Nullable in ClickHouse to use CoalescingMergeTree to support postgres unchanged TOAST values easily without getting mixed up on nulls
david2ndaccount 3 hours ago [-]
The answer is they don't in practical code but they come up in generic code, where you want to wrap something that is already returning an optional in your optional. In languages that do support optional[optional[whatever]], if you every actually write that you know you've messed up your API.
yccs27 8 hours ago [-]
If you have a map, get(key) returns Option[ValueT]: None if the key is missing, Some(value) otherwise. If the value type itself is, say, Optional[String], your return type is Optional[Optional[String]]. Here None still always means the key is missing, while Some(None) means the value is just None.
Waterluvian 9 hours ago [-]
I’m curious what that would practically look like in Python.
I bet this would be a good use of vibe coding. Experiment with what-ifs. I can imagine being able to “show not tell” and reduce the signal-to-noise of comment opinions.
evolve-maz 4 hours ago [-]
Python makes it very easy to do these experiments without changing cpython itself. For example, a lot of the cases for the "not set" Sentinel can be handled by making it MyClass | None. Treat None as "not set". If you need a "no-op" then you can give MyClass a field or something which implies it does nothing. And then override the bool dunder method to make it return false if the no op field is set to true. If you really want, you can have a classmethod which returns a "no op" instance of MyClass so people can easily call your func with MyClass.no_operation() when you want no op, and null when its not actually set.
Sentinels just help you wrap that up nicely and for more general use cases, but theres value in stepping back and thinking about where the unset vs no op case matters.
amluto 8 hours ago [-]
It would look like a very different language that would be thoroughly incompatible with the existing ecosystem.
_ZeD_ 3 hours ago [-]
Or, you know, you can use the exceptions
lpghatguy 9 hours ago [-]
When I was writing a bunch of Lua, I was a big fan of having a None sentinel value (Lua uses 'nil' for representing absence) and it was useful for a lot of things similar to Pydantic's MISSING.
This distinction is something that I think is actually handy about JavaScript, and distinguishing between 'null' and 'undefined' is sometimes useful for similar reasons.
nomel 7 hours ago [-]
I feel like faster exceptions would be a better solution for this sort of thing. Leave functions returning the happy path with excepted values. Exceptional things, like missing values being handled at the level where of abstraction that actually cares to catch it:
RantyDave 10 hours ago [-]
"A simple cache returns None both for a missing key and for a key that maps to None."
No, it returns KeyError. Or at least it should.
travisd 9 hours ago [-]
it’s maybe less “pythonic” but at $dayjob we’ve very actively moved away from exceptions for things in favor of things that are statically type checked. too many things that become runtime bugs instead of type checking bugs.
with match now being available, we even return explicit error objects and the ergonomics are pretty good (reminds me of rust).
nomel 7 hours ago [-]
How are these two things related? If you don't use exceptions, you just have to implement them yourself by manually bubbling up the error, all the way up the stack, rather than using the dedicated side channel to do so.
The problem I see is there's no way to enforce the check that you actually handled/bubbled up the error. With exceptions, you just crash if you don't handle it, which is much preferred if determinism is the goal. Is there some way to use the type system to do this? Maybe incoming are UnhandledResult outgoing are some HandledResult or something like that?
aw1621107 6 hours ago [-]
If you're going to use the returned value you'd probably return the moral equivalent of Rust's Result/Haskell's Either/etc. types. As long as the returned types don't happen to share the necessary API(s) the type checker should yell at you if you don't branch on the returned value.
I don't think Python has the equivalent of C++'s [[nodiscard]] or Rust's #[must_use], though, so if there are cases where the return value can legitimately be ignored then return values risk missing errors that would show up under exceptions.
0cf8612b2e1e 9 hours ago [-]
The author’s example chose to use the syntax that returns None on missing. If they wanted clarity the standard API does raise.
There are better examples that could have been highlighted.
lyorig 9 hours ago [-]
Perhaps the author has had their fair share of run-ins with libraries that didn't think their API through like that. But yes, they should.
nomel 7 hours ago [-]
Very many people have been taught that exceptions are bad and should be avoided, because google avoids them.
deathanatos 4 hours ago [-]
The adverse knee-jerk hate for exceptions far predates Google also saying they're bad, but certainly that didn't help.
jMyles 4 hours ago [-]
> No, it returns KeyError. Or at least it should.
Surely you meant _raises_? Or did you literally mean to return an exception class?
shepherdjerred 2 hours ago [-]
I hate sentinel values. Even/especially with AI-generated code.
1. If your date does not have a value, do not return 0, an empty string, or None. Use a discriminated union and model your domain rigorously.
2. If your data model accepts None/undefined as a value, that should translate to your business model as well. For example
3. Do not make your `name` field as `string | undefined` just because there might be an intermediate loading or processing state. It's better to model that as something like `Loaded<string>` where Loaded is something like a Result or Optional type (though ideally more rigorously defined)
I like it. I can't see how to do better while remaining compatible with Python idioms.
michalc 8 hours ago [-]
You could even say you never knew what you were… MISSING ;-)
est 5 hours ago [-]
I just return a customized Exception type. Works about the same.
throwaway2037 7 hours ago [-]
Here is the sample code from the article:
> MISSING = sentinel("MISSING")
> def update_email(user_id: int, email: str | None | MISSING = MISSING) -> None:
> if email is MISSING:
> return # nothing to update
> elif email is None:
> db.delete_email(user_id)
> else:
> db.set_email(user_id, email.lower()) # narrowed to str
I see this style frequently from long time Python programmers. The language is so dynamic that people begin to think this is a good way to program. Hot take: It is not. Why not two different functions called delete_email() and set_email()? That would be more clear.
tlonny 6 hours ago [-]
I think this sort of pattern makes more sense when the fn is update_user and allows the caller to mutate multiple fields at once.
jMyles 4 hours ago [-]
I have a library that's approximately half joking that I use for (sentinel and other) special value constants. If you like reading test suites, this one will get ya:
Another fun one is List[Foo]. Runtime Python has lists but you can’t ask a list what type of list it is, and you can’t write isinstance(a, List[Foo]).
- You have an inner value and the inner value is something non-null (an integer, a string, another optional, etc.)
- You have an inner value and the inner value is null
- You do not have an inner value
The problem, from what I understand, is that Option[T] in Python is equivalent to the union type T | None, and union types are flattened (e.g., (T | U) | V is equivalent to T | U | V) and duplicates are ignored (e.g., T | U | T is equivalent to T | U). As a result, Option[Option[T]] is equivalent to (T | None) | None, which is equivalent to T | None | None, which is finally equivalent to T | None, which means you lose the distinction between "No inner value" and "Inner value is present but is None".
As a more concrete (albeit somewhat contrived) example, say there's some API you can query for the owner of a particular document ID where the document ID may or may not refer to an actual document (e.g., maybe you got it from a cache and the document has since been deleted). One way to express this would be (using Rust syntax)
So if you get None, the document does not exist; if you get Some(None), the document exists but it has no owner; and if you get Some(<id>) the document exists and it has an owner.(This can probably be done better by creating new enums, but it's an example so give me a bit of slack)
A naive port of this signature to Python would be
But to type checkers this is equivalent to So now you're in the pickle the OP describes: you can end up with a None, but you no longer knonw whether it means "no document" or "no owner". The Python solution is to basically create a different name for either the outer None (e.g., MISSING as in the OP) or the inner one (e.g., NO_OWNER or the like).The name clash is unfortunate, but luckily naming things is famously one of the easiest problems in CS so that should be trivial to solve :P
The right side value could be null because it was missing, or because it was literally null. You resolve by checking the join key, and if that is itself nullable, you’re forced to introduce a sentinel
A user can set custom settings, and those settings may or may not have a value assigned to them. For example, one setting is optimization level, another option is a debug flag.
You want to have a unified interface to get information about settings, queried through some intermediate system. (lets just say it is limited to querying specific settings, rather than getting the entire list, due to legacy constraints in this case) Because for some settings you just care about them existing and for others you care about the actual value, you have to differentiate between a setting that contains something, a setting that contains nothing, and one that doesnt exist.
Depends on what exactly you plan on doing with the information, I'd imagine. I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).
> Why are you using get_owner to try and figure out if the document exists?
Potential TOCTOU issues, perhaps? As I mentioned the document could have vanished out from under you so detecting this situation could let handle things appropriately (though to be fair that argument could probably extend to future operations as well).
Maybe a versioned filesystem would have made a bit more sense, where None would indicate a new version is available? Another example I've seen mentioned is trying to get a key-value pair from a JSON where the key may be missing.
But in any case, I did say it was a bit of a contrived example, so cut me a bit of slack :P It was just the first thing that came to mind.
The examples sibling comments give are probably more practical.
If you care whether a document exists or not, you should use an API that exclusively tells you that instead of expecting everything related to documents telling you if the document exists or not. Otherwise you’re overloading the meaning. A non existent document has no owner. An API to fetch the owner of the document should not care if it exists or not, it does not matter.
But again, this was an off-the-cuff example that I explicitly described as contrived from the start. I could probably flesh it out to make more sense, but I think it'd be easier just to refer to sibling comments rather than bikeshed this to death.
"I don't know" is not the same as "definitely no owner". You may handle the two answers the same for a particular scenario, but then again you may not.
Again, consider the case where get_owner is working on some kind of cache (in which case sure, the function name could stand to be improved). If the document is no longer in the cache it could be because it was actually deleted or because it was simply evicted to make room for other entries. In general you cannot assume that a missing cache entry is equivalent to the document existing and having no owner.
Or similarly (and also yet again), consider the case where get_owner is working on a versioned filesystem, in which case the outer optional could indicate that the (generational) document ID is out of date. In such a case treating "I don't know" and "no owner" the same is incorrect because the owner could have been updated from "no owner" to something else (such a case would likely be more realistically handled by returning a Result<Option<UserId>, SomeErrorKind>, though, given the myriad other ways a file system handle can go stale)
> Why do you need get_owner to tell you if the document exists or not?
Because sometimes there is no good alternative. Asking a cache whether it has an entry and then trying to get the entry value in a separate operation (particularly in a concurrent environment) is a classic setup for TOCTOU errors. You can say something similar for the versioned file system example.
> If anything it should be an error if the document doesn’t exist because you’ve violated a precondition.
I mean, enabling that kind of handling is precisely the point of the outer optional?
Undefined = No value known. Think 0 rows found, the 1st row's column is undefined.
Null = Value known. It is void of value. The 1st row is populated, and this column has no value.
I'd like nested Nullable in ClickHouse to use CoalescingMergeTree to support postgres unchanged TOAST values easily without getting mixed up on nulls
I bet this would be a good use of vibe coding. Experiment with what-ifs. I can imagine being able to “show not tell” and reduce the signal-to-noise of comment opinions.
Sentinels just help you wrap that up nicely and for more general use cases, but theres value in stepping back and thinking about where the unset vs no op case matters.
This distinction is something that I think is actually handy about JavaScript, and distinguishing between 'null' and 'undefined' is sometimes useful for similar reasons.
No, it returns KeyError. Or at least it should.
with match now being available, we even return explicit error objects and the ergonomics are pretty good (reminds me of rust).
The problem I see is there's no way to enforce the check that you actually handled/bubbled up the error. With exceptions, you just crash if you don't handle it, which is much preferred if determinism is the goal. Is there some way to use the type system to do this? Maybe incoming are UnhandledResult outgoing are some HandledResult or something like that?
I don't think Python has the equivalent of C++'s [[nodiscard]] or Rust's #[must_use], though, so if there are cases where the return value can legitimately be ignored then return values risk missing errors that would show up under exceptions.
Surely you meant _raises_? Or did you literally mean to return an exception class?
1. If your date does not have a value, do not return 0, an empty string, or None. Use a discriminated union and model your domain rigorously.
2. If your data model accepts None/undefined as a value, that should translate to your business model as well. For example
3. Do not make your `name` field as `string | undefined` just because there might be an intermediate loading or processing state. It's better to model that as something like `Loaded<string>` where Loaded is something like a Result or Optional type (though ideally more rigorously defined)
Here is what I've done for years in TypeScript, though now in a nice package thanks for AI: https://github.com/shepherdjerred/monorepo/blob/main/package...
https://github.com/nucypher/constantSorrow/blob/master/tests...