This message was deleted.

# compose

[Slackbot]

This message was deleted.

[shikasd]

Correct, the states are not cleared after invalidation, it only reset on rerun or when SnapshotStateObserver#clear is called.

[shikasd]

Zach is not the author, this was mostly thought out and implemented by Chuck Jazdzewski

[Zach Klippenstein (he/him) [MOD]]

Good catch, i'll work on fixing it

[Chuck Jazdzewski [G]]

I am confused by (1) and (2) as you ask for where to find this code that does what you ask but the code you quote is exactly the code you are looking for. I am uncertain what you were expecting to find, maybe you could explain what you thought would be there that you didn't find.

For example, (1) includes a call to advance { which is the call that advances the snapshot. Advancing the snapshot is a rather trivial operation that changes ID to the highest snapshot number and ensures that all open snapshots prior to it are explicitly excluded. It does this when a new snapshot is taken to ensure that the child snapshot will see all changes made to the parent at up to the point of the snapshot but not after.

For (2), the code you are looking for is the if statement you quote. The parent doesn't need to advance if changes in the parent are already not visible to the child.

Snapshots is an implement of the MVCC algorithm. An explanation of the underlying principle behind the snapshot system can be found here https://en.wikipedia.org/wiki/Multiversion_concurrency_control. The comments do not explain this algorithm as it is fairly well-known and external sources explain it better than I could. This doesn't cover some of the details around nested transactions (which I call snapshots as snapshots are not durable) but @Zach Klippenstein (he/him) [MOD] explains this better than I could in his talks and blog posts.

For (3), this is to implement `SnapshotMutationPolicy`'s merge method which can be used to implement conflict free data types. By default, any modification to a value is considered conflicting with the same value modified in another snapshot. Through a mutation policy you can define that two snapshot that change the value can be merged. This can result in a third value that includes the changes made by both snapshots. I give an example for this in the documentation regarding the count of a set of physical items where snapshot A and snapshot B can both consume or produce a physical items without conflict. You can learn more about this here (https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). Any CFRD an be implemented by this mechanism as well as non-replicating conflict free values such a count of resources.

For why (3) needs to advance the snapshot here is because, taking a count as an example, if you decrement the count in both A and B, merging them means adjusting the value by the sum of the differences (e.g. if foo is a mergable count then foo = foo - 2 in A and foo = foo - 3 in B would merge to produce foo = foo - 5. If foo starts at 10 then A sees `8`and B sees 7 but the merged value is 5.) Since neither A or B has seen the new value (5 in the example), a new snapshot ID is required to contain the merged value. Advancing the snapshot allocates this new snapshot.

[Chuck Jazdzewski [G]]

The assertions you are making about the snapshot ID are not valid for a number of reasons. The only constraint on the ID of a snapshot is that new snapshots have a greater ID than previous snapshots. The ID of a snapshot may change overtime but what is important as the effect on the which record is considered valid. This is a combination of the ID and the invalid ID sets. The ID is an arbitrary point in time where record with an ID greater than the snapshot ID are invalid. An ID less than the snapshot is valid if it is not explicitly invalid. The ID is effectively the timestamp of when the snapshot was taken. Any assertions over how fast or slow this number changes are invalid as long as the constraint that it is monotonically increasing is preserved. When a snapshot advances or the implications to the global snapshot's ID are should be viewed as implementation details that may change in the future.

As for stating the IDs are corrupted. They are not. It is critical that the snapshots provide the MVCC model, perform well, and use as little memory as practicable. To do this, the implementation is rather intricate in places. For example, the fact that the ID of a record changes is not "corruption" but record reuse. When the snapshot system determines that a snapshot ID has been retired (it has been applied or abandoned) the records that are only visible to it are eligible to be reused. This is a case where the records are no longer visible and were cleared by copying the current record to all of the now record for reuse. It is important we do this to avoid holding on to reference passed the point when they are needed.

[Chuck Jazdzewski [G]]

I am unclear what you mean by "the write of`300`to the state when unobserved". By whom? If you mean the apply observer then that is by design as the apply observer only observes changes to the global snapshot. As the change occurs in a nested snapshot, it is not visible to the global snapshot yet. The apply observers are only notified when an apply occurs on the global snapshot. The intent is to observe changes to global state, not nested snapshot state.

"...even if I accept that the exact IDs are an implementation detail and can be any values the system sees fit for as long as applying a snapshot guarantees the parent has a larger ID. What are the rules?

The rules is as I stated them above. The only thing that maters is what the `valid`function returns. valid is true when a record is potentially visible to the snapshot. This is currently when the record id (the snapshot a record was created in) is less than or equal to its own snapshot and not explicitly made invalid by being in the `invalid`set. readable returns the newest valid record (i.e. with the highest id) .

Snapshots could be implemented by each snapshot having just an `invalid`set and, as new snapshots are added, they are added to every snapshot's invalid set. Using the snapshot id relative value is effectively the same thing. As new snapshots always have ids greater than previous snapshots new snapshots are effectively added to the set of invalid snapshots of older snapshots. The manipulation of these number is just a trick to avoid having to maintain a list of snapshots and updating them whenever a new one is created. What is important is which record is valid and which is not. How this is determined is not visible through public API.

With this in mind, you must ignore the snapshotId. It is an implementation detail that may be useful when debugging but should never be used in a way that changes the behavior of the application as you don't have enough public information to reason about it. The values returned by a state object is the only contract (e.g. what is returned by readable and writable). What the snapshotId is an at any time is an implementation detail.

The general rule for snapshots are that snapshots see only the values the parent snapshot had at the point the snapshot was taken, all changes that have been explicitly made to them, and all changes from nested snapshots applied into them. They are isolated from all other changes. When a changes is applied to a snapshot, all the changes applied are seen as one atomic change. In other words, snapshots are atomic, consistent and isolated. The guarantees are to the snapshot instances, not to what id they may have at any moment.

[Chuck Jazdzewski [G]]

There was no write of 300 (or 30). It's a value that was produced by the merge lambda given to MyIntState. This reflects the merge policy of the type which interprets changes to the value as a delta, not the literal values. In this case writing 200 means increase MyIntState by 200 and a write of 100 means increase it by 100. Therefore a merge of these two requests is an increase of the MyIntState instance to 300 (i.e. 100 + 200). If you remove the merge policy you will get a merge conflict as one snapshot writes 100 and the other writes 200. Only one can succeed. A merge policy allows snapshots that would otherwise collide to produce a merged value instead of failing. This merging happens when the snapshot is applied. Also the statement (4) above about `nested1.apply()`in `write2`is misleading, at least from what I can tell. The value of state is 300 after the `nested1.apply()`which is asserted by `kotlin.test.assertEquals(300, state.read())`above. This means it is consistent the 10, 20 case above it.

[Chuck Jazdzewski [G]]

That is because you registered an apply observer for the global which sees all changes and write observers in the nested snapshots which observe writes to in the snapshot. These are not the same thing and have different behavior as you point out. Merges are not writes, as I discussed above, but they are changes. Snapshots do not have apply observers (which would see these changes), only the global snapshot does,

Read observers observe calls to readable and write observers observes calls to writable. The apply observer observes calls to apply of the global snapshot. The changes show up in a apply observer because it was applied to the global snapshot. They do not show up to a write observer as they were not written to by the snapshot that had the write observer. The merged values were "written" in a synthetic snapshot implied by the merge.

[Chuck Jazdzewski [G]]

This is inconsistent but intentionally so. Nested snapshot read and write observers will call the parent snapshot read and write observers. This should be optional behavior since, as you point out, it is inconsistent with the global snapshot. I will create a feature request to make this behavior optional.

[Chuck Jazdzewski [G]]

b/492143967

[Chuck Jazdzewski [G]]

Not exactly. The write observers are notified synchronously in the call to writable. For example, someIntState.value = 10 will call the write observer immediately. Same with read observers, they are called immediately during readable. Read and write observers are ignored during apply.

Like I said above, write observers and apply observers are two entirely different things. They are inconsistent because they are different. You should only use apply observers when observing for changes. Write observers should be almost never be used. For example, they are needed for the global snapshot manager (to know when to schedule an advance of the global snapshot) but are not used for anything else in Compose.

The pattern is to use a read observer to detect which objects an expression reads and an apply observer to detect when the state objects the expression read change. The write observer should not be used for observation. As you point out, write observers are incomplete for this as they only are notified when a write occurs; not when the object changes. Also they will be notified of write even if the write eventually is abandoned and they are called redundantly (multiple time, once for each write). Write observers are both called to little and to much. Apply observers, on the other hand, will see every object that has changed and only the objects that have changed (e.g. not snapshot that are abandoned) and are called once per non-nested snapshot apply (or global advance).

The snapshot id is only relevant to understand how the snapshot system delivers on the promise of consistent, atomic, and isolated changes. It otherwise should be ignored.

[Chuck Jazdzewski [G]]

> But this seems to be in direct contradiction, namely of ⓒ and ⓓ, which confuses me.
The test is incomplete. The write observer is called earlier than the call to apply. The test just test if it occurred at all, not when. Your test seems to conflate write and apply observers. As I said, they are different and they are called at completely different times. Write is called while in the snapshot (e.g. in enter()). Apply observers are called after the snapshot applies.

> Are you just trying to say that applying to a nested snapshot is equivalent to writing (ⓒ) all diffs to it before merging (ⓓ)?
No. I write observers are not called during apply at all. Nested snapshot do not call any observers when they are applied.

> Or this in Recomposer.composing?
I missed that one! Sorry about that. That is used to detect writes during composition that may invalidate other parts of composition. I am not sure how I missed that... Write observer allow the snapshot to observe itself which is compose does. Composition uses the write observer observers itself (e.g. changes made during composition). The apply observer observe changes outside composition.

[Chuck Jazdzewski [G]]

I answer the question about how nested snapshot call their parent snapshots above. I also submitted a feature request to allow this to be more consistent with the global snapshot.

As for how composition works, I am willing to answer specific questions but a general question like this is more than I can answer in a chat window. There are blog posts and videos that describe this and at least one book. I recommend you start there.

[Chuck Jazdzewski [G]]

Yes.