Skip to content

CBEF-128: Generate integer keys from a Couchbase sequence by default - #189

Draft
davidkelly wants to merge 1 commit into
sync-option-bfrom
int-key-sequences
Draft

davidkelly wants to merge 1 commit into
sync-option-bfrom
int-key-sequences

Conversation

@davidkelly

@davidkelly davidkelly commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Couchbase has no identity columns, and nothing generated a value for an integer key that EF
Core marks as generated on add: every new entity was written as document 0, and the second
insert failed with DocumentExistsException. This gives such keys a Couchbase sequence by
default. Stacked on #188 (which is on #190), so the diff here shows only this change.

Change

  • CouchbaseKeySequenceConvention: a single-property integer key generated on add, with no
    explicit generator, uses a sequence named <EntityName>Sequence. Derived types share the
    root's sequence, so TPC keys are unique across collections.
  • Values are assigned at SaveChanges. Add gets a temporary value and does no I/O, so it works
    synchronously; the SaveChanges interceptor fetches all the values each sequence needs in one
    query, and EF updates the dependents' foreign keys.
  • A missing sequence throws InvalidOperationException with the CREATE SEQUENCE statement to
    run. EnsureCreatedAsync creates the sequences.
  • TPC integer keys are now generated, and the model validator accepts a sequence-backed TPC key.
  • CouchbaseModelValidator was never registered. It is now, without its empty
    ValidatePropertyMapping override, which would have disabled EF's check for unsupported
    property types.
  • Docs (sequences.md, including the collision risk with existing integer-keyed documents) and
    CHANGELOG.
  • Spec harness: removes the int-key emulation from 28 fixtures. It also forced TPC keys that EF
    configures as ValueGeneratedNever to be generated, which is what failed every TPC
    GearsOfWar test. Test sequences now start at 1,000,000, above the keys EF's fixtures seed.
    Deletes the int-key-not-generated and tpc-key-generation rows from root-causes.tsv.

Tests

  • Unit 1,223/1,223 (15 new); integration 404/404 (4 new: synchronous Add, foreign keys, a
    key the app sets, a missing sequence).
  • Spec tests, local 3-node 8.0.1 cluster: no new failures. LoadCouchbaseTest 0 → 3,137/3,137,
    TPCGearsOfWar 0 → 846/1,183. known-failures.txt will be regenerated from a full CI run
    before merge.

Jira: CBEF-128

@emilienbev emilienbev left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really like this one. Assigning sequence values at save time keeps Add free of I/O, and CouchbaseSequenceLocation puts the location logic in one place. A few small things, and one question about the batch query.


// NEXT VALUE FOR increments once per document, so one value per array element. The array
// comes back as one JSON string because ExecuteScalar returns a single value.
var query = $"SELECT RAW ENCODE_JSON(ARRAY NEXT VALUE FOR {sequence} FOR x IN ARRAY_RANGE(0, {count}) END)";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gets all the values from one row, so it depends on NEXT VALUE FOR running once per array element. The comment says it increments once per document, and this query has a single document. Is the per-element behaviour documented for every server version we support? If not, maybe SELECT RAW NEXT VALUE FOR s FROM ARRAY_RANGE(0, n) AS x with a reader is safer, and it drops the ENCODE_JSON step.

/// EF Core set from one) is in the update entry only: EF Core copies it to the CLR object when
/// it accepts the changes, after the documents are written.
/// </remarks>
internal static string GetLegacyPrimaryKey(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small thing. This copies GetPrimaryKey(IEntityType, object) and only changes where the values come from, and the old method has no caller in src now. Could we keep one loop that takes a value reader?

/// </summary>
/// <param name="dependencies">The selector's dependencies.</param>
/// <param name="optionsBuilder">Unused; kept so existing service registrations still resolve.</param>
public CouchbaseValueGeneratorSelector(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small thing. Nothing creates CouchbaseSequenceValueGenerator anymore. Only its IsTypeSupported is still used. Could we move that check to a small helper and remove the generator, or mark it [Obsolete] if we want to keep the public API? The unused optionsBuilder parameter could go too, since DI resolves the constructor without it.

// successful insert/update regardless of whether it's also a concurrency token (harmless and
// keeps the property useful for inspection even without concurrency enforcement).
/// <summary>
/// Gives every added entry whose key uses a Couchbase sequence its value from the sequence, as

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. This method landed between the [CouchbaseMeta(Cas)] comment and GetCasProperty, so that comment now sits on the wrong method. Could we move the new method above it?

/// </summary>
private async Task GenerateSequenceValuesAsync(IList<IUpdateEntry> entries, CancellationToken cancellationToken)
{
Dictionary<(CouchbaseSequenceLocation, CouchbaseSequenceOptions), List<(IUpdateEntry, IProperty)>>? groups = null;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. The options only feed the error message, but they're part of the group key. Two properties on the same sequence with different options would cause two queries. Could we group by location only?

public class ConferencePlannerCouchbaseTest(ConferencePlannerCouchbaseTest.ConferencePlannerCouchbaseFixture fixture)
: ConferencePlannerTestBase<ConferencePlannerCouchbaseTest.ConferencePlannerCouchbaseFixture>(fixture)
{
// The base tests rely on a rolled-back transaction to undo their changes; Couchbase does not

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, this works around a rollback that doesn't undo changes. That sounds like a provider issue rather than a harness one. Could it get a row in root-causes.tsv, so it stays visible until someone looks at it?

int-key-not-generated Insert failed for key 0 in keyspace .*\n.*DocumentExistsException A generated integer key gets no value, so every new entity is written as document 0
seed-doc-exists Insert failed for key .*\n.*DocumentExistsException Seeding inserts a key that already exists (owned types with explicit keys, among others)
seed-empty-key Insert failed for key in keyspace Document key is empty: the save path reads the key by CLR reflection, not from the update entry
key-getter Property Get method was not found Document key read by CLR reflection fails for a property with no getter

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, the save path now reads key values from the update entry, so the key-getter row might be fixed too. Worth checking in the full CI run before merge?

Couchbase has no identity columns, and nothing generated a value for an
integer key that EF Core marks as generated on add. EF kept a temporary
value in the change tracker, the save path read 0 from the entity, and
every new entity was written as document "0": the second insert failed
with DocumentExistsException.

- CouchbaseKeySequenceConvention gives such a key (single property,
  integer, generated on add, no explicit generator) a sequence named
  <EntityName>Sequence. Derived types share the root's sequence.
- Values are assigned at SaveChanges: Add gets a temporary value from
  the relational selector, with no I/O, so a synchronous Add works.
  The database wrapper fetches all the values each sequence needs in
  one query (ENCODE_JSON(ARRAY NEXT VALUE FOR ... ARRAY_RANGE(0, N)))
  and sets them as store-generated values, like an identity column:
  EF updates the dependents' foreign keys, and the entity gets the key
  when the save completes. A failed save keeps the temporary values.
- The legacy document key reads its values from the update entry, in
  the old part order, so a composite key that includes a generated
  foreign key is not written as "0_0".
- A missing sequence (query error 19106) throws InvalidOperationException
  with the CREATE SEQUENCE statement to run.
- TPC: CouchbaseValueGenerationConvention lets TPC integer keys be
  generated (the relational convention does not, as an identity column
  is per table), and the model validator accepts a sequence-backed TPC
  key. Any other generated TPC key still gets EF's warning.
- CouchbaseModelValidator was never registered. It is now, with its
  empty ValidatePropertyMapping override removed: that override would
  have disabled EF's check for unsupported property types.
- CouchbaseSequenceLocation is now the one place that resolves a
  sequence's bucket and scope, for value generation, EnsureCreated and
  EnsureDeleted. The scope is still the context's, so existing
  UseSequence sequences are not moved.

Unit tests cover the convention, TPC validation, temporary values on
Add and sequence locations; integration tests cover a synchronous Add,
foreign keys, a key the application sets, a missing sequence, a
read-only-before-save key, and a composite key with a
sequence-generated foreign key.
Docs (sequences.md) and CHANGELOG describe the default and the
collision risk with existing integer-keyed documents.

Spec harness: CouchbaseSpecModel.Apply(generateIntKeys: true) emulated
this fix in 28 fixtures and also forced TPC keys that EF configures as
ValueGeneratedNever (GearsOfWar's Faction.Id) to be generated, which
failed every TPC GearsOfWar test with TpcStoreGeneratedIdentityWarning.
The emulation is gone; the harness only starts test sequences at
1,000,000, above the explicit keys EF's fixtures seed.
ConferencePlanner reseeds after each test: SessionsController_Put now
passes and its rename leaked into Search("C#").
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants