Keyset paginator for AR::Relation

What does this MR do?

This is a proof of concept to build keyset based paginator which can be used in HAML views and APIs.

How it works

The backend uses the generic keyset pagination implementation and makes it available within the ActiveRecord::Relation instances.

The paginator requests per_page + 1 rows from the database. This helps makes it possible to tell if there is next page without an extra query. To make it work for both direction we include a direction _kd key into the cursor (n - next, p - previous).

• If no cursor is given, we assume that we're on the first page.
  • In this case we're sure that there is no prev page.
• If the returned record count is per_page + 1 and the current cursor direction is "next" (n), then we're sure that there is a next page.
• If the returned record count is per_page + 1 and the current cursor direction is "prev" (p), then we paginate backwards, and we're sure that there is a prev page.
• If the returned record count is less or equal to per_page and the current cursor direction is "next" (n), then we're at the last page.
• If the returned record count is less or equal to per_page and the current cursor direction is "prev" (p), then we're at the first page.

Usage

To compare the API's let's see how kaminari works:

users = User.order(id: :desc).page(params[:page]).per(20)

Keyset pagination:

users = User.order(id: :desc).keyset_paginate(cursor: params[:cursor], per_page: 20)

Main differences:

• No page parameter. The cursor parameter contains the keyset attributes (in this example the id and the direction: { id: 2, _kd: 'n'}), base64 encoded.
• keyset_paginate method call will not alter the ActiveRecord::Relation object.
• keyset_paginate method returns a paginator object which contains the loaded record and cursors for the first, previous, next and last pages.
• No page number. Keyset pagination does not support page number.

Feedback needed:

• Not sure about the Base64 encoded cursor.
• Does it make sense to have link to first and last pages?
• Should we remove some params from the cursor like kaminari does (might be added if we use POST)? https://github.com/kaminari/kaminari/blob/00657f252244413b512349f4de20c63ee478baa5/kaminari-core/lib/kaminari/helpers/tags.rb#L5

Limitations

Not all order() calls can be turned to keyset paginated queries automatically. I'm planning to have a doc about this, the summary:

• order(id: :asc) or order(id: :desc) works
• ordering by other column also works, if we have a tie-breaker (primary key): order(created_at: :asc, id: :asc)

Example

This diff contains an example implementation for the admin / users page. You can apply it a diff and test it out.

keyset_pagination.diff

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

• 📋 Does this MR need a changelog?
  • I have included a changelog entry.
  • I have not included a changelog entry because not use facing change.
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team