git - GitLab workflow to prevent regressing binary files - Stack Overflow

Summary: How can my team manage merge requests in GitLab to prevent a binary file from ever going "back in time"?

Background

Details of the actual root need, to avoid an XY problem.

I'm helping with infrastructure for a team that's ~10 developers and growing quickly. The project requires certain binary files to be used as ~source that's paired with code, including .xlsx files. We are trying to establish processes and configurations that ~prevent a developer resolving a merge conflict in these files from running over someone else's changes.

Below is the process we've come up with, but there's a race condition in this process. How can we get git/GitLab to prevent this race condition?

Proposed Process (for a single file)

1. Source of truth for a binary file lives in a single-source, editable system with source control like SharePoint.
2. Developer makes desired changes to the binary file online in SharePoint, and runs a command that copies that version of the file into their local git repo branch, along with a UTC timestamp (or version number) written to a sidecar text file.
3. Developer modifies code to account for this version, and (per normal workflow) pushes the branch to GitLab remote and creates a merge request (MR).
4. GitLab CI/CD compares the timestamp in the merge with the timestamp on main, and fails if the merge would create an earlier timestamp.

The Race Condition

1. The SharePoint copy and main are using v5 of the file.
2. Developer A creates v6, snapshots it, and creates a merge request.
   • CI/CD sees that 6 >= 5 and the pipeline passes.
3. Developer B creates v7, snapshots it, and creates a merge request.
   • CI/CD sees that 7 >= 5 and the pipeline passes.
4. Developer B gets human review approval first, and merges v7 into main.
5. Developer A finally gets human review approval, and merges v6 into main…destroying v7. GitLab does not prevent this, because CI/CD only runs when the merge request is created or the source branch updated, not when the merge finally happens.

Considerations

• Our GitLab instance is self-hosted, but the company will not allow server-side hooks to run on it. We cannot use a pre-receive hook to prevent the second merge from succeeding.

• Changes to the source asset require paired code changes. Modifying the asset and pushing it to main without modifying other code will break the codebase.

• The process of developing the code for a change can be days long. Even if we had some process to set an exclusive lock, we cannot afford to prevent anyone from changing or working on the source file until one developer's MR is finally merged.

• We have looked deeply into removing binary files as source assets to the greatest extent possible, and breaking them down into the smallest reasonable pieces when they must be binary. Please assume the requirement stands that some assets must remain ~undiffable binary.

• If our CI/CD could look across all open MRs, it could fail whenever another MR is open that has a later version than the current MR. That would catch the case where developer A submitted their v6 MR after developer B's v7 MR was created (but not yet merged). Not sure if this would be possible?

  • If our CI/CD could affect other open MRs when it runs, then when developer B creates the v7 MR CI/CD could revoke the approval for the v6 MR. Is this possible?