Elastic search

Index Versioning

Overview

Indexes can be versioned. These versions will handle breaking data changes. To facilitate this the repositories will use an alias. This alias allows us to couple the most recent version of the version on an elastic level.

The mechanism used for this is called reindex .

Backup the current index

To kick of our changes, we will first reindex the existing index. We will add a version number on the end. In this example we will call this version 1.

To reindex use the following request:

POST _reindex
{
  "source": {
    "index": "my-index-000001"
  },
  "dest": {
    "index": "my-new-index-000001"
  }
}

Additionally we want to remove fields we don’t want in our new index. This will allow us to create a clean index when needed. To exclude fields we can use the following syntax. By using the script we can remove fields we don’t want in the new index

POST /_reindex
{
  "source": {
    "index": "my-index-000001"
  },
  "dest": {
    "index": "my-index-000001"
  },
  "conflicts": "proceed",
  "script" : {
    "source": "ctx._source.data.remove('properties.review_meta')"
  }

}

Create an alias

We will create an alias. This alias can be used in the application. We can change this alias when we need changed the index underneath. This will prevent the need to update the application on a data change.

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "my-data-stream",
        "alias": "my-alias"
      }
    }
  ]
}

Github

Github provides a lot of functionality out of the box for a small fee. It is true that choosing other providers might offer additional functionality, but this mostly comes at a price. As an enterprise the decision really depends on the added value of the functionality. This mostly boils down to reporting, visualization and ease of use. In beginning projects I would advocate to use Github and the functionality it offers to the fullest.

Github Actions

Github actions allow us to automate a lot of things, that would otherwise require more expensive tools. The trade of is, that it is not a point and click interface. The advantage is, that it is widely used and therefor a lot of documentation can be found online. Github actions allow us to build and deploy our application automatically after changes are pushed to a branch.

NuGet Registry

To automatically create a NuGet package from code, all we need to do is create a file .github/forkflows/release.yaml from the root of our project.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

name: Build Nuget Packages
on:
  workflow_dispatch: # Allow running the workflow manually from the GitHub UI
  push:
    branches:
      - 'main'

env:
  NuGetDirectory: ${{ github.workspace}}/nuget

jobs:
  create_nuget:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0 # Get all history to allow automatic versioning using MinVer
      
      # Install the .NET SDK indicated in the global.json file
      - name: Setup .NET
        uses: actions/setup-dotnet@v3
        with:
          dotnet-version: 7.0.x #${{ matrix.dotnet-version }}

      - name: Update path
        run: export PATH="/usr/share/dotnet:$PATH"

      - name: Install dependencies
        run: dotnet restore

      - name: Build
        run: dotnet build --configuration Release --no-restore
      
      #      - name: Test
      #        run: dotnet test --no-restore --verbosity normal
      
      
      # Create the NuGet package in the folder from the environment variable NuGetDirectory
      - run: dotnet pack --configuration Release --output ${{ env.NuGetDirectory }}
      
      # Publish the NuGet package as an artifact, so they can be used in the following jobs
      - uses: actions/upload-artifact@v3
        with:
          name: nuget
          if-no-files-found: error
          retention-days: 7
          path: ${{ env.NuGetDirectory }}/*.nupkg

      - name: Display generated packages
        run: ls -la ${{ env.NuGetDirectory }}/*.nupkg
  

  run_test:
    runs-on: ubuntu-latest
    needs: [ create_nuget ]
    steps:
      - uses: actions/checkout@v3
      - name: Setup .NET
        uses: actions/setup-dotnet@v3
        with:
          dotnet-version: 7.0.x #${{ matrix.dotnet-version }}

      - name: Update path
        run: export PATH="/usr/share/dotnet:$PATH"
      - name: Run tests
        run: dotnet test --configuration Release

  deploy:
    # Publish only when creating a GitHub Release
    # https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository
    # You can update this logic if you want to manage releases differently
    #if: github.event_name == 'release'
    runs-on: ubuntu-latest
    needs: [ run_test ]
    steps:
      # Download the NuGet package created in the previous job
      - uses: actions/download-artifact@v3
        with:
          name: nuget
          path: ${{ env.NuGetDirectory }}

      # Install the .NET SDK indicated in the global.json file
      - name: Setup .NET Core
        uses: actions/setup-dotnet@v3
        with:
          dotnet-version: 7.0.x #${{ matrix.dotnet-version }}

      - name: Update path
        run: export PATH="/usr/share/dotnet:$PATH"

      # Publish all NuGet packages to NuGet.org
      # Use --skip-duplicate to prevent errors if a package with the same version already exists.
      # If you retry a failed workflow, already published packages will be skipped without error.
      - name: Publish NuGet package
        shell: pwsh
        run: |
          foreach($file in (Get-ChildItem "${{ env.NuGetDirectory }}" -Recurse -Include *.nupkg)) {
              dotnet nuget push $file --api-key "${{ secrets.NUGET_STORE_API_KEY }}" --source "${{ secrets.NUGET_STORE }}" --skip-duplicate
          }          

Explaining the code:

• line 5: Script will run when changes are pushed to the main branch
• line 9: Sets the nuget directory for the build, this will not influence your build.
• line 23: This example depends on dotnet 7.0.x, where x is a wildcard to get the most recent version.
• line 32: Runs the build with the release option enabled.
• line 99: The option --skip-duplicate is used to prevent failure when the version already exists. Remove this if you want the build to fail on duplicate versions.

Docker container registry

We can create docker containers and deploy them to the Github container registry easily.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136

name: Release Workflow

on:
  push:
    branches:
      - 'release/*.*.*'
      - 'main'
      - 'develop'

env:
  ASPNETCORE_ENVIRONMENT: Development
  REGISTRY: "ghcr.io/microservice-world"
  IMAGE_NAME: "my-api"
  REGISTRY_USER: my-github-username
jobs:
  version:
    name: Determine version
    runs-on: ubuntu-latest
    outputs:
      branchName:            ${{ steps.version_step.outputs.branchName }}
      fullSemVer:            ${{ steps.version_step.outputs.fullSemVer }}
  
      GitVersion_BranchName: ${{ steps.version_step.outputs.GitVersion_BranchName }}
      GitVersion_SemVer: ${{ steps.version_step.outputs.GitVersion_SemVer }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install GitVersion
        uses: gittools/actions/gitversion/setup@v1.1.1
        with:
          versionSpec: '5.x'

      - name: Determine Version
        id: version_step # step id used as reference for output values
        uses: gittools/actions/gitversion/execute@v1.1.1
          
      - name: Display GitVersion variables (without prefix)
        run: |
             echo "SemVer (env.semVer)            : ${{ steps.version_step.outputs.semVer }}"             
    
    
  build:
    name: build
    needs: version
    runs-on: ubuntu-latest
    outputs:
      DockerImage: "${{ env.IMAGE_NAME}}:${{ needs.version.outputs.GitVersion_SemVer }}"
      DockerImageUrl: "https://github.com/${{ github.repository }}/pkgs/container/${{ env.IMAGE_NAME}}"
    steps:
      - name: Checkout the repo
        uses: actions/checkout@v3

      - name: Build container image
        run:  |
          docker build  \
              --no-cache \
              -f Dockerfile \
              -t $(echo $REGISTRY)/$(echo $IMAGE_NAME):${{ needs.version.outputs.GitVersion_SemVer }} \
              -t $(echo $REGISTRY)/$(echo $IMAGE_NAME):latest \
              --label "org.opencontainers.image.source=https://github.com/$(echo $REGISTRY)/$(echo $IMAGE_NAME)" \
              .          
      - name: Login GITHUB container store
        run:  docker login ghcr.io -u $(echo $REGISTRY_USER) --password ${{ secrets.GITHUB_TOKEN }}

      - name: Push image to Github Container Registry
        run:  docker push $(echo $REGISTRY)/$(echo $IMAGE_NAME):${{ needs.version.outputs.GitVersion_SemVer }}

  
  release:
    name: release
    needs: [version, build]
    runs-on: ubuntu-latest
    permissions:
      contents: write
      packages: write
    env:
      myvar_fullSemVer: ${{ needs.version.outputs.SemVer }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0


      - name: Setup dotnet
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: |
            8.0.x            
          
      - name: Create Change log
        id: changelog
        uses: TriPSs/conventional-changelog-action@v5.1.0
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          output-file: "false"
          preset: "conventionalcommits"
          skip-git-pull: true
          git-pull-method: --force
          skip-commit: true
          pre-release: true
          skip-version-file: true
          skip-tag: true
#          fallback-version: v${{ steps.version_step.outputs.semVer }} 
          fallback-version: v${{ needs.version.outputs.GitVersion_SemVer }}
          release-count: 0
          skip-on-empty: false
          tag-prefix: "v"
      #          create-summary: true
      #          skip-bump: true

      - name: Create tag
        uses: actions/github-script@v5
        continue-on-error: true
        with:
          script: |
                  github.rest.git.createRef({
                    owner: context.repo.owner,
                    repo: context.repo.repo,
                    ref: 'refs/tags/${{ needs.version.outputs.GitVersion_SemVer }}',
                    sha: context.sha
                  }) || true
                                    
      - name: Create GitHub Release
        uses: ncipollo/release-action@v1
        with:
          skipIfReleaseExists: true
          name: v${{ needs.version.outputs.GitVersion_SemVer }}
          tag: v${{ needs.version.outputs.GitVersion_SemVer }}
          body: "${{ steps.changelog.outputs.clean_changelog }} \r\n \r\n ### Related docker images \r\n * [${{ needs.build.outputs.DockerImage }}](${{ needs.build.outputs.DockerImageUrl }})"
          makeLatest: true
          

Explaining the code

• line 6 -> 7: sets up release management based on branching
• line 10 -> 14: Environment variables needed for the build

Jobs

1. Version: determines the version number using GitVersion
2. Build: Builds the docker container using the Dockerfile and pushes the image to the container registry
3. Release: Creates a release using Semantic Versioning

You will be able to see the result in your repository

Gita

Gita is a wonderful tool that helps in managing multiple repositories as one. If you find yourself doing a lot of git actions through the command line. Check out this awesome tool.

Gita: a command-line tool to manage multiple git repos

This tool has two main features

• display the status of multiple git repos such as branch, modification, commit message side by side
• (batch) delegate git commands/aliases and shell commands on repos from any working directory

In this screenshot, the gita ll command displays the status of all repos. The gita remote dotfiles command translates to git remote -v for the dotfiles repo, even though we are not in the repo. The gita fetch command fetches from all repos and two of them have updates. To see the pre-defined commands, run gita -h or take a look at cmds.json. To add your own commands, see the customization section. To run arbitrary git command, see the superman mode section. To run arbitrary shell command, see the shell mode section.

I also made a youtube video to demonstrate the common usages

The branch color distinguishes 5 situations between local and remote branches:

The choice of purple for ahead and yellow for behind is motivated by blueshift and redshift, using green as baseline. You can change the color scheme using the gita color command. See the customization section.

The additional status symbols denote

The bookkeeping sub-commands are

• gita add <repo-path(s)> [-g <groupname>]: add repo(s) to gita, optionally into an existing group
• gita add -a <repo-parent-path(s)>: add repo(s) in <repo-parent-path(s)> recursively and automatically generate hierarchical groups. See the customization section for more details.
• gita add -b <bare-repo-path(s)>: add bare repo(s) to gita. See the customization section for more details on setting custom worktree.
• gita add -r <repo-parent-path(s)>: add repo(s) in <repo-parent-path(s)> recursively
• gita clear: remove all groups and repos
• gita clone <URL>: clone repo from URL at current working directory
• gita clone <URL> -C <directory>: change to directory and then clone repo
• gita clone -f <config-file>: clone repos in config-file (generated by gita freeze) to current directory.
• gita clone -p -f <config-file>: clone repos in config-file to prescribed paths.
• gita context: context sub-command
  • gita context: show current context
  • gita context <group-name>: set context to group-name, all operations then only apply to repos in this group
  • gita context auto: set context automatically according to the current working directory
  • gita context none: remove context
• gita color: color sub-command
  • gita color [ll]: Show available colors and the current coloring scheme
  • gita color reset: Reset to the default coloring scheme
  • gita color set <situation> <color>: Use the specified color for the local-remote situation
• gita flags: flags sub-command
  • gita flags set <repo-name> <flags>: add custom flags to repo
  • gita flags [ll]: display repos with custom flags
• gita freeze: print information of all repos such as URL, name, and path. Use with gita clone.
• gita group: group sub-command
  • gita group add <repo-name(s)> -n <group-name>: add repo(s) to a new or existing group
  • gita group [ll]: display existing groups with repos
  • gita group ls: display existing group names
  • gita group rename <group-name> <new-name>: change group name
  • gita group rm <group-name(s)>: delete group(s)
  • gita group rmrepo <repo-name(s)> -n <group-name>: remove repo(s) from existing group
• gita info: info sub-command
  • gita info [ll]: display the used and unused information items
  • gita info add <info-item>: enable information item
  • gita info rm <info-item>: disable information item
• gita ll: display the status of all repos
• gita ll <group-name>: display the status of repos in a group
• gita ll -g: display the repo summaries by groups
• gita ls: display the names of all repos
• gita ls <repo-name>: display the absolute path of one repo
• gita rename <repo-name> <new-name>: rename a repo
• gita rm <repo-name(s)>: remove repo(s) from gita (won’t remove files on disk)
• gita -v: display gita version

The git delegating sub-commands are of two formats

• gita <sub-command> [repo-name(s) or group-name(s)]: optional repo or group input, and no input means all repos.
• gita <sub-command> <repo-name(s) or groups-name(s)>: required repo name(s) or group name(s) input

They translate to git <sub-command> for the corresponding repos. By default, only fetch and pull take optional input. In other words, gita fetch and gita pull apply to all repos. To see the pre-defined sub-commands, run gita -h or take a look at cmds.json. To add your own sub-commands or override the default behaviors, see the customization section. To run arbitrary git command, see the superman mode section.

If more than one repos are specified, the git command runs asynchronously, with the exception of log, difftool and mergetool, which require non-trivial user input.

Repo configuration global is saved in $XDG_CONFIG_HOME/gita/repos.csv (most likely ~/.config/gita/repos.csv) or if you prefered at project configuration add environment variable GITA_PROJECT_HOME.

Installation

To install the latest version, run

pip3 install -U gita

If you prefer development mode, download the source code and run

pip3 install -e <gita-source-folder>

In either case, calling gita in terminal may not work, then put the following line in the .bashrc file.

alias gita="python3 -m gita"

Windows users may need to enable the ANSI escape sequence in terminal for the branch color to work. See this stackoverflow post for details.

Auto-completion

You can download the generated auto-completion file in the following locations for your specific shell. Alternatively, if you have installed argcomplete on your system, you can also directly run eval "$(register-python-argcomplete gita -s SHELL)" (e.g. SHELL as bash/zsh) in your dotfile.

Bash

Download .gita-completion.bash and source it in shell.

Zsh

There are 2 options :

• .gita-completion.zsh. Use the help of gita command to display options. It uses the bash completion system for zsh. Add autoload -U +X bashcompinit && bashcompinit in .zshrc and source the zsh file
• _gita. Completion more Zsh style. Copy it in a folder and add this folder path in FPATH variable. This completion file doesn’t take account to command from cmds.json

Fish

Download gita.fish and place it in ~/.config/fish/completions/

Superman mode

The superman mode delegates any git command or alias. Usage:

gita super [repo-name(s) or group-name(s)] <any-git-command-with-or-without-options>

Here repo-name(s) or group-name(s) are optional, and their absence means all repos. For example,

• gita super checkout master puts all repos on the master branch
• gita super frontend-repo backend-repo commit -am 'implement a new feature' executes git commit -am 'implement a new feature' for frontend-repo and backend-repo

Shell mode

The shell mode delegates any shell command. Usage:

gita shell [repo-name(s) or group-name(s)] <any-shell-command>

Here repo-name(s) or group-name(s) are optional, and their absence means all repos. For example,

• gita shell ll lists contents for all repos
• gita shell repo1 repo2 mkdir docs create a new directory docs in repo1 and repo2
• gita shell "git describe --abbrev=0 --tags | xargs git checkout": check out the latest tag for all repos

Customization

define repo group and context

When the project contains several independent but related repos, we can define a group and execute gita command on this group. For example,

gita group add repo1 repo2 -n my-group
gita ll my-group
gita pull my-group

To save more typing, one can set a group as context, then any gita command is scoped to the group

gita context my-group
gita ll
gita pull

The most useful context maybe auto. In this mode, the context is automatically determined from the current working directory (CWD): the context is the group whose member repo’s path contains CWD. To set it, run

gita context auto

To remove the context, run

gita context none

It is also possible to recursively add repos within a directory and generate hierarchical groups automatically. For example, running

gita add -a src

on the following folder structure

src
├── project1
│   ├── repo1
│   └── repo2
├── repo3
├── project2
│   ├── repo4
│   └── repo5
└── repo6

gives rise to 3 groups:

src:repo1,repo2,repo3,repo4,repo5,repo6
src-project1:repo1,repo2
src-project2:repo4,repo5

add user-defined sub-command using json file

Custom delegating sub-commands can be defined in $XDG_CONFIG_HOME/gita/cmds.json (most likely ~/.config/gita/cmds.json) And they shadow the default ones if name collisions exist.

Default delegating sub-commands are defined in cmds.json. For example, gita stat <repo-name(s)> is registered as

1
2
3
4

"stat":{
  "cmd": "git diff --stat",
  "help": "show edit statistics"
}

which executes git diff --stat for the specified repo(s).

To disable asynchronous execution, set disable_async to be true. See the difftool example:

1
2
3
4
5

"difftool":{
  "cmd": "git difftool",
  "disable_async": true,
  "help": "show differences using a tool"
}

If you want a custom command to behave like gita fetch, i.e., to apply to all repos when no repo is specified, set allow_all to be true. For example, the following snippet creates a new command gita comaster [repo-name(s)] with optional repo name input.

1
2
3
4
5

"comaster":{
  "cmd": "checkout master",
  "allow_all": true,
  "help": "checkout the master branch"
}

Any command that runs in the superman mode mode or the shell mode can be defined in this json format. For example, the following command runs in shell mode and fetches only the current branch from upstream.

1
2
3
4
5
6

"fetchcrt":{
  "cmd": "git rev-parse --abbrev-ref HEAD | xargs git fetch --prune upstream",
  "allow_all": true,
  "shell": true,
  "help": "fetch current branch only"
}

customize the local/remote relationship coloring displayed by the gita ll command

You can see the default color scheme and the available colors via gita color. To change the color coding, use gita color set <situation> <color>. The configuration is saved in $XDG_CONFIG_HOME/gita/color.csv.

customize information displayed by the gita ll command

You can customize the information displayed by gita ll. The used and unused information items are shown with gita info, and the configuration is saved in $XDG_CONFIG_HOME/gita/info.csv.

For example, the default setting corresponds to

branch,commit_msg,commit_time

Here branch includes both branch name and status. The status symbols are similar to the ones used in spaceship-prompt.

To customize these symbols, add a file in $XDG_CONFIG_HOME/gita/symbols.csv. The default settings corresponds to

dirty,staged,untracked,local_ahead,remote_ahead,diverged,in_sync,no_remote
*,+,?,↑,↓,⇕,,∅

Only the symbols to be overridden need to be defined. You can search unicode symbols here.

customize git command flags

One can set custom flags to run git commands. For example, with

gita flags set my-repo --git-dir=`gita ls dotfiles` --work-tree=$HOME

any git command/alias triggered from gita on dotfiles will use these flags. Note that the flags are applied immediately after git. For example, gita st dotfiles translates to

git --git-dir=$HOME/somefolder --work-tree=$HOME status

running from the dotfiles directory.

This feature was originally added to deal with bare repo dotfiles.

Requirements

Gita requires Python 3.6 or higher, due to the use of f-string and asyncio module.

Under the hood, gita uses subprocess to run git commands/aliases. Thus the installed git version may matter. I have git 1.8.3.1, 2.17.2, and 2.20.1 on my machines, and their results agree.

Tips

Contributing

To contribute, you can

• report/fix bugs
• request/implement features
• star/recommend this project

Read this article if you have never contribute code to open source project before.

Chat room is available on

To run tests locally, simply pytest in the source code folder. Note that context should be set as none. More implementation details are in design.md. A step-by-step guide to reproduce this project is here.

You can also sponsor me on GitHub. Any amount is appreciated!

Other multi-repo tools

I haven’t tried them but I heard good things about them.

• myrepos
• repo
• mu-repo