How to include submodules in template?

[jvacek]

Describe the problem

I'd like to have a submodule checked out inside of my template but I don't find an easy way to do it.

It would be nice if this could be documented.

I suspect the below doesn't work because the changes are being made in the temp folder that is not the actual folder with .git in it

How would you suggest working around this?

Template

# ...
_tasks:
  - command: |
      unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE
      PROJECT_ROOT=$(pwd)
      if [ ! -f "$PROJECT_ROOT/.gitmodules" ] || ! grep -q "my_submodule" "$PROJECT_ROOT/.gitmodules"; then
        git -C "$PROJECT_ROOT" submodule add --force git@git.internal.com:my_submodule.git .my_submodule
      fi
      if git -C "$PROJECT_ROOT" submodule status | grep -q -e "^-"; then
        git -C "$PROJECT_ROOT" submodule update --init --recursive
      fi
      git -C "$PROJECT_ROOT/.my_submodule" checkout tags/v1.0.10
# ...

To Reproduce

No response

Logs

#...
Reactivating local git directory for submodule '.my_submodule'
HEAD is now at 517672c Merge branch 'bump-ruff-0157' into 'master'
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: cannot change to '/private/var/folders/n_/1mvxkf8n6b5_vqjw6_1_y81r0000gp/T/copier._main.new_copy.zrqestmy/.my_submodule': No such file or directory
Task '<snipped command above>' returned non-zero exit status 128.

Expected behavior

It would be amazing if I could just specify the submodule and the tag to check out in the yaml, otherwise a way to run commands in the real dir would be nice.

Screenshots/screencasts/logs

No response

Operating system

macOS

Operating system distribution and version

macos 26.4.1

Copier version

copier 9.14.3

Python version

cpy 3.13

Installation method

uvx+pypi

Additional context

No response

[jvacek]

Also worth noting that I'm trying to introduce this submodule in an update

[sisp]

Is it an option for you to track the submodule in your template directly instead of creating it in the target project via a task?

Also worth noting that I'm trying to introduce this submodule in an update

In this case, a post-update migration might be better suited which does run in the target project's directory:

_migrations:
  - command: ... # your submodule commands
    version: v1.2.3 # your template version that shall introduce the submodule in the target project
    when: "{{ _stage == 'after' }}" # run the migration only *after* the update

[jvacek]

I'm not sure if I'm missing something but if I add an actual submodule to the template (i.e. git submodule add my_submodule template/my_suybmodule) then it will just copy the files instead of treating it as a git submodule. The .gitmodules doesn't make it across as it's outside of the template.

[sisp]

It seems you're right. I've never worked with submodules, but since you brought them up I did some research. It seems there are some obstacles:

• The .gitmodules file is located in the top-level directory of a Git working tree. If a template uses the _subdirectory setting in copier.yml, it will be located in a subdirectory, so the Copier template's Git repository won't recognize this .gitmodules file. If the _subdirectory setting isn't used while the Copier template uses a submodule itself (not intended to be included in the generated project), there's a clash. (I generally recommend to use a subdirectory for template content, but it isn't the default or even required for historical reasons.)

• Git records the submodule's commit SHA in the index via a gitlink entry. Even if the template repository held this information, it would be lost when Copier generates a project, especially because the destination may not be a Git repository.

Nevertheless, I think it's a valid ask to support generating projects with submodules if the destination is a Git repository. Here is an untested idea how we might be able to support submodules: A Copier template could contain a .gitmodules.jinja file with the appropriate content – it could even contain Jinja syntax. A simple text file at the submodule's target path, conceptually equivalent to the submodule link shown on GitHub etc., could contain the submodule's commit SHA. For example:

• File tree:

  .
  ├── copier.yml
  └── template
      ├── .gitmodules.jinja
      └── path
          └── to
              └── my-submodule
  

• copier.yml:

  _subdirectory: template/

• template/.gitmodules.jinja:

  [submodule "my-submodule"]
          path = path/to/my-submodule
          url = https://git.example.com/other/repo.git
  

• template/path/to/my-submodule:

  5d3fff119c14317979140e280c6b7acc321b6f5e
  

Then, Copier could handle Git submodules like this:

1. When rendering a file whose destination path is .gitmodules in the top-level directory of a Git working tree, parse the file and extract and memorize the submodule paths.

2. After all files have been generated, extract the commit SHAs from the text files at the submodule paths.

3. Update the Git index of the destination repository by adding gitlink entries:

   git update-index --add --cacheinfo 160000 <SHA> path/to/submodule

4. Delete the text files at the submodule paths.

A Copier template author would be responsible for ensuring/checking that the destination is a Git repository, else copier copy would fail when trying to run the git update-index ... command.

WDYT?

[jvacek]

I'm curious if there would be a way to standardise this to some degree in the copier.yml as this is quite a lot of "boilerplate" for something that I believe will mostly be a similar flow with very few variables (url, path, ref).

[sisp]

Moving submodule configuration to copier.yml feels like a conceptual break to me because .gitmodules wouldn't be part of the template filetree, and the submodule revision would be unstable because the gitlink entry would be missing (unless we added this information to the copier.yml submodule config). Making submodules part of the template filetree as similar as possible to regular submodule use seems more intuitive IMHO.

[jvacek]

Fair enough, that makes sense! If the docs are good, storing this in the template/ could work indeed.