Source Hydrator¶

Warning

Beta Feature (Since v3.5.0)

This is a beta-quality feature that pushes hydrated manifests to git before syncing them to the cluster.

Tools like Helm and Kustomize allow users to express their Kubernetes manifests in a more concise and reusable way (keeping it DRY - Don't Repeat Yourself). However, these tools can obscure the actual Kubernetes manifests that are applied to the cluster.

The rendered manifest pattern is a feature of Argo CD that allows users to push the hydrated manifests to git before syncing them to the cluster. This allows users to see the actual Kubernetes manifests that are applied to the cluster.

Enabling the Source Hydrator¶

The source hydrator is disabled by default.

To enable the source hydrator, you need to enable the "commit server" component and set the hydrator.enabled field in argocd-cmd-params-cm ConfigMap to "true".

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  hydrator.enabled: "true"

Important

After updating the ConfigMap, you must restart the Argo CD controller and API server for the changes to take effect.

If you are using one of the *-install.yaml manifests to install Argo CD, you can use the *-install-with-hydrator.yaml version of that file instead.

For example,

Without hydrator: https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
With hydrator:    https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install-with-hydrator.yaml

Important

The *-install-with-hydrator.yaml manifests will eventually be removed when the source hydrator is either enabled by default or removed. The upgrade guide will note if the install-with-hydrator.yaml manifests are no longer available.

Using the Source Hydrator¶

To use the source hydrator, you must first install a push and a pull secret. This example uses a GitHub App for authentication, but you can use any authentication method that Argo CD supports for repository access.

apiVersion: v1
kind: Secret
metadata:
  name: my-push-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository-write
type: Opaque
stringData:
  url: "https://github.com/<your org or user>/<your repo>"
  type: "git"
  githubAppID: "<your app ID here>"
  # githubAppInstallationID is optional and will be auto-discovered if omitted
  githubAppInstallationID: "<your installation ID here>" # Optional
  githubAppPrivateKey: |
    <your private key here>
---
apiVersion: v1
kind: Secret
metadata:
  name: my-pull-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  url: "https://github.com/<your org or user>/<your repo>"
  type: "git"
  githubAppID: "<your app ID here>"
  # githubAppInstallationID is optional and will be auto-discovered if omitted
  githubAppInstallationID: "<your installation ID here>"  # Optional
  githubAppPrivateKey: |
    <your private key here>

The only difference between the secrets above, besides the resource name, is that the push secret contains the label argocd.argoproj.io/secret-type: repository-write, which causes the Secret to be used for pushing manifests to git instead of pulling from Git. Argo CD requires different secrets for pushing and pulling to provide better isolation.

Once your secrets are installed, set the spec.sourceHydrator field of the Application. For example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      targetRevision: HEAD
    syncSource:
      targetBranch: environments/dev
      path: helm-guestbook

In this example, the hydrated manifests will be pushed to the environments/dev branch of the argocd-example-apps repository. The drySource field tells Argo CD where your original, unrendered configuration lives. This can be a Helm chart, a Kustomize directory, or plain manifests. Argo CD reads this source, renders the final Kubernetes manifests from it, and then writes those hydrated manifests into the location specified by syncSource.path.

Separate destination repository¶

By default, hydrated manifests are written to the same Git repository as the dry source. Set syncSource.repoURL to hydrate into a different repository. When syncSource.repoURL is omitted, it defaults to drySource.repoURL.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/my-org/config
      path: helm-guestbook
      targetRevision: HEAD
    syncSource:
      repoURL: https://github.com/my-org/deployments
      targetBranch: environments/dev
      path: helm-guestbook

Both repositories must be permitted in the application's AppProject (spec.sourceRepos). The hydrator needs read access to the dry source repository and write access to the destination repository (a repository-write secret; see the push secret example above).

Note

hydrateTo inherits its repository and path from syncSource. When syncSource.repoURL points to a separate repository, staged manifests are pushed to that repository as well.

When using source hydration, the syncSource.path field is required and must always point to a non-root directory in the repository. Setting the path to the repository root (for example "." or "") is not supported. This ensures that hydration is always scoped to a dedicated subdirectory, which avoids unintentionally overwriting or removing files that may exist in the repository root.

During each hydration run, Argo CD overwrites the files it generates (such as manifest.yaml) in the application's configured path, but it does not delete other files already present in that path. Stale files from a previous hydration that are not overwritten will remain in the output directory.

Because the generated manifest.yaml is fully rewritten on every run, resources that were removed from the dry source disappear from it and are pruned on the next sync (when the prune sync option is enabled); however, extra leftover files are not removed automatically.

The repository root is never written to, so files such as CI/CD configuration, README files, or other root-level assets remain untouched.

If an application’s path changes, the old directory is not removed automatically. Likewise, if an application is deleted, its output path remains in the repository and must be cleaned up manually by the repository owner if desired. This design is intentional: it prevents accidental deletion of files when applications are restructured or removed, and it protects critical files like CI pipelines that may coexist in the repository.

Note

The hydrator triggers only when a new commit is detected in the dry source. Adding or removing Applications does not on its own cause hydration to run. If the set of Applications changes but the dry-source commit does not, hydration will wait until the next commit.

This is by design: the hydrator produces one hydration per dry-source commit and maintains atomicity across the Applications that share a destination. If you add Applications that depend on a dry-source commit that has already been hydrated (for example, Applications generated by an ApplicationSet), push an empty commit to the dry source so the new Applications are hydrated.

Important

Project-Scoped Repositories

Repository Secrets may contain a project field, making the secret only usable by Applications in that project. The source hydrator only supports project-scoped repositories if all Applications writing to the same repository and branch are in the same project. If Applications in different projects write to the same repository and branch, the source hydrator will not be able to use a project-scoped repository secret and will require a global repository secret. This behavior may change in the future.

If there are multiple repository-write Secrets available for a repo, the source hydrator will non-deterministically select one of the matching Secrets and log a warning saying "Found multiple credentials for repoURL".

Source Configuration Options¶

The source hydrator supports various source types through inline configuration options in the drySource field. This allows you to use Helm charts, Kustomize applications, directories, and plugins with environment-specific configurations.

Helm Charts¶

You can use Helm charts by specifying the helm field in the drySource:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-helm-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      targetRevision: HEAD
      helm:
        valueFiles:
          - values-prod.yaml
        parameters:
          - name: image.tag
            value: v1.2.3
        releaseName: my-release
    syncSource:
      targetBranch: environments/prod
      path: helm-guestbook-hydrated

Kustomize Applications¶

For Kustomize applications, use the kustomize field:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-kustomize-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: kustomize-guestbook
      targetRevision: HEAD
      kustomize:
        namePrefix: prod-
        nameSuffix: -v1
        images:
          - gcr.io/heptio-images/ks-guestbook-demo:0.2
    syncSource:
      targetBranch: environments/prod
      path: kustomize-guestbook-hydrated

Directory Applications¶

For plain directory applications with specific options, use the directory field:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-directory-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: guestbook
      targetRevision: HEAD
      directory:
        recurse: true
    syncSource:
      targetBranch: environments/prod
      path: guestbook-hydrated

Config Management Plugins¶

You can also use Config Management Plugins by specifying the plugin field:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-plugin-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: my-plugin-app
      targetRevision: HEAD
      plugin:
        name: my-custom-plugin
        env:
          - name: ENV_VAR
            value: prod
    syncSource:
      targetBranch: environments/prod
      path: my-plugin-app-hydrated

Feature Parity

The source hydrator supports the same configuration options as the regular Application source field. You can use any combination of these source types with their respective configuration options to match your application's needs.

Pushing to a "Staging" Branch¶

The source hydrator can be used to push hydrated manifests to a "staging" branch instead of the syncSource branch. This provides a way to prevent the hydrated manifests from being applied to the cluster until some prerequisite conditions are met (in effect, providing a way to handle environment promotion via Pull Requests).

To use the source hydrator to push to a "staging" branch, set the spec.sourceHydrator.hydrateTo field of the Application. For example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  project: my-project
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      targetRevision: HEAD
    syncSource:
      targetBranch: environments/dev
      path: helm-guestbook
    hydrateTo:
      targetBranch: environments/dev-next

In this example, the hydrated manifests will be pushed to the environments/dev-next branch, and Argo CD will not sync the changes until something moves them to the environments/dev branch.

You could use a CI action to move the hydrated manifests from the hydrateTo branch to the syncSource branch. To introduce a gating mechanism, you could require a Pull Request to be opened to merge the changes from the hydrateTo branch to the syncSource branch.

Argo CD will only push changes to the hydrateTo branch, it will not create a PR or otherwise facilitate moving those changes to the syncSource branch. You will need to use your own tooling to move the changes from the hydrateTo branch to the syncSource branch.

Commit Tracing¶

It's common for CI or other tooling to push DRY manifest changes after a code change. It's important for users to be able to trace the hydrated commits back to the original code change that caused the hydration.

Source Hydrator makes use of some custom git commit trailers to facilitate this tracing. A CI job that builds an image and pushes an image bump to DRY manifests can use the following commit trailers to link the hydrated commit to the code commit.

git commit -m "Bump image to v1.2.3" \
  # Must be an RFC 5322 name
  --trailer "Argocd-reference-commit-author: Author Name <author@example.com>" \
  # Must be a hex string 5-40 characters long
  --trailer "Argocd-reference-commit-sha: <code-commit-sha>" \
  # The subject is the first line of the commit message. It cannot contain newlines.
  --trailer "Argocd-reference-commit-subject: Commit message of the code commit" \
   # The body must be a valid JSON string, including opening and closing quotes
  --trailer 'Argocd-reference-commit-body: "Commit message of the code commit\n\nSigned-off-by: Author Name <author@example.com>"' \
   # The repo URL must be a valid URL
  --trailer "Argocd-reference-commit-repourl: https://git.example.com/owner/repo" \
  # The date must by in ISO 8601 format
  --trailer "Argocd-reference-commit-date: 2025-06-09T13:50:18-04:00"

Note

The commit trailers must not contain newlines.

So the full CI script might look something like this:

# Clone code repo
git clone https://git.example.com/owner/repo.git
cd repo

# Build the image and get the new image tag
# <custom build logic here>

# Get the commit information
author=$(git show -s --format="%an <%ae>")
sha=$(git rev-parse HEAD)
subject=$(git show -s --format='%s')
body=$(git show -s --format='%b')
jsonbody=$(jq -n --arg body "$body" '$body')
repourl=$(git remote get-url origin)
date=$(git show -s --format='%aI')

# Clone the dry source repo
git clone https://git.example.com/owner/deployment-repo.git
cd deployment-repo

# Bump the image in the dry manifests
# <custom bump logic here, e.g. `kustomize edit`>

# Commit the changes with the commit trailers
git commit -m "Bump image to v1.2.3" \
  --trailer "Argocd-reference-commit-author: $author" \
  --trailer "Argocd-reference-commit-sha: $sha" \
  --trailer "Argocd-reference-commit-subject: $subject" \
  --trailer "Argocd-reference-commit-body: $jsonbody" \
  --trailer "Argocd-reference-commit-repourl: $repourl" \
  --trailer "Argocd-reference-commit-date: $date"

The commit metadata will appear in the hydrated commit's root hydrator.metadata file:

{
  "author": "CI <ci@example.com>",
  "subject": "chore: bump image to b82add2",
  "date": "2025-06-09T13:50:08-04:00",
  "body": "Signed-off-by: CI <ci@example.com>\n",
  "drySha": "6cb951525937865dced818bbdd78c89b2d2b3045",
  "repoURL": "https://git.example.com/owner/manifests-repo",
  "references": [
    {
      "commit": {
        "author": {
          "name": "Author Name",
          "email": "author@example.com"
        },
        "sha": "b82add298aa045d3672880802d5305c5a8aaa46e",
        "subject": "chore: make a change",
        "body": "make a change\n\nSigned-off-by: Author Name <author@example.com>",
        "repoURL": "https://git.example.com/owner/repo",
        "date": "2025-06-09T13:50:18-04:00"
      }
    }
  ]
}

The top-level "body" field contains the commit message of the DRY commit minus the subject line and any Argocd-reference-commit-* trailers that were used in references. Unrecognized or invalid trailers are preserved in the body.

Although references is an array, the source hydrator currently only supports a single related commit. If a trailer is specified more than once, the last one will be used.

All trailers are optional. If a trailer is not specified, the corresponding field in the metadata will be omitted.

Commit Message Template¶

The commit message is generated using a Go text/template, optionally configured by the user via the argocd-cm ConfigMap. The template is rendered using the values from hydrator.metadata. The template can be multi-line, allowing users to define a subject line, body and optional trailers. To define the commit message template, you need to set the sourceHydrator.commitMessageTemplate field in argocd-cm ConfigMap.

The template can invoke functions from the Sprig function library.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  sourceHydrator.commitMessageTemplate: |
    {{.metadata.drySha | trunc 7}}: {{ .metadata.subject }}
    {{- if .metadata.body }}

    {{ .metadata.body }}
    {{- end }}
    {{ range $ref := .metadata.references }}
    {{- if and $ref.commit $ref.commit.author }}
    Co-authored-by: {{ $ref.commit.author }}
    {{- end }}
    {{- end }}
    {{- if .metadata.author }}
    Co-authored-by: {{ .metadata.author }}
    {{- end }}

README Template¶

The hydration README is generated using a Go text/template, optionally configured by the user via the argocd-cm ConfigMap. The template is rendered using the values from hydrator.metadata and can be multi-line to define structured documentation. This allows users to customize how the hydration process and references are documented.

To define the README template, set the sourceHydrator.readmeMessageTemplate field in the argocd-cm ConfigMap.

The template may also use functions from the Sprig function library.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  sourceHydrator.readmeMessageTemplate: |
    # Manifest Hydration

    To hydrate the manifests in this repository, run the following commands:

    ```shell
    git clone {{ .RepoURL }}
    git checkout {{ .DrySHA }}
    {{ range $command := .Commands }}
    {{ $command }}
    {{ end }}
    ```

    {{ if .References }}
    ## References

    {{ range $ref := .References }}
    {{ if $ref.Commit }}
    * [{{ $ref.Commit.SHA | trunc 7 }}]({{ $ref.Commit.RepoURL }}): {{ $ref.Commit.Subject }} ({{ $ref.Commit.Author }})
    {{ end }}
    {{ end }}
    {{ end }}

Commit Author Configuration¶

You can customize the git commit author name and email used by the source hydrator when committing hydrated manifests. This is configured via the argocd-cm ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  commit.author.name: "GitOps Bot"
  commit.author.email: "gitops@company.com"

Configuration Keys: * commit.author.name: The git commit author name (defaults to "Argo CD" if not set) * commit.author.email: The git commit author email (defaults to "argo-cd@example.com" if not set)

Both values are optional. If only one is configured, the configured value will be used and the other will use its default.

Credential Templates¶

Credential templates allow a single credential to be used for multiple repositories. The source hydrator supports credential templates. For example, if you setup credential templates for the URL prefix https://github.com/argoproj, these credentials will be used for all repositories with this URL as prefix (e.g. https://github.com/argoproj/argocd-example-apps) that do not have their own credentials configured. For more information, please refer to Credential templates. An example of repo-write-creds secret.

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-write-creds
stringData:
  type: git
  url: https://github.com/argoproj
  password: my-password
  username: my-username

Git Note-based Hydration State Tracking¶

The Source Hydrator does not create a new hydrated commit for a DRY commit if the commit doesn't affect the hydrated manifests. Instead, the hydration state (the DRY SHA last hydrated) is tracked using a git note in a dedicated source-hydrator namespace.

On each run, the hydrator: * Checks the git note for the last hydrated DRY SHA. * If manifests have not changed since that SHA, only updates the note. * If manifests have changed, commits the new manifests and updates the note as well.

This improves efficiency and reduces commit noise in your repository.

Hydration failures and retries¶

When hydration fails, the application remains in the Failed phase and the error message is kept on status.sourceHydrator.currentOperation so you can diagnose the problem.

Note

After a failed hydration, the controller waits about 2 minutes before automatically retrying unattended hydration. That interval matches ARGOCD_RECONCILIATION_TIMEOUT (the application controller --app-resync period, default 120s).

You do not need to wait for that cooldown when:

You trigger a manual or API refresh (or a dry-source webhook), which retries hydration immediately.
A new dry commit is detected while a dry revision baseline is already recorded in status.sourceHydrator.lastComparedDryRevision (for example, hydration failed at the commit step after manifests were resolved).

If the first hydration attempt fails before any dry revision is recorded, automatic detection of a new commit during the cooldown may be delayed until the cooldown expires or you refresh manually.

Manifest Generate Paths¶

The source hydrator honors the manifest-generate-paths annotation to avoid unnecessary hydration. When the annotation is set, a new dry-source commit that does not touch the annotated paths (resolved relative to the drySource path) does not trigger re-hydration.

This applies to both webhook-driven and periodically-reconciled refreshes:

A webhook for the dry source triggers hydration only when the changed files match the annotated paths.
A webhook for the sync source (for example, when an external promotion process merges the hydrateTo branch into the syncSource branch) triggers a normal refresh and sync of the hydrated manifests.

The annotation's path filtering applies to the dry source. The sync source is always watched at its configured syncSource.path; the annotation value is not applied to the sync source.

Limitations¶

Signature Verification¶

Current Status: Alpha (Since v3.5)

The hydrator enforces the project's SourceIntegrity policy (e.g. GPG signature verification) on the DRY revision before producing manifests. If the project requires verification and the DRY commit fails it (or was not verified), hydration is rejected. Verification is opted into per project by configuring .spec.sourceIntegrity on the AppProject. See Source Integrity Verification for how to set it up (for example, using Git GnuPG verification).

The hydrator does not sign the commits it pushes to git, so if signature verification is enabled for the hydrated branch, those commits will fail verification when Argo CD attempts to sync the hydrated manifests.

Project-Scoped Push Secrets¶

If all the Applications for a given destination repo/branch are under the same project, then the hydrator will use any available project-scoped push secrets. If two Applications for a given repo/branch are in different projects, then the hydrator will not be able to use a project-scoped push secret and will require a global push secret.

Multiple Argo CD Instances Hydrating to One Branch¶

A hydrated repository and branch should be owned by a single Argo CD instance. Two separate Argo CD instances that share the same drySource and write hydrated manifests to the same syncSource repository and branch are not supported: the hydrator records hydration state in a per-dry-commit git note on the destination branch, so once one instance records a commit, the other instance will short-circuit and leave its manifests stale. Hydrate each instance to its own branch instead.

Prerequisites¶

Handle Secrets on the Destination Cluster¶

Do not use the source hydrator with any tool that injects secrets into your manifests as part of the hydration process (for example, Helm with SOPS or the Argo CD Vault Plugin). These secrets would be committed to git. Instead, use a secrets operator that populates the secret values on the destination cluster.

Best Practices¶

Make Hydration Deterministic¶

The source hydrator should be deterministic. For a given dry source commit, the hydrator should always produce the same hydrated manifests. This means that the hydrator should not rely on an external state or configuration that is not stored in Git.

Examples of non-deterministic hydration:

A Helm chart using unpinned dependencies
A Helm chart is using a non-deterministic template function such as randAlphaNum or lookup
Config Management Plugins which retrieve non-git state, such as secrets
Kustomize manifests referencing unpinned remote bases

Enable Branch Protection¶

Argo CD should be the only thing pushing hydrated manifests to the hydrated branches. To prevent other tools or users from pushing to the hydrated branches, enable branch protection in your SCM.

It is best practice to prefix the hydrated branches with a common prefix, such as environments/. This makes it easier to configure branch protection rules on the destination repository.

Note

To maintain reproducibility and determinism in the Hydrator’s output, Argo CD-specific metadata (such as argocd.argoproj.io/tracking-id) is not written to Git during hydration. These annotations are added dynamically during application sync and comparison.

Application Path Cleaning Behavior¶

The Source Hydrator does not clean (remove) files from the application's configured output path before writing new manifests. This means that any files previously generated by hydration (or otherwise present) that are not overwritten by the new hydration run will remain in the output directory.