Data sources
A data source is a company-level description of a database — its network address, its connection parameters and the accounts DataForge signs in with — configured once by an administrator and reused by the database connections of every project. Data sources are managed in Settings → Integrations, on the Data sources tab.
Purpose and scope
Before data sources existed, every project connection carried its own address and its own credentials, and the same database had to be described again in every project version. A data source moves that description one level up:
- The address, the port, the SSL settings and the service account are stored once for the whole company and are reused by every connection that points at this database
- A project connection no longer asks for a host, a port or a password — it picks a data source and then selects the database and schema it works with
- The data source decides whose database account a query runs under: one shared service account for everybody, or each user's own credentials, so that the results a person sees respect the permissions they have in the database itself
- All configuration changes are recorded in the System Journal
Data sources are supported for PostgreSQL, Microsoft SQL Server and ClickHouse — the same database types as project connections.
Access permissions
| Role | Manage data sources (view, create, edit, test, activate, delete, map users) | Use a data source in a project connection |
|---|---|---|
| Super administrator | Yes (own company's data sources) | Yes |
| Administrator | Yes (own company's data sources) | Yes |
| Project manager, Developer | No | Yes |
| Analyst, Viewer | No | No — these roles cannot create or edit connections |
The Integrations page is available only to administrators. A Super administrator, like an Administrator, sees and uses only the data sources of their own company — both on this page and in the data source list of the connection dialog. Users with other roles never open the page, but they select the configured data sources when they create database connections in their projects.
Data sources list
The Data sources tab shows the company's data sources as a table:
| Column | Description |
|---|---|
| Data source | Name set by the administrator |
| Database type | PostgreSQL, Microsoft SQL Server or ClickHouse |
| Database host | Address of the database server |
| Database port | Port of the database server |
| Authentication mode | Shared account or Individual credentials |
| Credential source | How individual credentials are obtained; shown as "—" for a shared account |
| Status | Active, Check failed or Disabled |
| Actions | Edit, data source users, disable or activate, delete |
Above the table there is an Add data source button. Database type, Authentication mode, Credential source and Status are filtered by picking a value from a list, the other columns by substring search; the list is sorted and paginated.
Data source status
| Status | Meaning |
|---|---|
| Active | The data source is in use and its last connection test did not fail |
| Check failed | The last connection test failed |
| Disabled | The data source has been taken out of use by an administrator |
Adding a data source
- Click Add data source above the list
- Fill in the name, the database type and the authentication mode at the top of the dialog
- Fill in the Parameters tab — the address of the database and the service account
- If the mode is Individual credentials, open the Credentials tab and describe where each user's credentials come from
- For ClickHouse, open the SSL tab if the server requires TLS
- Click Verify to check the values currently in the form
- Click Save
General settings
These fields sit above the tabs and apply to the whole data source:
| Field | Required | Description |
|---|---|---|
| Data source name | Yes | Name shown in the list and in the connection dialog. Must be unique within the company regardless of letter case: names that differ only in case, such as "Sales" and "sales", count as the same name. A taken name is reported under the field as it is typed and when saving is refused — "A data source with this name already exists" — and Save stays unavailable until the name is changed |
| Data source type | Yes | PostgreSQL, Microsoft SQL Server or ClickHouse. Cannot be changed after the data source has been saved — connections and their stored database structure are built for that engine |
| Authentication mode | Yes | Shared account — every query runs under the service account of this data source; Individual credentials — a query a user asks for runs under that user's own database account |
Parameters tab
| Field | Required | Description |
|---|---|---|
| Database host | Yes | Domain name or IP address of the database server, without a scheme, port or path |
| Database port | Yes | Port of the database server |
| Default database | No | Database offered first when a connection is created from this data source |
| Default schema | No | Schema offered first. Shown for PostgreSQL only; public by default |
| Database service user | No | Account DataForge uses for its own operations, and for everybody when the mode is Shared account |
| Service user password | Yes | Password of that account. It is encrypted before it leaves the browser and is stored encrypted |
| Transfer access rights to created data mart views | — | Whether the access rights of the source tables are carried over to the objects created in the database when data mart views are deployed. Cleared by default |
The service account needs the rights to read the metadata of the database and the source tables. It is always kept, whichever authentication mode is chosen: DataForge uses it for the operations that must not depend on one person's rights.
Credentials tab
The tab is available only when the mode is Individual credentials. It starts with the Credential source — where a user's database login and password come from:
| Credential source | How credentials are obtained |
|---|---|
| Manual configuration | An administrator stores a database login and password for each user on the Data source users screen |
| Credential vault | The login and password are read from an external secret vault by a path built for the requesting user |
| LDAP bind | The corporate password the user entered when signing in to DataForge through an LDAP / AD identity provider is passed on to the database |
Fields of the Credential vault source:
| Field | Required | Description |
|---|---|---|
| Vault URL | Yes | Address of the secret vault |
| Authentication method | Yes | Token (default) or AppRole |
| Token / Role ID:Secret ID | Yes | The secret DataForge authenticates in the vault with. Stored encrypted and never shown again |
| Path template | Yes | Path the user's secret is read from. Must contain the {userEmail} or {userId} placeholder, so that every user gets their own secret |
| Username field | No | Name of the field holding the database login. username by default |
| Password field | No | Name of the field holding the password. password by default |
Fields of the LDAP bind source:
| Field | Required | Description |
|---|---|---|
| LDAP server | Yes | Host name or IP address of the directory server |
| Port | Yes | Directory server port. 636 by default |
| Enable SSL (LDAPS) | — | Connect to the directory over LDAPS. Selected by default |
| Base DN | Yes | Branch of the directory the user entries are searched in |
| Search attribute | No | Attribute the user is matched by. uid by default |
| Domain | No | Windows domain. Shown for Microsoft SQL Server only — with a domain set, the database accepts the directory identity as Windows authentication |
For the Manual configuration source the tab also shows the users already mapped to this data source and a button that opens the Data source users screen. The button becomes available after the data source has been saved — there is nothing to map credentials to before that, and the dialog says so.
At the bottom of the tab, for every credential source:
| Field | Description |
|---|---|
| Use the service account when individual credentials are missing | With the flag set, a user who has no credentials of their own is served by the service account of the data source, and a warning tells them that their individual rights are not applied. With the flag cleared, such a user is refused |
SSL tab
The tab is available for ClickHouse data sources only. The certificate fields become active after Use SSL is selected.
| Field | Description |
|---|---|
| Use SSL | Turns on TLS and activates the certificate fields |
| Certificate authority (CA) certificate | Root certificate of the certification authority |
| Client certificate | Client certificate for mutual authentication |
| Client private key | Private key matching the client certificate |
Each field has an Upload button and a Delete button. Allowed extensions are .crt, .pem, .cer, .ca-bundle and .bundle. The browser rejects a file larger than 1 MB with the message "Maximum file size: 1 megabyte", and the server accepts a file of at most 64 KB — a certificate normally fits well within that. Uploaded certificates are stored encrypted; only their file names stay visible in the form.
Testing the connection
The Verify button in the creation or editing dialog checks the values currently in the form. The check runs as a sequence of steps, and a failure names the step that failed:
- reaching the server — the database server answers at the given address and port
- service account sign-in — the service user and password are accepted
- database and schema — the default database and, for PostgreSQL, the default schema exist
A successful check is reported with the notification "Data source connection successful". A failure is reported as "Could not connect to the data source" followed by the failed step and the reason the database returned. The check made from the dialog does not change the Status of the data source in the list.
When editing, an untouched Service user password field is checked with the stored password, so a data source can be verified by an administrator who does not know it.
Individual credentials of users
For a data source with the Manual configuration credential source, the database login and password of each user are stored on the Data source users screen. Open it with the data source users button in the list row, or from the Credentials tab of the editing dialog.
The screen lists the users that already have credentials for this data source: First name, Last name, Email address and Data source username. Passwords are never shown — they can only be replaced.
To map several users at once:
- Click Add users
- In each row pick the User, then enter their Data source username and Data source password
- Use Add user to add another row and the delete button to drop one
- Check a row against the database with the check button next to it — a wrong login or password is much cheaper to find here than in a failed preview later
- Click Save
Only company users who do not have a mapping for this data source yet are offered in the list, and one user may have one mapping per data source.
To change a mapping, use the edit button in its row. The user's name and email address are shown read-only; the Data source username can be changed, and the Data source password only by clicking Edit next to it and typing a new one. Verify checks the mapping against the database — with the password left untouched, the stored one is used.
The delete button removes a mapping through a confirmation dialog. After that the user has no credentials of their own for this data source and is served according to the fallback setting.
Using a data source in a project connection
Database connections of a project no longer carry an address or credentials of their own. In the connection dialog the user:
- selects a Data source — its Database host and Database port are shown read-only, so it is clear where the connection points
- selects the Database, and for PostgreSQL the Schema, from the lists the data source loads from the server
Everything else about connections — verifying them, refreshing the cached database structure, activation and deletion — works as described on the Managing database connections page.
Which account a query runs under
With the Shared account mode, every query of every user runs under the service account of the data source, exactly as it did when connections carried their own credentials.
With the Individual credentials mode, the account depends on what is being done:
- Results a person asks for — a data mart preview, a data mart or measure verification — run under that person's own database account, so they see only what their permissions in the database allow
- Reading the structure of the database, creating and updating data mart objects in the database, and checking formulas against the database always run under the service account: these are design-time operations that must not depend on one person's rights
Messages a user may see when their own credentials are used:
| Message | Cause |
|---|---|
| "You do not have configured credentials for this database. Please contact your administrator." | The user has no credentials of their own and the fallback to the service account is off |
| "Database authentication failed. Please verify your credentials." | The database rejected the login or the password |
| "You do not have sufficient permissions to access the requested data in the database." | The account exists but has no rights on the requested data |
| "Shared account is being used for data access. Your individual access rights are not applied." | A warning shown when the fallback to the service account has taken over |
| "The data source “name” has been disabled by an administrator" | The data source has been taken out of use |
User queries — data mart previews, data mart and measure verifications — that go through one connection under one set of credentials share a limited number of database sessions, and each session is released as soon as the query ends, so many users can query the same database at the same time. When the database cannot accept a connection, the message names the cause:
| Message | Cause |
|---|---|
| "The database has reached its connection limit: too many connections are open. Try again later or ask the administrator to raise max_connections" | The database refused a new session because its connection limit is reached |
| "Could not connect to the database: the server is unreachable or dropped the connection. Check that it is running and reachable" | The database server does not respond, refused the connection or dropped it |
| "The database did not accept a connection in time: all connections are busy or the server does not respond. Try again later" | No session became free, or the connection was not established, within the waiting time |
Transferring access rights to data mart views
The Transfer access rights to created data mart views flag on the Parameters tab decides whether the access rights of the source tables are carried over to the objects a data mart creates in the target database. It is cleared by default: the transfer requires extra privileges for the service account, so an administrator turns it on deliberately. What exactly is transferred, and how it affects deployment and updating, is described on the Access rights transfer page.
Editing a data source
- The Data source type cannot be changed on a saved data source — the connections built on it, and the database structure stored for them, belong to that engine. To move to another engine, create a new data source
- Stored secrets — the Service user password and the vault token or AppRole — are never shown again. Click Edit next to the field to replace one; leaving it untouched keeps the stored value
- Everything else, including the address, the authentication mode, the credential source and the flags, can be changed at any time. The change applies to every project connection that uses this data source
Activating and disabling a data source
The disable button in the list row takes a data source out of use without deleting it: its status becomes Disabled and every operation through it is refused with the message naming the data source. The connections that use it stay in their projects and start working again as soon as the data source is activated with the activate button in the same row.
Deleting a data source
The delete button removes a data source through a confirmation dialog, which warns that the connections referring to it have to be pointed at another source first. A data source that is still used by project connections is not deleted — the message "The data source is used by project connections and cannot be deleted" is shown. Repoint or delete those connections, then delete the data source.
Only connections of existing projects and project versions count. Connections of deleted projects and of deleted project versions, including versions in the trash, do not prevent the deletion: they are detached from the data source when it is deleted. When a project is deleted, its connections are deleted together with it.
Audit log entries
Data source configuration is recorded in the System Journal under the data source management category: a data source added, changed or deleted, and a data source user added, changed or deleted. A connection test is not recorded. Every changed field is recorded with its old and new value; passwords and vault secrets are marked as changed without revealing their values.
Query-time events are recorded as well: whose credentials a connection was opened with, when the service account stepped in instead of a user's own credentials, and when the database rejected credentials or refused access.