GitHub

The GitHub connector is a Palantir-provided driver for GitHub.

To create a new GitHub source, follow the standard setup flow for Palantir-provided drivers, then use the sections below for GitHub-specific configuration and networking. For the complete property reference, see the official GitHub driver documentation ↗.

Introduction

The GitHub connector is best suited for reading data in bulk. It surfaces the GitHub data model — repositories, issues, pull requests, commits, and related entities — as relational tables. You can browse these in the source explorer and pull them into Foundry with a batch sync. This makes it a good fit for mirroring an entire repository or organization into Foundry for downstream analysis.

The GitHub source explorer previewing tables such as Commits before adding them to a sync.

To create or update records in GitHub, reuse your existing GitHub source and call the GitHub REST API ↗ from an external transform or external function. No second source is required. See Use GitHub sources in code below.

Authentication

The connector authenticates with GitHub over OAuth 2.0. You register an application on GitHub, then enter its Client ID and Client secret on the Foundry source (OAuthClientId / OAuthClientSecret). With InitiateOAuth set to REFRESH, the driver runs GitHub's OAuth web application flow ↗ on the first sync and keeps the token refreshed.

GitHub offers two application types, both built on OAuth 2.0 and configured the same way. A GitHub App (recommended) adds fine-grained, per-resource permissions — for example, read-only access to a single repository's issues — and short-lived tokens. An OAuth app uses broad, account-level scopes, such as repo. The steps below use a GitHub App; an OAuth app is identical apart from the provisioning permissions workflow, where you instead select OAuth app from Settings > Developer settings. For a full comparison, see Differences between GitHub Apps and OAuth apps ↗.

Create the application

In GitHub, select Settings > Developer settings > GitHub Apps > New GitHub App ↗ and set the Callback URL to the value shown on the Foundry source setup page.

The GitHub Create GitHub App form, showing the app name, homepage URL, and callback URL fields.

Under Permissions & events, grant only what the sync needs (for example, read-only Contents and Issues), create the app, and install it from the Install App tab on the target repositories or organization. Then, on the app's General page, copy the Client ID and generate a Client secret.

The GitHub App General page, showing the Client ID and Client secrets sections.

Add your GitHub client ID and secret to the source

Navigate to Data Connection in Foundry and enter the Client ID and Client secret into the source's OAuthClientId and OAuthClientSecret properties and keep InitiateOAuth set to REFRESH.

The Foundry GitHub source connection settings, showing the OAuth client ID, client secret, and callback URL properties alongside the approved egress policies.

Configuration

The properties below are mandatory or recommended.

PropertyRequired?DescriptionDefault
InitiateOAuthRecommendedSpecifies the process for obtaining or refreshing the OAuth access token, which maintains user access while an authenticated, authorized user is working.REFRESH
OAuthClientIdRecommendedSpecifies the client ID (also known as the consumer key) assigned to your custom OAuth application. This ID is required to identify the application to the OAuth authorization server during authentication.
OAuthClientSecretRecommendedSpecifies the client secret assigned to your custom OAuth application. This confidential value is used to authenticate the application to the OAuth authorization server.
URLRecommendedThe base URL for the GitHub environment you are connecting to.

Networking

The table below lists the domains that the source needs to be able to access in order to successfully run.

For each domain, add a corresponding egress policy. If the source is hosted on-premises and not directly reachable from Foundry, use an agent proxy egress policy instead; the agent host itself must also be able to reach the listed domains. See using an agent as a proxy for details.

DomainRequired
github.comAlways
api.github.comAlways

Use GitHub sources in code

To create or update records in GitHub, or to use any part of the GitHub REST API ↗ beyond bulk reads, call the API from an external transform or external function. Reuse your existing GitHub source, authenticating with a short-lived (roughly one hour time-to-live) GitHub App installation token minted in code. No second source is required.

Next, import the source into your external transforms or functions repository. The source does not expose a built-in HTTPS client (get_https_connection().get_client()), but it inherits its api.github.com egress, which is covered by the network egress policies above. You can call the GitHub REST API directly with requests in Python or the source's fetch in TypeScript.

Example: Read and update GitHub issues

The example below demonstrates how to read and update GitHub issues using Python or TypeScript through a GitHub App.

This workflow is not supported by an OAuth app, which requires a personal access token.

First, navigate to your source's Connection details page in Data Connection and add the GitHub App's private key by selecting Add property > New encrypted property. Name the property Other, enabling it to be read back with get_secret("Other").

Next, set GITHUB_APP_ID in your code to the GitHub App ID or client ID. This identifier is public and does not need secret storage. The TypeScript example instead reads it from the source with getSecret("OAuthClientId") — either method works, since the value is not secret. You can reference complete code examples below.

All four example tabs share the same GitHub App authentication and REST logic. However, they differ in source wiring, registration, and how they issue HTTP requests. The Python examples use requests, the TypeScript v1 example uses the source's fetch, and the TypeScript v2 example uses getFetch(source) from @palantir/functions-sources.

Function preview failures

Live preview cannot bind the source to the function, so these examples always error during preview. Instead, deploy the function and run it from a Workshop application, or via any serverless execution, to confirm it works.

Copied!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 import time import jwt import pandas as pd import requests from transforms.api import LightweightOutput, Output, transform from transforms.external.systems import ( ResolvedSource, Source, external_systems, ) API_URL = "https://api.github.com" GITHUB_APP_ID = "<github_app_id_or_client_id>" OWNER = "<owner>" REPOSITORY = "<repository>" ISSUE_NUMBER = "<issue_number>" UPDATED_BODY = "<updated_issue_body>" GITHUB_HEADERS = { "Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28", } def _installation_token( github: ResolvedSource, owner: str, repository: str, ) -> str: private_key = github.get_secret("Other") now = int(time.time()) app_jwt = jwt.encode( { "iat": now - 60, "exp": now + 540, "iss": GITHUB_APP_ID, }, private_key, algorithm="RS256", ) app_headers = { **GITHUB_HEADERS, "Authorization": f"Bearer {app_jwt}", } installation_response = requests.get( f"{API_URL}/repos/{owner}/{repository}/installation", headers=app_headers, timeout=30, ) installation_response.raise_for_status() installation_id = installation_response.json()["id"] token_response = requests.post( ( f"{API_URL}/app/installations/{installation_id}" "/access_tokens" ), headers=app_headers, timeout=30, ) token_response.raise_for_status() return token_response.json()["token"] def _installation_headers(token: str) -> dict[str, str]: return { **GITHUB_HEADERS, "Authorization": f"Bearer {token}", } @external_systems( github=Source("<source_rid>"), ) @transform.using( output=Output("<issues_output_dataset_rid>"), ) def get_issues( github: ResolvedSource, output: LightweightOutput, ) -> None: token = _installation_token( github, OWNER, REPOSITORY, ) response = requests.get( f"{API_URL}/repos/{OWNER}/{REPOSITORY}/issues", headers=_installation_headers(token), params={ "state": "all", "per_page": 100, }, timeout=30, ) response.raise_for_status() output.write_table(pd.DataFrame([ { "number": issue["number"], "title": issue["title"], "body": issue["body"], "state": issue["state"], } for issue in response.json() ])) @external_systems( github=Source("<source_rid>"), ) @transform.using( output=Output("<update_output_dataset_rid>"), ) def patch_issue( github: ResolvedSource, output: LightweightOutput, ) -> None: token = _installation_token( github, OWNER, REPOSITORY, ) response = requests.patch( ( f"{API_URL}/repos/{OWNER}/{REPOSITORY}" f"/issues/{ISSUE_NUMBER}" ), headers=_installation_headers(token), json={"body": UPDATED_BODY}, timeout=30, ) response.raise_for_status() issue = response.json() output.write_table(pd.DataFrame([{ "issue_number": issue["number"], "status_code": response.status_code, "body": issue["body"], "state": issue["state"], "updated_at": issue["updated_at"], "html_url": issue["html_url"], }]))
Copied!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 import time from dataclasses import dataclass import jwt import requests from functions.api import function from functions.sources import get_source API_URL = "https://api.github.com" GITHUB_APP_ID = "<github_app_id_or_client_id>" SOURCE_ALIAS = "<source_alias>" GITHUB_HEADERS = { "Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28", } @dataclass class GitHubIssue: number: int title: str body: str | None state: str @dataclass class GitHubIssueUpdate: issue_number: int status_code: int body: str | None state: str updated_at: str html_url: str def _installation_token( github, owner: str, repository: str, ) -> str: private_key = github.get_secret("Other") now = int(time.time()) app_jwt = jwt.encode( { "iat": now - 60, "exp": now + 540, "iss": GITHUB_APP_ID, }, private_key, algorithm="RS256", ) app_headers = { **GITHUB_HEADERS, "Authorization": f"Bearer {app_jwt}", } installation_response = requests.get( f"{API_URL}/repos/{owner}/{repository}/installation", headers=app_headers, timeout=30, ) installation_response.raise_for_status() installation_id = installation_response.json()["id"] token_response = requests.post( ( f"{API_URL}/app/installations/{installation_id}" "/access_tokens" ), headers=app_headers, timeout=30, ) token_response.raise_for_status() return token_response.json()["token"] def _installation_headers(token: str) -> dict[str, str]: return { **GITHUB_HEADERS, "Authorization": f"Bearer {token}", } @function(sources=["<source_alias>"]) def get_issues( owner: str, repository: str, ) -> list[GitHubIssue]: github = get_source(SOURCE_ALIAS) token = _installation_token( github, owner, repository, ) response = requests.get( f"{API_URL}/repos/{owner}/{repository}/issues", headers=_installation_headers(token), params={ "state": "all", "per_page": 100, }, timeout=30, ) response.raise_for_status() return [ GitHubIssue( number=issue["number"], title=issue["title"], body=issue["body"], state=issue["state"], ) for issue in response.json() ] @function(sources=["<source_alias>"]) def patch_issue( owner: str, repository: str, issue_number: int, body: str, ) -> GitHubIssueUpdate: github = get_source(SOURCE_ALIAS) token = _installation_token( github, owner, repository, ) response = requests.patch( ( f"{API_URL}/repos/{owner}/{repository}" f"/issues/{issue_number}" ), headers=_installation_headers(token), json={"body": body}, timeout=30, ) response.raise_for_status() issue = response.json() return GitHubIssueUpdate( issue_number=issue["number"], status_code=response.status_code, body=issue["body"], state=issue["state"], updated_at=issue["updated_at"], html_url=issue["html_url"], )
Copied!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 import { ExternalSystems, Function, Integer, UserFacingError, } from "@foundry/functions-api"; import { GitHubSource, } from "@foundry/external-systems/sources"; import * as jwt from "jsonwebtoken"; const API_URL = "https://api.github.com"; const GITHUB_HEADERS: Record<string, string> = { "Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28", }; export interface GitHubIssue { number: Integer; title: string; body?: string; state: string; } export interface GitHubIssueUpdate { issueNumber: Integer; statusCode: Integer; body?: string; state: string; updatedAt: string; htmlUrl: string; } interface GitHubInstallationResponse { id: number; } interface GitHubInstallationTokenResponse { token: string; } interface GitHubIssueResponse { number: number; title: string; body: string | null; state: string; updated_at: string; html_url: string; } function appJwt(): string { const appId = GitHubSource.getSecret("OAuthClientId"); const privateKey = GitHubSource.getSecret("Other"); const now = Math.floor(Date.now() / 1000); return jwt.sign( { iat: now - 60, exp: now + 540, iss: appId, }, privateKey, { algorithm: "RS256", }, ); } function appHeaders(): Record<string, string> { return { ...GITHUB_HEADERS, "Authorization": `Bearer ${appJwt()}`, }; } async function installationToken( owner: string, repository: string, ): Promise<string> { const headers = appHeaders(); const installationResponse = await GitHubSource.fetch( ( `${API_URL}/repos/${owner}/${repository}` + "/installation" ), { method: "GET", headers, }, ); if (!installationResponse.ok) { throw new UserFacingError( ( "GitHub installation lookup returned HTTP " + `${installationResponse.status}.` ), ); } const installation = await installationResponse.json() as GitHubInstallationResponse; const tokenResponse = await GitHubSource.fetch( ( `${API_URL}/app/installations/${installation.id}` + "/access_tokens" ), { method: "POST", headers, }, ); if (!tokenResponse.ok) { throw new UserFacingError( ( "GitHub installation-token request returned HTTP " + `${tokenResponse.status}.` ), ); } const tokenResult = await tokenResponse.json() as GitHubInstallationTokenResponse; return tokenResult.token; } function installationHeaders( token: string, ): Record<string, string> { return { ...GITHUB_HEADERS, "Authorization": `Bearer ${token}`, }; } function issueResult( issue: GitHubIssueResponse, ): GitHubIssue { return { number: issue.number, title: issue.title, body: issue.body ?? undefined, state: issue.state, }; } export class GitHubFunctions { @ExternalSystems({ sources: [GitHubSource], }) @Function() public async getIssues( owner: string, repository: string, ): Promise<GitHubIssue[]> { const token = await installationToken( owner, repository, ); const response = await GitHubSource.fetch( ( `${API_URL}/repos/${owner}/${repository}` + "/issues?state=all&per_page=100" ), { method: "GET", headers: installationHeaders(token), }, ); if (!response.ok) { throw new UserFacingError( `GitHub returned HTTP ${response.status}.`, ); } const issues = await response.json() as GitHubIssueResponse[]; return issues.map(issueResult); } @ExternalSystems({ sources: [GitHubSource], }) @Function() public async patchIssue( owner: string, repository: string, issueNumber: Integer, body: string, ): Promise<GitHubIssueUpdate> { const token = await installationToken( owner, repository, ); const response = await GitHubSource.fetch( ( `${API_URL}/repos/${owner}/${repository}` + `/issues/${issueNumber}` ), { method: "PATCH", headers: { ...installationHeaders(token), "Content-Type": "application/json", }, body: JSON.stringify({ body }), }, ); if (!response.ok) { throw new UserFacingError( `GitHub returned HTTP ${response.status}.`, ); } const issue = await response.json() as GitHubIssueResponse; return { issueNumber: issue.number, statusCode: response.status, body: issue.body ?? undefined, state: issue.state, updatedAt: issue.updated_at, htmlUrl: issue.html_url, }; } }
Copied!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 // githubClient.ts — shared GitHub App authentication and REST logic import { Integer, UserFacingError, } from "@osdk/functions"; import { getFetch, getSource, type Source, } from "@palantir/functions-sources"; import jwt from "jsonwebtoken"; export const GITHUB_SOURCE_RID = "<source_rid>"; const API_URL = "https://api.github.com"; const GITHUB_HEADERS: Record<string, string> = { "Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28", }; export interface GitHubIssue { number: Integer; title: string; body?: string; state: string; } export interface GitHubIssueUpdate { issueNumber: Integer; statusCode: Integer; body?: string; state: string; updatedAt: string; htmlUrl: string; } interface GitHubInstallationResponse { id: number; } interface GitHubInstallationTokenResponse { token: string; } interface GitHubIssueResponse { number: number; title: string; body: string | null; state: string; updated_at: string; html_url: string; } function requiredSecret( source: Source, name: string, ): string { const secret = source.secrets[name]; if (secret === undefined || secret.length === 0) { throw new UserFacingError( `The GitHub source secret ${name} is not configured.`, ); } return secret; } function appHeaders( source: Source, ): Record<string, string> { const appId = requiredSecret( source, "OAuthClientId", ); const privateKey = requiredSecret( source, "Other", ); const now = Math.floor(Date.now() / 1000); const appJwt = jwt.sign( { iat: now - 60, exp: now + 540, iss: appId, }, privateKey, { algorithm: "RS256", }, ); return { ...GITHUB_HEADERS, "Authorization": `Bearer ${appJwt}`, }; } async function installationClient( owner: string, repository: string, ) { const source = await getSource({ rid: GITHUB_SOURCE_RID, }); const request = await getFetch(source); const headers = appHeaders(source); const installationResponse = await request( ( `${API_URL}/repos/${owner}/${repository}` + "/installation" ), { method: "GET", headers, }, ); if (!installationResponse.ok) { throw new UserFacingError( ( "GitHub installation lookup returned HTTP " + `${installationResponse.status}.` ), ); } const installation = await installationResponse.json() as GitHubInstallationResponse; const tokenResponse = await request( ( `${API_URL}/app/installations/${installation.id}` + "/access_tokens" ), { method: "POST", headers, }, ); if (!tokenResponse.ok) { throw new UserFacingError( ( "GitHub installation-token request returned HTTP " + `${tokenResponse.status}.` ), ); } const tokenResult = await tokenResponse.json() as GitHubInstallationTokenResponse; return { request, headers: { ...GITHUB_HEADERS, "Authorization": `Bearer ${tokenResult.token}`, }, }; } function issueResult( issue: GitHubIssueResponse, ): GitHubIssue { return { number: issue.number, title: issue.title, body: issue.body ?? undefined, state: issue.state, }; } export async function getIssuesImpl( owner: string, repository: string, ): Promise<GitHubIssue[]> { const { request, headers, } = await installationClient( owner, repository, ); const response = await request( ( `${API_URL}/repos/${owner}/${repository}` + "/issues?state=all&per_page=100" ), { method: "GET", headers, }, ); if (!response.ok) { throw new UserFacingError( `GitHub returned HTTP ${response.status}.`, ); } const issues = await response.json() as GitHubIssueResponse[]; return issues.map(issueResult); } export async function patchIssueImpl( owner: string, repository: string, issueNumber: Integer, body: string, ): Promise<GitHubIssueUpdate> { const { request, headers, } = await installationClient( owner, repository, ); const response = await request( ( `${API_URL}/repos/${owner}/${repository}` + `/issues/${issueNumber}` ), { method: "PATCH", headers: { ...headers, "Content-Type": "application/json", }, body: JSON.stringify({ body }), }, ); if (!response.ok) { throw new UserFacingError( `GitHub returned HTTP ${response.status}.`, ); } const issue = await response.json() as GitHubIssueResponse; return { issueNumber: issue.number, statusCode: response.status, body: issue.body ?? undefined, state: issue.state, updatedAt: issue.updated_at, htmlUrl: issue.html_url, }; } // getIssues.ts — registered getIssues Function import { GITHUB_SOURCE_RID, getIssuesImpl, type GitHubIssue, } from "./githubClient.js"; export const config = { sources: [GITHUB_SOURCE_RID], }; export default async function getIssues( owner: string, repository: string, ): Promise<GitHubIssue[]> { return getIssuesImpl( owner, repository, ); } // patchIssue.ts — registered patchIssue Function import { Integer } from "@osdk/functions"; import { GITHUB_SOURCE_RID, patchIssueImpl, type GitHubIssueUpdate, } from "./githubClient.js"; export const config = { sources: [GITHUB_SOURCE_RID], }; export default async function patchIssue( owner: string, repository: string, issueNumber: Integer, body: string, ): Promise<GitHubIssueUpdate> { return patchIssueImpl( owner, repository, issueNumber, body, ); }