Connectors Overview¶

Connectors enable Moltler to pull data from external services like GitHub, Jira, Datadog, Salesforce, and more.

What are Connectors?¶

A Connector is a configured connection to an external data source that allows you to:

Query external data with familiar SQL-like syntax
Sync data to Elasticsearch for analysis
Execute actions on external systems

-- Create a connector
CREATE CONNECTOR github_myorg
TYPE 'github'
CONFIG {
    "token": "{{secrets.github_token}}",
    "org": "mycompany"
};

-- Query it
SELECT * FROM github_myorg.issues
WHERE state = 'open' AND labels CONTAINS 'bug';

-- Sync to Elasticsearch
SYNC CONNECTOR github_myorg TO github-issues-*;

Available Connectors¶

DevOps & Development¶

Connector	Description	Key Entities
GitHub	Repositories, issues, PRs, actions	`repos`, `issues`, `pulls`, `actions`
GitLab	Projects, issues, MRs, pipelines	`projects`, `issues`, `merges`, `pipelines`
Jira	Issues, projects, sprints	`issues`, `projects`, `sprints`
Confluence	Pages, spaces, comments	`pages`, `spaces`

Monitoring & Observability¶

Connector	Description	Key Entities
Datadog	Metrics, monitors, events	`metrics`, `monitors`, `events`
New Relic	APM, infrastructure, alerts	`applications`, `hosts`, `alerts`
Prometheus	Metrics, alerts	`metrics`, `alerts`
Grafana	Dashboards, alerts	`dashboards`, `alerts`

Incident Management¶

Connector	Description	Key Entities
PagerDuty	Incidents, services, on-call	`incidents`, `services`, `oncall`
Opsgenie	Alerts, schedules	`alerts`, `schedules`
Statuspage	Incidents, components	`incidents`, `components`

Communication¶

Connector	Description	Key Entities
Slack	Messages, channels, users	`messages`, `channels`, `users`
Microsoft Teams	Messages, channels, teams	`messages`, `channels`

Cloud Providers¶

Connector	Description	Key Entities
AWS	EC2, S3, Lambda, CloudWatch	`ec2`, `s3`, `lambda`, `cloudwatch`
GCP	Compute, Storage, Functions	`compute`, `storage`, `functions`
Azure	VMs, Storage, Functions	`vms`, `storage`, `functions`

CRM & Support¶

Connector	Description	Key Entities
Salesforce	Accounts, opportunities, cases	`accounts`, `opportunities`, `cases`
Zendesk	Tickets, users, organizations	`tickets`, `users`, `organizations`
HubSpot	Contacts, deals, tickets	`contacts`, `deals`, `tickets`

Creating Connectors¶

Basic Syntax¶

CREATE CONNECTOR connector_name
TYPE 'connector_type'
CONFIG {
    -- Configuration options
};

With Secrets¶

Use secret references for sensitive data:

CREATE CONNECTOR my_connector
TYPE 'github'
CONFIG {
    "token": "{{secrets.github_token}}"
};

Configure secrets:

moltler secrets set github_token ghp_xxxxxxxxxxxx

With Options¶

CREATE CONNECTOR my_connector
TYPE 'jira'
CONFIG {
    "url": "https://mycompany.atlassian.net",
    "email": "{{secrets.jira_email}}",
    "token": "{{secrets.jira_token}}"
}
OPTIONS {
    "sync_interval": "5m",
    "batch_size": 100,
    "rate_limit": 10
};

Querying Connectors¶

SELECT Syntax¶

SELECT columns FROM connector.entity
WHERE conditions
ORDER BY column
LIMIT n;

Examples¶

-- GitHub: Open bugs assigned to me
SELECT title, created_at, html_url
FROM github_org.issues
WHERE state = 'open' 
  AND assignee = 'myusername'
  AND labels CONTAINS 'bug'
ORDER BY created_at DESC;

-- Jira: High priority incidents
SELECT key, summary, status, priority
FROM jira_project.issues
WHERE project = 'OPS'
  AND type = 'Incident'
  AND priority IN ('High', 'Critical')
  AND status != 'Resolved';

-- Datadog: Recent alerts
SELECT name, status, last_triggered
FROM datadog_prod.monitors
WHERE status = 'Alert'
ORDER BY last_triggered DESC
LIMIT 10;

Joins (Coming Soon)¶

-- Correlate GitHub PRs with Jira tickets
SELECT 
    g.title as pr_title,
    j.summary as ticket_summary,
    j.status as ticket_status
FROM github_org.pulls g
JOIN jira_project.issues j ON g.title LIKE '%' || j.key || '%'
WHERE g.state = 'open';

Syncing Data¶

One-Time Sync¶

SYNC CONNECTOR github_org TO github-issues-*;

Scheduled Sync¶

SYNC CONNECTOR github_org TO github-issues-*
SCHEDULE '*/15 * * * *';  -- Every 15 minutes

Incremental Sync¶

SYNC CONNECTOR github_org TO github-issues-*
INCREMENTAL ON updated_at
SCHEDULE '*/5 * * * *';

Selective Sync¶

-- Only sync specific entities
SYNC CONNECTOR github_org.issues TO github-issues-*
WHERE labels CONTAINS 'incident'
SCHEDULE '*/5 * * * *';

Connector Actions¶

Some connectors support write operations:

Create¶

CONNECTOR_EXEC github_org.issues.create({
    "title": "New issue from Moltler",
    "body": "Created automatically",
    "labels": ["automated"]
});

Update¶

CONNECTOR_EXEC github_org.issues.update(
    issue_number => 123,
    data => {"state": "closed"}
);

Custom Actions¶

CONNECTOR_EXEC pagerduty_prod.incidents.acknowledge(
    incident_id => 'P123ABC'
);

Managing Connectors¶

List Connectors¶

SHOW CONNECTORS;

Output:

┌─────────────────┬──────────┬────────────┬───────────────┐
│ Name            │ Type     │ Status     │ Last Sync     │
├─────────────────┼──────────┼────────────┼───────────────┤
│ github_myorg    │ github   │ connected  │ 5 minutes ago │
│ jira_ops        │ jira     │ connected  │ 2 minutes ago │
│ datadog_prod    │ datadog  │ error      │ 1 hour ago    │
└─────────────────┴──────────┴────────────┴───────────────┘

View Connector Details¶

SHOW CONNECTOR github_myorg;

Test Connection¶

TEST CONNECTOR github_myorg;

Delete Connector¶

DROP CONNECTOR github_myorg;

Connector Architecture¶

┌──────────────────────────────────────────────────────────────┐
│                        Moltler                                │
├──────────────────────────────────────────────────────────────┤
│                                                               │
│   QUERY                    SYNC                   ACTION      │
│   SELECT * FROM            SYNC CONNECTOR        CONNECTOR_   │
│   connector.entity         TO index              EXEC(...)    │
│        │                        │                    │        │
│        └────────────┬───────────┴────────────────────┘        │
│                     │                                          │
│              ┌──────┴──────┐                                  │
│              │  Connector  │                                  │
│              │   Engine    │                                  │
│              └──────┬──────┘                                  │
│                     │                                          │
├─────────────────────┼──────────────────────────────────────────┤
│                     ▼                                          │
│   ┌──────────────────────────────────────────────────────┐    │
│   │                 Elasticsearch                          │    │
│   │   ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │    │
│   │   │  Cache   │  │  Index   │  │ Connector Config │   │    │
│   │   └──────────┘  └──────────┘  └──────────────────┘   │    │
│   └──────────────────────────────────────────────────────┘    │
│                                                               │
└───────────────────────────────────────────────────────────────┘
                              │
                              ▼
              ┌───────────────────────────────┐
              │       External Services       │
              │  GitHub  Jira  Datadog  AWS   │
              └───────────────────────────────┘

Best Practices¶

1. Use Secrets for Credentials¶

-- Good: Use secret references
CREATE CONNECTOR my_conn TYPE 'github'
CONFIG {"token": "{{secrets.github_token}}"};

-- Bad: Hardcoded credentials
CREATE CONNECTOR my_conn TYPE 'github'
CONFIG {"token": "ghp_actualtoken123"};

2. Set Rate Limits¶

CREATE CONNECTOR my_conn TYPE 'github'
CONFIG {...}
OPTIONS {
    "rate_limit": 10,       -- Requests per second
    "rate_limit_burst": 50  -- Burst allowance
};

3. Handle Errors Gracefully¶

BEGIN
    TRY
        SELECT * FROM my_connector.items;
    CATCH connector_error THEN
        -- Fallback to cached data
        SELECT * FROM my-connector-cache-*;
    END TRY;
END;

4. Optimize Queries¶

-- Good: Specific fields and filters
SELECT id, title, status FROM jira.issues
WHERE project = 'OPS' AND status = 'Open'
LIMIT 100;

-- Bad: Full table scan
SELECT * FROM jira.issues;

What's Next?¶

GitHub Connector

Connect to GitHub repositories.

GitHub
Jira Connector

Connect to Jira projects.

Jira
Datadog Connector

Connect to Datadog monitoring.

Datadog
Build Your Own

Create custom connectors.

Building Connectors