Skip to main content

Databricks integrations

Sync your Simplesat responses to a Databricks Delta table and query them alongside the rest of your data.

Connect Databricks, and Simplesat copies your survey responses into a table in the catalog and schema you choose. It then keeps the table current for the surveys and sentiments you pick.

Before you start

The integration is available on every Simplesat plan. To connect it and switch it on, you need the Manage integrations permission, which Owner and Admin have by default.

In Databricks, you need:

  • A workspace on Databricks on AWS, whose address ends in .cloud.databricks.com. Azure Databricks and Databricks on Google Cloud workspaces can't connect.

  • A SQL warehouse, and its ID.

  • A catalog and a schema for the table.

  • A service principal with an OAuth secret, for the integration to connect as.

The service principal needs:

  • Permission to use the SQL warehouse

  • USE CATALOG on the catalog

  • USE SCHEMA and CREATE TABLE on the schema

  • SELECT and MODIFY on the table, if the table already exists and someone else created it

Integration security, permissions and data flow covers what a connected tool can read and write.

Connect Databricks

Start in Databricks, where you create the secret and find the warehouse ID:

  1. Add a service principal for Simplesat to the workspace.

  2. In the workspace's Settings, go to Identity and access, click Manage next to Service principals, and select the service principal.

  3. On Secrets, click Generate secret. Set the lifetime in days, up to 730, and if you choose Scopes, include sql. Then click Generate.

  4. Copy the client ID and the secret, which Databricks shows only once.

  5. Find the warehouse ID in SQL Warehouses. Open the warehouse, go to Connection details, and copy the part of the HTTP path after /sql/1.0/warehouses/.

Then, in Simplesat:

  1. On the Integrations page, open Databricks and click Authenticate.

  2. Fill in Client ID and Client Secret with the service principal's client ID and secret, and Workspace ID with the first part of your workspace's address. For https://dbc-a1b2c3d4-e5f6.cloud.databricks.com, enter dbc-a1b2c3d4-e5f6. Click Authenticate. If the Databricks page doesn't show Authenticated with Databricks after that, reload it and try again.

  3. When the page shows Authenticated with Databricks, click Activate integration on Sync feedback from Simplesat to Databricks. It starts working right away, so fill in its settings next.

  4. In its Field mapping, type a value for each setting, then click Save.

Setting

What to enter

Catalog

The catalog that holds the schema

Schema

The schema the table goes in

Table

The table's name. If no table has that name, the integration creates it

Warehouse ID

The SQL warehouse's ID

  1. Under When should this integration run?, keep Every response from all current and future surveys. To narrow it down, select Let me choose and pick the surveys and, for each one, the sentiments, with or without a comment.

What lands in Databricks and when

The table

The table is a Delta table. It's created when the first response reaches the integration after you activate it. Then every existing response in your workspace is copied in, from every survey, whatever you chose under When should this integration run?. With a lot of history, that takes a while. After that, only responses from the surveys and sentiments you chose are added.

If you change Table to a name that doesn't exist yet, the next response creates that table and copies every response into it.

When rows are added and updated

Each response has one row, matched on id. A new response adds a row, and a response that changes has its row updated, for example when the customer answers another question later or someone edits a rating.

The integration gets a response no sooner than five minutes after it arrives, and again whenever it changes. Only active responses are sent, not withheld, hidden or spam ones.

Tagging a response in Simplesat doesn't update its row, so the tag reaches Databricks only when the response itself next changes.

What each row holds

Column

Type

What it holds

id

BIGINT

The response ID

survey

STRING (JSON)

The survey's ID and name

tags

STRING (JSON)

The response's tags

created, modified

TIMESTAMP

When the response was created and last changed

ip_address

STRING

The respondent's IP address

ticket

STRING (JSON)

The ticket's ID in Simplesat and in your helpdesk, its subject and its ticket fields

team_members

STRING

Always empty

customer

STRING (JSON)

The customer's IDs, name, email, company, tags and attributes

answers

STRING (JSON)

Each answer: the question, the rating or choice, the comment, the sentiment, and the follow-up question and answer

source

STRING

Where the response came from

language

STRING

The response's language

What answers and tags hold depends on how a row was written:

  • Rows copied in when the table is created hold every answer in answers, and the customer's tags in tags, not the response's.

  • Rows written as responses arrive hold the response's tags in tags, and in answers only the answer to the question your metric rating comes from. When a response changes, its row is rewritten this way.

Replace the secret or change the workspace

A Databricks OAuth secret lasts 730 days at most. When it expires or is deleted, every sync fails, while the Databricks page still shows Authenticated with Databricks. Replace it before then:

  1. In Databricks, generate a new secret for the service principal.

  2. On the Databricks page in Simplesat, click Revoke access.

  3. Click Authenticate and enter the client ID, the new secret and the workspace ID.

  4. Click Activate integration on Sync feedback from Simplesat to Databricks. Reconnecting keeps its settings but leaves it off until you do.

To connect a different workspace or service principal, follow the same steps with its details.

Heads up: Responses that arrive while the integration is off or failing aren't added later.

Troubleshooting

Nothing has reached my Databricks table since a certain date

Start with the secret. If it expired or was deleted, every sync fails while the page still shows Authenticated with Databricks. Generate a new secret, then click Revoke access, Authenticate with the new secret, and click Activate integration. If the secret is fine, check that the SQL warehouse still exists and that the service principal still has its permissions. If none of that explains it, contact us. Responses that arrived while syncs were failing aren't added later.

The table hasn't appeared in my schema

The table appears with the first response that reaches the integration after you activate it, at least five minutes after that response arrives. If a response has come in since then, check that:

  • Catalog, Schema, Table and Warehouse ID are filled in and saved.

  • The survey and the response's sentiment are included under When should this integration run?.

  • The service principal can use the warehouse and has USE CATALOG, USE SCHEMA and CREATE TABLE.

  • The workspace is on Databricks on AWS.

My Databricks rows are missing answers or team members

Rows copied in when the table is created hold every answer until their response changes and the row is rewritten. Rows written as responses arrive hold only the answer to the question your metric rating comes from. team_members is always empty.

Tags I add in Simplesat don't show up in Databricks

Tagging a response doesn't update its row, so the tag reaches Databricks only when the response itself next changes. Rows copied in when the table is created hold the customer's tags in tags, not the response's.

FAQ

Can I connect Azure Databricks or Databricks on Google Cloud?

No. The integration connects only to workspaces on Databricks on AWS, whose address ends in .cloud.databricks.com. To get your responses into another workspace, export them as a CSV from Feedback, or read them through the Simplesat API.

Does the integration create the catalog, schema or table?

It creates the table if it doesn't exist. The catalog, the schema and the SQL warehouse have to exist already.

How do I read the JSON columns?

In Databricks SQL, put a colon after the column name. For example, customer:email returns the customer's email, and survey:name returns the survey's name. To cast a value, add ::, for example ::int.

Which responses go into the table when it's created?

Every existing response in your workspace, from every survey. After that, only responses from the surveys and sentiments you chose are added or updated.

Did this answer your question?