From Data Lakehouse to Support Agent: Syncing Vital Data with Zendesk Custom Objects

This past week, I focused on a key integration project designed to bring up-to-date sales and professional services data directly into our support team’s Zendesk Agent Workspace. The goal? To equip our agents with the context they need to provide faster, smarter support.

The Challenge: Bridging Data Silos

Our sales data (specifically, contract items) and professional services data (project phases) live in our internal systems and are synced regularly to our Databricks data lakehouse. While this is great for internal reporting, getting this dynamic information seamlessly into Zendesk, where our support agents operate daily, required a robust, automated solution. Agents needed to see, for example, which contract tier a customer is on or which phase a current project is in.

The Solution: A Cloud-Powered Integration Pipeline

I designed a solution using Databricks, Google Cloud Functions, and Zendesk Custom Objects to create a reliable, scheduled sync.

1. Data Retrieval in Databricks

The first step was creating an efficient way to pull only the new or updated data since the last successful sync.

The Databricks Notebook: I built a Databricks notebook that accepts a single date variable. It runs two separate queries against the contract_items and project_phases tables, fetching records where the updated_at date is more recent than the provided variable.
The Output: The results from both queries are loaded into a single Pandas dataframe and then converted into a JSON string—the ideal format for passing to our next processing step. The entire job typically takes between 5 to 10 minutes.

2. Orchestration with Google Cloud Functions

To manage the execution and ensure a reliable, stateful job, I leveraged two Google Cloud Functions under our Support Services project, scheduled to run every morning:

Function 1: The Trigger: This function is scheduled to run right after our origin data syncs to Databricks. It reads the last successful run date from a cursor, triggers the Databricks notebook job, and records the job ID in Firebase under the scheduled tasks cursors document.
Function 2: The Collector & Processor: Scheduled 30 minutes later, this function retrieves the job ID from Firebase, collects the Databricks job output, and processes the new data.
State Management: If the data collection is successful, this function saves a new, successful run date to Firebase, which the first function will use to run the next job.

3. Data Delivery to Zendesk Custom Objects

Once the JSON output is collected, the final step is ensuring the data lands in the right place within Zendesk.

Locating the Target: For each updated data item, the corresponding Organization is first found in Zendesk.
Creating/Updating Records: A new or updated Custom Object record is then created with the job output data.
The Directory Object: All custom object records contain a lookup field referencing the parent Organization. They also link to a central "directory" Custom Object record for that organization, allowing all custom data points to be easily accessed in one place by agents.

The Impact

This integration pipeline ensures that our support agents now have up-to-date contract and project phase information readily available inside the Agent Workspace. This clarity allows them to provide faster, more informed support for our clients.

Next up, I'll be looking at expanding this framework to sync other relevant operational data as well as bringing Custom Object data into the Agent Ticketing View through a Zendesk Custom App!