Integrating InDepthAnalysis into your application

Prequisites

In this guide we are building an integration that covers the following scenarios:

  • Uploading audio files to Cyanite.ai
  • Enqueuing a file analysis
  • Setup a webhook for getting notified for when the file analysis has finished
  • Fetching the analysis result.

The tutorial will mostly provide Node.js code. However, all listed GraphQL Operations can be used with any other programming language.

You can find the whole source code of the example integration over on Github: cyanite-ai/cyanite-integration-example.

Scaffolding the project

Create project a folder

First of all we create an empty project

mkdir cyanite-integration && cd cyanite-integration && yarn init -y

Install dependencies

Then we install all the required packages:

yarn add -E envalid@6.0.0 express@4.17.1 body-parser@1.19.0 node-fetch@2.6.0
PackageUsed for
envalidvalidate environment variables
expressminimalist framework for creating a webhook listener
body-parserrequest body parsing middleware for express
node-fetchHTTP request module for sending requests/upload files to the Cyanite.ai API

Obtaining API credentials

Follow Creating an integration for obtaining an access token and a webhook secret.

The create a new file cyanite-integration/.env with the following contents:

ACCESS_TOKEN=YOUR_ACCESS_TOKEN
SECRET=YOUR_SECRET
PORT=8080
API_URL=https://app.cyanite.ai/graphql

Make sure you replace YOUR_SECRET and YOUR_ACCESS_TOKEN` with the corresponding values for your integration.

Uploading and enqueuing files

Pick a file

Scan your music collection for an mp3 that you want to analyze. For receiving the best result your mp3 files should have a bitrate of 320kbs.

You can easily convert your wav (or any othe audio format) files to mp3 with ffmpeg.

ffmpeg -i inputfile.wav -ab 320k outputfile.mp3

Uploading a file

The Upload can be divided into three steps:

  1. Requesting a file upload
  2. Uploading the file
  3. Create an InDepthAnalysis from an uploaded file

Requesting a file upload

For requesting a file upload we use the fileUploadRequest mutation field.

mutation fileUploadRequest {
fileUploadRequest {
id
uploadUrl
}
}

On the FileUploadRequest we select the id and the uploadUrl. We need both fields to proceed.

The uploadUrl is the url to which we are uploading our file.

The id is a unique identifier which we need for creating the InDepthAnalysis from our uploaded file.

Let's start by requesting our file upload!

src/file-upload.js

"use strict";
const envalid = require("envalid");
const fs = require("fs");
const fetch = require("node-fetch");
const { API_URL, ACCESS_TOKEN } = envalid.cleanEnv(process.env, {
API_URL: envalid.str(),
ACCESS_TOKEN: envalid.str()
});
const fileUploadRequestMutation = /* GraphQL */ `
mutation fileUploadRequest {
fileUploadRequest {
id
uploadUrl
}
}
`;
const requestFileUpload = async () => {
const result = await fetch(API_URL, {
method: "POST",
body: JSON.stringify({
query: fileUploadRequestMutation
}),
headers: {
Authorization: "Bearer " + ACCESS_TOKEN,
"Content-Type": "application/json"
}
}).then(res => res.json());
console.log("[info] fileUploadRequest response: ");
console.log(JSON.stringify(result, undefined, 2));
};
const main = async filePath => {
const { id, uploadUrl } = await requestFileUpload(filePath);
console.log({ id, uploadUrl });
};
main(process.argv[2]).catch(err => {
console.error(err);
process.exitCode = 1;
});

Uploading the file

Now that we have all the necessary information for uploading the file, we can perform our file upload.

For uploading we use the HTTP PUT method. You can verify that the upload was successfull by checking the HTTP status code of the response which should be equal to 200.

src/file-upload.js

"use strict";
const envalid = require("envalid");
const fs = require("fs");
const fetch = require("node-fetch");
const { API_URL, ACCESS_TOKEN } = envalid.cleanEnv(process.env, {
API_URL: envalid.str(),
ACCESS_TOKEN: envalid.str()
});
const fileUploadRequestMutation = /* GraphQL */ `
mutation fileUploadRequest {
fileUploadRequest {
id
uploadUrl
}
}
`;
const requestFileUpload = async () => {
const result = await fetch(API_URL, {
method: "POST",
body: JSON.stringify({
query: fileUploadRequestMutation
}),
headers: {
Authorization: "Bearer " + ACCESS_TOKEN,
"Content-Type": "application/json"
}
}).then(res => res.json());
console.log("[info] fileUploadRequest response: ");
console.log(JSON.stringify(result, undefined, 2));
return result.data.fileUploadRequest;
};
const uploadFile = async (filePath, uploadUrl) => {
const result = await fetch(uploadUrl, {
method: "PUT",
body: fs.createReadStream(filePath),
headers: {
"Content-Type": fs.statSync(filePath).size
}
}).then(res => {
if (res.status !== 200) {
throw Error("Failed to upload file.");
}
return res.text();
});
console.log(result);
};
const createInDepthAnalysis = async fileUploadRequestId => {};
const main = async filePath => {
const { id, uploadUrl } = await requestFileUpload(filePath);
await uploadFile(filePath, uploadUrl);
};
main(process.argv[2]).catch(err => {
console.error(err);
process.exitCode = 1;
});

Create an InDepthAnalysis from an uploaded file

Now we have uploaded the file. Up next we now need create an InDepthAnalysis from our uploaded file. For creating the InDepthAnalyis via the Cyanite.ai API we use the inDepthAnalysisCreate mutation.

mutation inDepthAnalysisCreate($data: InDepthAnalysisCreateInput!) {
inDepthAnalysisCreate(data: $data) {
__typename
... on InDepthAnalysisCreateResultSuccess {
inDepthAnalysis {
id
status
}
}
... on Error {
message
}
}
}

The inDepthAnalysisFileUpload mutation field returns the union type InDepthAnalysisCreateResult. A union type specifies that the mutation can return a variety of possible results. Depending on the result we use InlineFragments for specifying our data requirements.

InDepthAnalysisCreateResultSuccess: The mutation finished like expected, we can query for the id, title, status or any other fields that belong to our InDepthAnalysis.

Error: An error occurred (e.g. InDepthAnalysisRecordLimitExceededError)

The Error interface type is implemented by all Errors. Instead of writing a SelectionSet for each error type we can use an InlineFragment on Error for gathering the error message. In case another error type provides some more detailed info that we want to use we can add additional SelectionSets. For now, we wanna keep things simple.

We also query for the type name (__typename). Each ObjectType (e.g. InDepthAnalysisCreateResultSuccess and InDepthAnalysisRecordLimitExceededError) have a __typename field. We use it for distinguishing between the single types.

Note: Despite we are selecting data on the interface Error, __typename can never be Error. An interface is an abstract type. However, all types that implement the Error interface will end with Error (e.g. InDepthAnalysisRecordLimitExceededError). In the integration code, we can therefore simply check whether __typename ends with Error.

src/file-upload.js

"use strict";
const envalid = require("envalid");
const fs = require("fs");
const fetch = require("node-fetch");
const { API_URL, ACCESS_TOKEN } = envalid.cleanEnv(process.env, {
API_URL: envalid.str(),
ACCESS_TOKEN: envalid.str()
});
const fileUploadRequestMutation = /* GraphQL */ `
mutation fileUploadRequest {
fileUploadRequest {
id
uploadUrl
}
}
`;
const inDepthAnalysisCreateMutation = /* GraphQL */ `
mutation inDepthAnalysisCreate($data: InDepthAnalysisCreateInput!) {
inDepthAnalysisCreate(data: $data) {
__typename
... on InDepthAnalysisCreateResultSuccess {
inDepthAnalysis {
id
status
}
}
... on Error {
message
}
}
}
`;
const requestFileUpload = async () => {
const result = await fetch(API_URL, {
method: "POST",
body: JSON.stringify({
query: fileUploadRequestMutation
}),
headers: {
Authorization: "Bearer " + ACCESS_TOKEN,
"Content-Type": "application/json"
}
}).then(res => res.json());
console.log("[info] fileUploadRequest response: ");
console.log(JSON.stringify(result, undefined, 2));
return result.data.fileUploadRequest;
};
const uploadFile = async (filePath, uploadUrl) => {
const result = await fetch(uploadUrl, {
method: "PUT",
body: fs.createReadStream(filePath),
headers: {
"Content-Type": fs.statSync(filePath).size
}
}).then(res => res.text());
console.log(result);
};
const createInDepthAnalysis = async fileUploadRequestId => {
const result = await fetch(API_URL, {
method: "POST",
body: JSON.stringify({
query: inDepthAnalysisCreateMutation,
variables: {
data: {
fileName: "My first InDepthAnalysis",
uploadId: fileUploadRequestId
}
}
}),
headers: {
Authorization: "Bearer " + ACCESS_TOKEN,
"Content-Type": "application/json"
}
}).then(res => res.json());
console.log("[info] inDepthAnalysisCreate response: ");
console.log(JSON.stringify(result, undefined, 2));
return result.data.inDepthAnalysisCreate;
};
const main = async filePath => {
const { id, uploadUrl } = await requestFileUpload(filePath);
await uploadFile(filePath, uploadUrl);
await createInDepthAnalysis(id);
};
main(process.argv[2]).catch(err => {
console.error(err);
process.exitCode = 1;
});

Let's execute this script for completing our first file upload to the Cyanite.ai API ๐Ÿš€

node src/file-upload.js

The terminal output should look similar to this:

n1ru4l@outerspace:~/cyanite-integration$ node src/file-upload.js "/Users/n1ru4l/Documents/music/2019_04_11/Jam Thieves & Voltage - LSD.mp3"
/Users/n1ru4l/Documents/music/2019_04_11/Jam Thieves & Voltage - LSD.mp3
[info] start transferring file
[info] inDepthAnalysisCreate response:
{
"data": {
"inDepthAnalysisCreate": {
"__typename": "InDepthAnalysisCreateResultSuccess",
"inDepthAnalysis": {
"id": "916",
"title": "My first InDepthAnalysis",
"status": "NOT_STARTED"
}
}
}
}

Analysis Status

As you can see the file is now successfully uploaded but it's status is NOT_STARTED.

The field InDepthAnalysis.status has the enum type AnalysisStatus. It can be either one of the following:

AnalysisStatusDescription
NOT_STARTEDFile was successfully uploaded
ENQUEUEDFile Analysis is enqueued and awaiting processing
PROCESSINGFile is being processed
FINISHEDFile Processing has finished successfully
FAILEDFile processing has failed

Enqueuing a file analysis

In order to process our uploaded file we need to send an additional mutation to the Cyanite.ai API. We can do this with the inDepthAnalysisEnqueueAnalysis mutation.

mutation inDepthAnalysisEnqueueAnalysis(
$input: InDepthAnalysisEnqueueAnalysisInput!
) {
inDepthAnalysisEnqueueAnalysis(data: $input) {
__typename
... on InDepthAnalysisEnqueueAnalysisResultSuccess {
success
inDepthAnalysis {
id
status
}
}
... on Error {
message
}
}
}

Like the inDepthAnalysisCreate mutation we also include InlineFragments for the expected result (InDepthAnalysisEnqueueAnalysisResultSuccess) and unexpected results (InDepthAnalysisNotFoundError, InDepthAnalysisLimitExceededError, InDepthAnalysisAlreadyEnqueuedError). The latter are ones are all covered by the Error Fragment.

src/file-enqueue-analysis.js

"use strict";
const envalid = require("envalid");
const fetch = require("node-fetch");
const { API_URL, ACCESS_TOKEN } = envalid.cleanEnv(process.env, {
API_URL: envalid.str(),
ACCESS_TOKEN: envalid.str()
});
const inDepthAnalysisEnqueueAnalysis = async inDepthAnalysisId => {
const mutationDocument = /* GraphQL */ `
mutation inDepthAnalysisEnqueueAnalysis(
$input: InDepthAnalysisEnqueueAnalysisInput!
) {
inDepthAnalysisEnqueueAnalysis(data: $input) {
__typename
... on InDepthAnalysisEnqueueAnalysisResultSuccess {
success
inDepthAnalysis {
id
status
}
}
... on Error {
message
}
}
}
`;
const result = await fetch(API_URL, {
method: "POST",
body: JSON.stringify({
query: mutationDocument,
variables: { input: { inDepthAnalysisId } }
}),
headers: {
Authorization: "Bearer " + ACCESS_TOKEN,
"Content-Type": "application/json"
}
}).then(res => res.json());
console.log("[info] inDepthAnalysisEnqueueAnalysis response: ");
console.log(JSON.stringify(result, undefined, 2));
if (result.data.inDepthAnalysisEnqueueAnalysis.__typename.endsWith("Error")) {
throw new Error(result.data.inDepthAnalysisFileUpload.message);
}
return result.data;
};
const main = async inDepthAnalysisId => {
await inDepthAnalysisEnqueueAnalysis(inDepthAnalysisId);
};
main(process.argv[2]).catch(err => {
console.error(err);
process.exitCode = 1;
});

Let's enqueue the file from earlier. Please note, that you must use the id of the file you have uploaded instead.

n1ru4l@outerspace:~/cyanite-integration$ node src/file-enqueue-analysis.js "916"
[info] inDepthAnalysisEnqueueAnalysis response:
{
"data": {
"inDepthAnalysisEnqueueAnalysis": {
"__typename": "InDepthAnalysisEnqueueAnalysisResultSuccess",
"success": true,
"inDepthAnalysis": {
"id": "916",
"status": "ENQUEUED"
}
}
}
}

We have now successfully uploaded and enqueued our first file analysis! Hurray ๐ŸŽ‰!

Listening to Webhook Events

The process of analyzing of our files is asyncronous, that means that there is no direct connection being kept alive until the file has finished processing. Instead of constantly polling the status of a file, the Cyanite.ai API allows registering a Webhook Endpoint that will be notified once a file has been processed/failed to process.

This approach is much more inituitive than sending unnecessary requests or keeping a connection alive.

Note: For your production integration you should have a server that is facing the public internet. For this demonstration we are going to use ngrok for exposing the local port of our machine to the internet.

Starting the ngrok proxy

npx ngrok http 8080

Forwarding http://xxxxxde.ngrok.io -> http://localhost:8080
Forwarding https://xxxxxde.ngrok.io -> http://localhost:8080

Open the integration section on Cyanite.ai and edit your integration by setting your webhook url http://b6d29bde.ngrok.io/incoming-webhook. Your url will slightly differ from the one above

Creating a webhook

Up next, lets finally writy our webhook code.

src/webhook.js

"use strict";
const crypto = require("crypto");
const envalid = require("envalid");
const express = require("express");
const bodyParser = require("body-parser");
const app = express();
const { PORT, SECRET } = envalid.cleanEnv(process.env, {
PORT: envalid.num(),
SECRET: envalid.str()
});
const isSignatureValid = (secret, signature, message) => {
const hmac = crypto.createHmac("sha512", secret);
hmac.write(message);
hmac.end();
const compareSignature = hmac.read().toString("hex");
return signature === compareSignature;
};
const WEBHOOK_ROUTE_NAME = "/incoming-webhook";
app.use(bodyParser.json());
app.post(WEBHOOK_ROUTE_NAME, (req, res) => {
if (!req.body) {
return res.sendStatus(422); // Unprocessable Entity
}
console.log("[info] incoming event:");
console.log(JSON.stringify(req.body, undefined, 2));
if (req.body.type === "TEST") {
console.log("[info] processing test event");
return res.sendStatus(200);
}
// verifying the request signature is not required but recommended
// by verifying the signature you can ensure the incoming request was sent by Cyanite.ai
if (
!isSignatureValid(SECRET, req.headers.signature, JSON.stringify(req.body))
) {
console.log("[info] signature is invalid");
return res.sendStatus(400);
}
console.log("[info] signature is valid");
if (req.body.type === "IN_DEPTH_ANALYSIS_FINISHED") {
console.log("[info] processing finish event");
// You can use the result here, but keep in mind that you should probably process the result asynchronously
// The request of the incoming webhook will be canceled after 3 seconds.
}
// Do something with the result here
return res.sendStatus(200);
});
app.listen(PORT, () => {
console.log(
`Server listening on http://localhost:${PORT}${WEBHOOK_ROUTE_NAME}`
);
});

Aftwerwards we can start the server (node src/webhook.js).

Receiving a webhook event

Before receiving any event, we first need to initiate an action that triggers an event. We can do that by re-enqueueing our previously uploaded file:

n1ru4l@outerspace:~/cyanite-integration$ node src/file-enqueue-analysis.js "916"
[info] inDepthAnalysisEnqueueAnalysis response:
{
"data": {
"inDepthAnalysisEnqueueAnalysis": {
"__typename": "InDepthAnalysisEnqueueAnalysisResultSuccess",
"success": true,
"inDepthAnalysis": {
"id": "916",
"status": "ENQUEUED"
}
}
}
}

After a few seconds the webhook terminal should output something similar to this:

n1ru4l@outerspace:~/cyanite-integration$ yarn start
yarn run v1.15.2
$ node src/webhook.js
Server listening on http://localhost:8080/incoming-webhook
[info] incoming event:
{
"type": "IN_DEPTH_ANALYSIS_FINISHED",
"data": {
"inDepthAnalysisId": "916"
}
}
[info] signature is valid
[info] processing finish event

That means our event was delivered successfully ๐Ÿš€!

Fetching the analysis result

Depending on what we need, we can fetch different parts of the analysis result. E.g. we could either query for similar sounding spotify tracks, the detected genre or both at the same time! You can explore all available fields on our GraphQL Playground.

For this demonstration we are going to fetch the analyzed genres.

GraphiQL
Query Variables
Please enter your integration access token

Feel free to explore the available data trough either GraphiQL or the GraphQL Playground.

Here you can find a more complex Query:

query inDepthAnalysis($inDepthAnalysisId: ID!) {
inDepthAnalysis(recordId: $inDepthAnalysisId) {
__typename
... on InDepthAnalysis {
id
status
result {
fileInfo {
duration
}
segments {
start
timestamp
duration
analysis {
arousal
valence
scores {
excited
angry
melancholic
relaxed
happy
sad
pumped
calm
}
}
}
labels {
title
type
start
end
amount
}
genres {
title
confidence
}
mainGenre
}
}
... on Error {
message
}
}
}

Conclusion

We sucessfully build our first Cyanite.ai Integration!

You can also find the whole project on Github

In case you have any open questions or suggestion on how we could improve this guide contact us via mail@cyanite.ai.