speechbrain.processing.diarization module

This script contains basic functions used for speaker diarization. This script has an optional dependency on open source sklearn library. A few sklearn functions are modified in this script as per requirement.

Reference

Authors
  • Nauman Dawalatabad 2020

Summary

Classes:

Spec_Clust_unorm

This class implements the spectral clustering with unnormalized affinity matrix.

Spec_Cluster

Functions:

distribute_overlap

Distributes the overlapped speech equally among the adjacent segments with different speakers.

do_spec_clustering

Performs spectral clustering on embeddings.

get_oracle_num_spkrs

Returns actual number of speakers in a recording from the ground-truth.

is_overlapped

Returns True if segments are overlapping.

merge_ssegs_same_speaker

Merge adjacent sub-segs from the same speaker.

prepare_subset_csv

Prepares csv for a given recording ID.

read_rttm

Reads and returns RTTM in list format.

spectral_clustering_sb

Performs spectral clustering.

spectral_embedding_sb

Returns spectral embeddings.

write_ders_file

Write the final DERs for individual recording.

write_rttm

Writes the segment list in RTTM format (A standard NIST format).

Reference

speechbrain.processing.diarization.read_rttm(rttm_file_path)[source]

Reads and returns RTTM in list format.

Parameters

rttm_file_path (str) – Path to the RTTM file to be read.

Returns

rttm – List containing rows of RTTM file.

Return type

list

speechbrain.processing.diarization.write_ders_file(ref_rttm, DER, out_der_file)[source]

Write the final DERs for individual recording.

Parameters
  • ref_rttm (str) – Reference RTTM file.

  • DER (array) – Array containing DER values of each recording.

  • out_der_file (str) – File to write the DERs.

speechbrain.processing.diarization.prepare_subset_csv(full_diary_csv, rec_id, out_csv_file)[source]

Prepares csv for a given recording ID.

Parameters
  • full_diary_csv (csv) – Full csv containing all the recordings

  • rec_id (str) – The recording ID for which csv has to be prepared

  • out_csv_file (str) – Path of the output csv file.

speechbrain.processing.diarization.is_overlapped(end1, start2)[source]

Returns True if segments are overlapping.

Parameters
  • end1 (float) – End time of the first segment.

  • start2 (float) – Start time of the second segment.

Returns

overlapped – True of segments overlapped else False.

Return type

bool

Example

from speechbrain.processing import diarization as diar diar.is_overlapped(5.5, 3.4) True diar.is_overlapped(5.5, 6.4) False

speechbrain.processing.diarization.merge_ssegs_same_speaker(lol)[source]

Merge adjacent sub-segs from the same speaker.

Parameters

lol (list of list) – Each list contains [rec_id, sseg_start, sseg_end, spkr_id].

Returns

new_lol – new_lol contains adjacent segments merged from the same speaker ID.

Return type

list of list

Example

from speechbrain.processing import diarization as diar lol=[[‘r1’, 5.5, 7.0, ‘s1’], [‘r1’, 6.5, 9.0, ‘s1’], [‘r1’, 8.0, 11.0, ‘s1’], [‘r1’, 11.5, 13.0, ‘s2’], [‘r1’, 14.0, 15.0, ‘s2’], [‘r1’, 14.5, 15.0, ‘s1’]] diar.merge_ssegs_same_speaker(lol) [[‘r1’, 5.5, 11.0, ‘s1’], [‘r1’, 11.5, 13.0, ‘s2’], [‘r1’, 14.0, 15.0, ‘s2’], [‘r1’, 14.5, 15.0, ‘s1’]]

speechbrain.processing.diarization.distribute_overlap(lol)[source]

Distributes the overlapped speech equally among the adjacent segments with different speakers.

Parameters

lol (list of list) – It has each list structure as [rec_id, sseg_start, sseg_end, spkr_id].

Returns

new_lol – It contains the overlapped part equally divided among the adjacent segments with different speaker IDs.

Return type

list of list

Example

from speechbrain.processing import diarization as diar lol = [[‘r1’, 5.5, 9.0, ‘s1’], [‘r1’, 8.0, 11.0, ‘s2’], [‘r1’, 11.5, 13.0, ‘s2’], [‘r1’, 12.0, 15.0, ‘s1’]] diar.distribute_overlap(lol) [[‘r1’, 5.5, 8.5, ‘s1’], [‘r1’, 8.5, 11.0, ‘s2’], [‘r1’, 11.5, 12.5, ‘s2’], [‘r1’, 12.5, 15.0, ‘s1’]]

speechbrain.processing.diarization.write_rttm(segs_list, out_rttm_file)[source]

Writes the segment list in RTTM format (A standard NIST format).

Parameters
  • segs_list (list of list) – Each list contains [rec_id, sseg_start, sseg_end, spkr_id].

  • out_rttm_file (str) – Path of the output RTTM file.

speechbrain.processing.diarization.get_oracle_num_spkrs(rec_id, spkr_info)[source]

Returns actual number of speakers in a recording from the ground-truth. This can be used when the condition is oracle number of speakers.

Parameters
  • rec_id (str) – Recording ID for which the number of speakers have to be obtained.

  • spkr_info (list) – Header of the RTTM file. Starting with SPKR-INFO.

Example

from speechbrain.processing import diarization as diar spkr_info = [‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.A <NA> <NA>’, ‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.B <NA> <NA>’, ‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.C <NA> <NA>’, ‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.D <NA> <NA>’, ‘SPKR-INFO ES2011b 0 <NA> <NA> <NA> unknown ES2011b.A <NA> <NA>’, ‘SPKR-INFO ES2011b 0 <NA> <NA> <NA> unknown ES2011b.B <NA> <NA>’, ‘SPKR-INFO ES2011b 0 <NA> <NA> <NA> unknown ES2011b.C <NA> <NA>’] diar.get_oracle_num_spkrs(‘ES2011a’, spkr_info) 4 diar.get_oracle_num_spkrs(‘ES2011b’, spkr_info) 3

speechbrain.processing.diarization.spectral_embedding_sb(adjacency, n_components=8, norm_laplacian=True, drop_first=True)[source]

Returns spectral embeddings.

Parameters
  • adjacency (array-like or sparse graph) – shape - (n_samples, n_samples) The adjacency matrix of the graph to embed.

  • n_components (int) – The dimension of the projection subspace.

  • norm_laplacian (bool) – If True, then compute normalized Laplacian.

  • drop_first (bool) – Whether to drop the first eigenvector.

Returns

embedding – Spectral embeddings for each sample.

Return type

array

Example

import numpy as np from speechbrain.processing import diarization as diar affinity = np.array([[1, 1, 1, 0.5, 0, 0, 0, 0, 0, 0.5], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0.5, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0.5, 0, 0, 0, 0, 0, 1, 1, 1, 1]]) embs = diar.spectral_embedding_sb(affinity, 3) # Notice similar embeddings print(np.around(embs , decimals=3)) [[ 0.075 0.244 0.285]

[ 0.083 0.356 -0.203] [ 0.083 0.356 -0.203] [ 0.26 -0.149 0.154] [ 0.29 -0.218 -0.11 ] [ 0.29 -0.218 -0.11 ] [-0.198 -0.084 -0.122] [-0.198 -0.084 -0.122] [-0.198 -0.084 -0.122] [-0.167 -0.044 0.316]]

speechbrain.processing.diarization.spectral_clustering_sb(affinity, n_clusters=8, n_components=None, random_state=None, n_init=10)[source]

Performs spectral clustering.

Parameters
  • affinity (matrix) – Affinity matrix.

  • n_clusters (int) – Number of clusters for kmeans.

  • n_components (int) – Number of components to retain while estimating spectral embeddings.

  • random_state (int) –

    A pseudo random number generator used by kmeans. n_init : int

    Number of time the k-means algorithm will be run with different centroid seeds.

Returns

labels – Cluster label for each sample.

Return type

array

Example

import numpy as np from speechbrain.processing import diarization as diar affinity = np.array([[1, 1, 1, 0.5, 0, 0, 0, 0, 0, 0.5], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0.5, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0.5, 0, 0, 0, 0, 0, 1, 1, 1, 1]]) labs = diar.spectral_clustering_sb(affinity, 3) # print (labs) # [2 2 2 1 1 1 0 0 0 0]

class speechbrain.processing.diarization.Spec_Cluster(n_clusters=8, *, eigen_solver=None, n_components=None, random_state=None, n_init=10, gamma=1.0, affinity='rbf', n_neighbors=10, eigen_tol=0.0, assign_labels='kmeans', degree=3, coef0=1, kernel_params=None, n_jobs=None, verbose=False)[source]

Bases: sklearn.cluster._spectral.SpectralClustering

perform_sc(X, n_neighbors=10)[source]

Performs spectral clustering using sklearn on embeddings.

Parameters
  • X (array (n_samples, n_features)) – Embeddings to be clustered.

  • n_neighbors (int) – Number of neighbors in estimating affinity matrix.

  • Reference

  • ---------

  • https (//github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/cluster/_spectral.py) –

class speechbrain.processing.diarization.Spec_Clust_unorm(min_num_spkrs=2, max_num_spkrs=10)[source]

Bases: object

This class implements the spectral clustering with unnormalized affinity matrix. Useful when affinity matrix is based on cosine similarities.

Von Luxburg, U. A tutorial on spectral clustering. Stat Comput 17, 395–416 (2007). https://doi.org/10.1007/s11222-007-9033-z

Example

from speechbrain.processing import diarization as diar clust = diar.Spec_Clust_unorm(min_num_spkrs=2, max_num_spkrs=10) emb = [[ 2.1, 3.1, 4.1, 4.2, 3.1], [ 2.2, 3.1, 4.2, 4.2, 3.2], [ 2.0, 3.0, 4.0, 4.1, 3.0], [ 8.0, 7.0, 7.0, 8.1, 9.0], [ 8.1, 7.1, 7.2, 8.1, 9.2], [ 8.3, 7.4, 7.0, 8.4, 9.0], [ 0.3, 0.4, 0.4, 0.5, 0.8], [ 0.4, 0.3, 0.6, 0.7, 0.8], [ 0.2, 0.3, 0.2, 0.3, 0.7], [ 0.3, 0.4, 0.4, 0.4, 0.7],] # Estimating similarity matrix sim_mat = clust.get_sim_mat(emb) print (np.around(sim_mat[5:,5:], decimals=3)) [[1. 0.957 0.961 0.904 0.966]

[0.957 1. 0.977 0.982 0.997] [0.961 0.977 1. 0.928 0.972] [0.904 0.982 0.928 1. 0.976] [0.966 0.997 0.972 0.976 1. ]]

# Prunning prunned_sim_mat = clust.p_pruning(sim_mat, 0.3) print (np.around(prunned_sim_mat[5:,5:], decimals=3)) [[1. 0. 0. 0. 0. ]

[0. 1. 0. 0.982 0.997] [0. 0.977 1. 0. 0.972] [0. 0.982 0. 1. 0.976] [0. 0.997 0. 0.976 1. ]]

# Symmetrization sym_prund_sim_mat = 0.5 * (prunned_sim_mat + prunned_sim_mat.T) print (np.around(sym_prund_sim_mat[5:,5:], decimals=3)) [[1. 0. 0. 0. 0. ]

[0. 1. 0.489 0.982 0.997] [0. 0.489 1. 0. 0.486] [0. 0.982 0. 1. 0.976] [0. 0.997 0.486 0.976 1. ]]

# Laplacian laplacian = clust.get_laplacian(sym_prund_sim_mat) print (np.around(laplacian[5:,5:], decimals=3)) [[ 1.999 0. 0. 0. 0. ]

[ 0. 2.468 -0.489 -0.982 -0.997] [ 0. -0.489 0.975 0. -0.486] [ 0. -0.982 0. 1.958 -0.976] [ 0. -0.997 -0.486 -0.976 2.458]]

# Spectral Embeddings spec_emb, num_of_spk = clust.get_spec_embs(laplacian, 3) print(num_of_spk) 3 # Clustering clust.cluster_embs(spec_emb, num_of_spk) # print (clust.labels_) # [0 0 0 2 2 2 1 1 1 1] # Complete spectral clustering clust.do_spec_clust(emb, k_oracle=3, p_val=0.3) # print(clust.labels_) # [0 0 0 2 2 2 1 1 1 1]

do_spec_clust(X, k_oracle, p_val)[source]

Function for spectral clustering.

Parameters
  • X (array) – (n_samples, n_features). Embeddings extracted from the model.

  • k_oracle (int) – Number of speakers (when oracle number of speakers).

  • p_val (float) – p percent value to prune the affinity matrix.

get_sim_mat(X)[source]

Returns the similarity matrix based on cosine similarities.

Parameters

X (array) – (n_samples, n_features). Embeddings extracted from the model.

Returns

M – (n_samples, n_samples). Similarity matrix with cosine similarities between each pair of embedding.

Return type

array

p_pruning(A, pval)[source]

Refine the affinity matrix by zeroing less similar values.

Parameters
  • A (array) – (n_samples, n_samples). Affinity matrix.

  • pval (float) – p-value to be retained in each row of the affinity matrix.

Returns

A – (n_samples, n_samples). Prunned affinity matrix based on p_val.

Return type

array

get_laplacian(M)[source]

Returns the un-normalized laplacian for the given affinity matrix.

Parameters

M (array) – (n_samples, n_samples) Affinity matrix.

Returns

L – (n_samples, n_samples) Laplacian matrix.

Return type

array

get_spec_embs(L, k_oracle=4)[source]

Returns spectral embeddings and estimates the number of speakers using maximum Eigen gap.

Parameters
  • L (array (n_samples, n_samples)) – Laplacian matrix.

  • k_oracle (int) – Number of speakers when the condition is oracle number of speakers, else None.

Returns

  • emb (array (n_samples, n_components)) – Spectral embedding for each sample with n Eigen components.

  • num_of_spk (int) – Estimated number of speakers. If the condition is set to the oracle number of speakers then returns k_oracle.

cluster_embs(emb, k)[source]

Clusters the embeddings using kmeans.

Parameters
  • emb (array (n_samples, n_components)) – Spectral embedding for each sample with n Eigen components.

  • k (int) – Number of clusters to kmeans.

Returns

self.labels_ – Labels for each sample embedding.

Return type

self

getEigenGaps(eig_vals)[source]

Returns the difference (gaps) between the Eigen values.

Parameters

eig_vals (list) – List of eigen values

Returns

eig_vals_gap_list – List of differences (gaps) between adjacent Eigen values.

Return type

list

speechbrain.processing.diarization.do_spec_clustering(diary_obj, out_rttm_file, rec_id, k, pval, affinity_type, n_neighbors)[source]

Performs spectral clustering on embeddings. This function calls specific clustering algorithms as per affinity.

Parameters
  • diary_obj (StatObject_SB type) – Contains embeddings in diary_obj.stat1 and segment IDs in diary_obj.segset.

  • out_rttm_file (str) – Path of the output RTTM file.

  • rec_id (str) – Recording ID for the recording under processing.

  • k (int) – Number of speaker (None, if it has to be estimated).

  • pval (float) – pval for prunning affinity matrix.

  • affinity_type (str) – Type of similarity to be used to get affinity matrix (cos or nn).