speechbrain.processing.diarization module¶
This script contains basic functions used for speaker diarization. This script has an optional dependency on open source sklearn library. A few sklearn functions are modified in this script as per requirement.
Reference¶
Von Luxburg, U. A tutorial on spectral clustering. Stat Comput 17, 395–416 (2007). https://doi.org/10.1007/s11222-007-9033-z
sklearn: https://github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/cluster/_spectral.py
- Authors
Nauman Dawalatabad 2020
Summary¶
Classes:
This class implements the spectral clustering with unnormalized affinity matrix. |
|
Functions:
Distributes the overlapped speech equally among the adjacent segments with different speakers. |
|
Performs spectral clustering on embeddings. |
|
Returns actual number of speakers in a recording from the ground-truth. |
|
Returns True if segments are overlapping. |
|
Merge adjacent sub-segs from the same speaker. |
|
Prepares csv for a given recording ID. |
|
Reads and returns RTTM in list format. |
|
Performs spectral clustering. |
|
Returns spectral embeddings. |
|
Write the final DERs for individual recording. |
|
Writes the segment list in RTTM format (A standard NIST format). |
Reference¶
-
speechbrain.processing.diarization.
read_rttm
(rttm_file_path)[source]¶ Reads and returns RTTM in list format.
-
speechbrain.processing.diarization.
write_ders_file
(ref_rttm, DER, out_der_file)[source]¶ Write the final DERs for individual recording.
-
speechbrain.processing.diarization.
prepare_subset_csv
(full_diary_csv, rec_id, out_csv_file)[source]¶ Prepares csv for a given recording ID.
-
speechbrain.processing.diarization.
is_overlapped
(end1, start2)[source]¶ Returns True if segments are overlapping.
- Parameters
- Returns
overlapped – True of segments overlapped else False.
- Return type
Example
from speechbrain.processing import diarization as diar diar.is_overlapped(5.5, 3.4) True diar.is_overlapped(5.5, 6.4) False
-
speechbrain.processing.diarization.
merge_ssegs_same_speaker
(lol)[source]¶ Merge adjacent sub-segs from the same speaker.
- Parameters
lol (list of list) – Each list contains [rec_id, sseg_start, sseg_end, spkr_id].
- Returns
new_lol – new_lol contains adjacent segments merged from the same speaker ID.
- Return type
list of list
Example
from speechbrain.processing import diarization as diar lol=[[‘r1’, 5.5, 7.0, ‘s1’], [‘r1’, 6.5, 9.0, ‘s1’], [‘r1’, 8.0, 11.0, ‘s1’], [‘r1’, 11.5, 13.0, ‘s2’], [‘r1’, 14.0, 15.0, ‘s2’], [‘r1’, 14.5, 15.0, ‘s1’]] diar.merge_ssegs_same_speaker(lol) [[‘r1’, 5.5, 11.0, ‘s1’], [‘r1’, 11.5, 13.0, ‘s2’], [‘r1’, 14.0, 15.0, ‘s2’], [‘r1’, 14.5, 15.0, ‘s1’]]
-
speechbrain.processing.diarization.
distribute_overlap
(lol)[source]¶ Distributes the overlapped speech equally among the adjacent segments with different speakers.
- Parameters
lol (list of list) – It has each list structure as [rec_id, sseg_start, sseg_end, spkr_id].
- Returns
new_lol – It contains the overlapped part equally divided among the adjacent segments with different speaker IDs.
- Return type
list of list
Example
from speechbrain.processing import diarization as diar lol = [[‘r1’, 5.5, 9.0, ‘s1’], [‘r1’, 8.0, 11.0, ‘s2’], [‘r1’, 11.5, 13.0, ‘s2’], [‘r1’, 12.0, 15.0, ‘s1’]] diar.distribute_overlap(lol) [[‘r1’, 5.5, 8.5, ‘s1’], [‘r1’, 8.5, 11.0, ‘s2’], [‘r1’, 11.5, 12.5, ‘s2’], [‘r1’, 12.5, 15.0, ‘s1’]]
-
speechbrain.processing.diarization.
write_rttm
(segs_list, out_rttm_file)[source]¶ Writes the segment list in RTTM format (A standard NIST format).
- Parameters
segs_list (list of list) – Each list contains [rec_id, sseg_start, sseg_end, spkr_id].
out_rttm_file (str) – Path of the output RTTM file.
-
speechbrain.processing.diarization.
get_oracle_num_spkrs
(rec_id, spkr_info)[source]¶ Returns actual number of speakers in a recording from the ground-truth. This can be used when the condition is oracle number of speakers.
- Parameters
Example
from speechbrain.processing import diarization as diar spkr_info = [‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.A <NA> <NA>’, ‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.B <NA> <NA>’, ‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.C <NA> <NA>’, ‘SPKR-INFO ES2011a 0 <NA> <NA> <NA> unknown ES2011a.D <NA> <NA>’, ‘SPKR-INFO ES2011b 0 <NA> <NA> <NA> unknown ES2011b.A <NA> <NA>’, ‘SPKR-INFO ES2011b 0 <NA> <NA> <NA> unknown ES2011b.B <NA> <NA>’, ‘SPKR-INFO ES2011b 0 <NA> <NA> <NA> unknown ES2011b.C <NA> <NA>’] diar.get_oracle_num_spkrs(‘ES2011a’, spkr_info) 4 diar.get_oracle_num_spkrs(‘ES2011b’, spkr_info) 3
-
speechbrain.processing.diarization.
spectral_embedding_sb
(adjacency, n_components=8, norm_laplacian=True, drop_first=True)[source]¶ Returns spectral embeddings.
- Parameters
adjacency (array-like or sparse graph) – shape - (n_samples, n_samples) The adjacency matrix of the graph to embed.
n_components (int) – The dimension of the projection subspace.
norm_laplacian (bool) – If True, then compute normalized Laplacian.
drop_first (bool) – Whether to drop the first eigenvector.
- Returns
embedding – Spectral embeddings for each sample.
- Return type
array
Example
import numpy as np from speechbrain.processing import diarization as diar affinity = np.array([[1, 1, 1, 0.5, 0, 0, 0, 0, 0, 0.5], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0.5, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0.5, 0, 0, 0, 0, 0, 1, 1, 1, 1]]) embs = diar.spectral_embedding_sb(affinity, 3) # Notice similar embeddings print(np.around(embs , decimals=3)) [[ 0.075 0.244 0.285]
[ 0.083 0.356 -0.203] [ 0.083 0.356 -0.203] [ 0.26 -0.149 0.154] [ 0.29 -0.218 -0.11 ] [ 0.29 -0.218 -0.11 ] [-0.198 -0.084 -0.122] [-0.198 -0.084 -0.122] [-0.198 -0.084 -0.122] [-0.167 -0.044 0.316]]
-
speechbrain.processing.diarization.
spectral_clustering_sb
(affinity, n_clusters=8, n_components=None, random_state=None, n_init=10)[source]¶ Performs spectral clustering.
- Parameters
affinity (matrix) – Affinity matrix.
n_clusters (int) – Number of clusters for kmeans.
n_components (int) – Number of components to retain while estimating spectral embeddings.
random_state (int) –
A pseudo random number generator used by kmeans. n_init : int
Number of time the k-means algorithm will be run with different centroid seeds.
- Returns
labels – Cluster label for each sample.
- Return type
array
Example
import numpy as np from speechbrain.processing import diarization as diar affinity = np.array([[1, 1, 1, 0.5, 0, 0, 0, 0, 0, 0.5], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [0.5, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0.5, 0, 0, 0, 0, 0, 1, 1, 1, 1]]) labs = diar.spectral_clustering_sb(affinity, 3) # print (labs) # [2 2 2 1 1 1 0 0 0 0]
-
class
speechbrain.processing.diarization.
Spec_Cluster
(n_clusters=8, *, eigen_solver=None, n_components=None, random_state=None, n_init=10, gamma=1.0, affinity='rbf', n_neighbors=10, eigen_tol=0.0, assign_labels='kmeans', degree=3, coef0=1, kernel_params=None, n_jobs=None, verbose=False)[source]¶ Bases:
sklearn.cluster._spectral.SpectralClustering
-
perform_sc
(X, n_neighbors=10)[source]¶ Performs spectral clustering using sklearn on embeddings.
- Parameters
X (array (n_samples, n_features)) – Embeddings to be clustered.
n_neighbors (int) – Number of neighbors in estimating affinity matrix.
Reference –
--------- –
https (//github.com/scikit-learn/scikit-learn/blob/0fb307bf3/sklearn/cluster/_spectral.py) –
-
-
class
speechbrain.processing.diarization.
Spec_Clust_unorm
(min_num_spkrs=2, max_num_spkrs=10)[source]¶ Bases:
object
This class implements the spectral clustering with unnormalized affinity matrix. Useful when affinity matrix is based on cosine similarities.
Von Luxburg, U. A tutorial on spectral clustering. Stat Comput 17, 395–416 (2007). https://doi.org/10.1007/s11222-007-9033-z
Example
from speechbrain.processing import diarization as diar clust = diar.Spec_Clust_unorm(min_num_spkrs=2, max_num_spkrs=10) emb = [[ 2.1, 3.1, 4.1, 4.2, 3.1], [ 2.2, 3.1, 4.2, 4.2, 3.2], [ 2.0, 3.0, 4.0, 4.1, 3.0], [ 8.0, 7.0, 7.0, 8.1, 9.0], [ 8.1, 7.1, 7.2, 8.1, 9.2], [ 8.3, 7.4, 7.0, 8.4, 9.0], [ 0.3, 0.4, 0.4, 0.5, 0.8], [ 0.4, 0.3, 0.6, 0.7, 0.8], [ 0.2, 0.3, 0.2, 0.3, 0.7], [ 0.3, 0.4, 0.4, 0.4, 0.7],] # Estimating similarity matrix sim_mat = clust.get_sim_mat(emb) print (np.around(sim_mat[5:,5:], decimals=3)) [[1. 0.957 0.961 0.904 0.966]
[0.957 1. 0.977 0.982 0.997] [0.961 0.977 1. 0.928 0.972] [0.904 0.982 0.928 1. 0.976] [0.966 0.997 0.972 0.976 1. ]]
# Prunning prunned_sim_mat = clust.p_pruning(sim_mat, 0.3) print (np.around(prunned_sim_mat[5:,5:], decimals=3)) [[1. 0. 0. 0. 0. ]
[0. 1. 0. 0.982 0.997] [0. 0.977 1. 0. 0.972] [0. 0.982 0. 1. 0.976] [0. 0.997 0. 0.976 1. ]]
# Symmetrization sym_prund_sim_mat = 0.5 * (prunned_sim_mat + prunned_sim_mat.T) print (np.around(sym_prund_sim_mat[5:,5:], decimals=3)) [[1. 0. 0. 0. 0. ]
[0. 1. 0.489 0.982 0.997] [0. 0.489 1. 0. 0.486] [0. 0.982 0. 1. 0.976] [0. 0.997 0.486 0.976 1. ]]
# Laplacian laplacian = clust.get_laplacian(sym_prund_sim_mat) print (np.around(laplacian[5:,5:], decimals=3)) [[ 1.999 0. 0. 0. 0. ]
[ 0. 2.468 -0.489 -0.982 -0.997] [ 0. -0.489 0.975 0. -0.486] [ 0. -0.982 0. 1.958 -0.976] [ 0. -0.997 -0.486 -0.976 2.458]]
# Spectral Embeddings spec_emb, num_of_spk = clust.get_spec_embs(laplacian, 3) print(num_of_spk) 3 # Clustering clust.cluster_embs(spec_emb, num_of_spk) # print (clust.labels_) # [0 0 0 2 2 2 1 1 1 1] # Complete spectral clustering clust.do_spec_clust(emb, k_oracle=3, p_val=0.3) # print(clust.labels_) # [0 0 0 2 2 2 1 1 1 1]
-
get_sim_mat
(X)[source]¶ Returns the similarity matrix based on cosine similarities.
- Parameters
X (array) – (n_samples, n_features). Embeddings extracted from the model.
- Returns
M – (n_samples, n_samples). Similarity matrix with cosine similarities between each pair of embedding.
- Return type
array
-
p_pruning
(A, pval)[source]¶ Refine the affinity matrix by zeroing less similar values.
- Parameters
A (array) – (n_samples, n_samples). Affinity matrix.
pval (float) – p-value to be retained in each row of the affinity matrix.
- Returns
A – (n_samples, n_samples). Prunned affinity matrix based on p_val.
- Return type
array
-
get_laplacian
(M)[source]¶ Returns the un-normalized laplacian for the given affinity matrix.
- Parameters
M (array) – (n_samples, n_samples) Affinity matrix.
- Returns
L – (n_samples, n_samples) Laplacian matrix.
- Return type
array
-
get_spec_embs
(L, k_oracle=4)[source]¶ Returns spectral embeddings and estimates the number of speakers using maximum Eigen gap.
- Parameters
L (array (n_samples, n_samples)) – Laplacian matrix.
k_oracle (int) – Number of speakers when the condition is oracle number of speakers, else None.
- Returns
emb (array (n_samples, n_components)) – Spectral embedding for each sample with n Eigen components.
num_of_spk (int) – Estimated number of speakers. If the condition is set to the oracle number of speakers then returns k_oracle.
-
-
speechbrain.processing.diarization.
do_spec_clustering
(diary_obj, out_rttm_file, rec_id, k, pval, affinity_type, n_neighbors)[source]¶ Performs spectral clustering on embeddings. This function calls specific clustering algorithms as per affinity.
- Parameters
diary_obj (StatObject_SB type) – Contains embeddings in diary_obj.stat1 and segment IDs in diary_obj.segset.
out_rttm_file (str) – Path of the output RTTM file.
rec_id (str) – Recording ID for the recording under processing.
k (int) – Number of speaker (None, if it has to be estimated).
pval (float) – pval for prunning affinity matrix.
affinity_type (str) – Type of similarity to be used to get affinity matrix (cos or nn).