1 Glossary

1.1 NanoBRET experiment

NanoBRET: proximity-based assay that can detect protein interactions by measuring energy transfer from a bioluminescent protein donor to a fluorescent protein acceptor. Please find more information here.
RLU: Luminescence intensity of the protein donor. Samples with low (< 10000) RLU are discarded from the analysis.
RFU: Luminescence intensity of the protein acceptor.
MBU (milliBRET units): ratio of RFU to RLU multiplied by 1000. MBU is an ultimate readout of NanoBRET which serves as a proxy for intensity of protein-protein interactions. Almost all the analysis of NanoBRET data deals with MBU values.
PercentOfInteractions: Percent of MBU explained only by treatment. Is calculated as 100*(MBU(treated sample) - Mean(MBUs of relative controls))/MBU(treated sample)

1.2 Experimental Design and Sample Table categories

Sample table: table where each row represents a sample, and columns are the attributes assigned to the sample. Further the most important attributes are listed.
Plate: Id of the plate used for the sample preparation. In NanoBRET experiment each sample is prepared on a single well within 384-wells Plate.
Sample: Id of the well corresponding to the sample. In NanoBRET experiment each sample is prepared on a single well within 384-wells Plate.
Sector: Region on a plate (or scattered around several plates) which covers the treated samples and their respective negative-controls. Samples within a sector are assumed to vary only by the applied concentrations and have as little technical variation as possible.
Target: Name of the protein, interaction with which is estimated via NanoBRET analysis.
HibitTag: Name of the luminescent (donor) tag protein used in NanoBRET experiment.
HaloTag: Name of the fluorescent (acceptor) tag protein used in NanoBRET experiment.
Compound: Name of the compound added to a sample in order to enhance protein-protein interactions.
Concentration: concentration (typically in micromoles) of the applied Compound. Note, that it is set to 0 for the negative-control samples.
ConcentrationUnit: Unit of the concetration (typically uM).
Replicate group: Samples which are biological replicates. In NanoBRET these are the samples with the same Sector, Target, HibitTag, Compound and Concentration.
Specimen: Cell line (or tissue type) used in the experiment.
Positive Control Compound: Compound used as a positive control (required for QC purposes).
Treatment scheme: subset of NanoBRET data for a particular combination of target protein, its tag and applied compound. Treatment scheme is also can be viewed as a dose-response data, with dose being a concentration of a compound, and response being MBU values. Please note, that the corresponding negative-control samples are also part of the treatment scheme with concentration set to 0. Treatment schemes corresponds exactly to the samples from a single Sector, except the ones filtered out due to poor quality. Example of a treatment scheme is all the samples for a ‘GSPT1’ (Target) tagged with ‘CTER’ (HibitTag) and treated with ‘CC885’ (Compound) including all the corresponding negative control samples (Concentration of these control samples is set to 0).

1.3 NanoBRET input files

NanoBRET experiment creates a set(s) of the input files.

Compound management file: a csv file where each row is an annotation of a Well. Most importantly, compound applied to this well and its concentration.
Target: a csv file where each row is an annotation (Target and HibitTag) of a compound applied.
Raw data files: a txt files in a nanobret specific format. Each of this file carries the information of measured intensities (RLU and RFU) on an individual plate.

Please note, that a single study may contain mutliple sets of files described above.

2 Samples Filtering Strategy

2.1 QC metrics and Flags

QC metrics and Flags are assigned to the individual samples based on their experimental quality (RLU value) and their consistency (whether their MBU value is similar to the MBU values of the samples within the same replicate group). QC metrics and Flags serve two purposes. First, they allow to assess the quality of the experiment. Second, they give a way to use only high-quality samples for the compound validation (that is increase sensitivity and specificity of the validation process). Please note that the descriptions of the flags and metrics may be perceived as rather abstract. Therefore they are followed by an example section. Here is the list of quality metrics and flags used in NanoBRET:

2.1.1 QC metrics

MaxRatioMBU: Maximum MBU ratio for the given Sector (treatment scheme). That is out of all the concentrations, the one with highest mean MBU is selected (not necessarily with highest concentration). Then this highest mean MBU is divided by the mean MBU of the corresponding control samples.
Zfactor: Is set for the whole replicate group. It serves as a proxy of normalized (to the variation) distance between mean MBU values for this replicate group and corresponding negative controls. Zfactor displays our ability to detect the difference between treated and control samples. Formula: Zfactor = 1 - 3*(sd(MBU control) + sd(MBU treated)/ abs(mean(MBU treated) - mean(MBU control))
SpreadPlate: average intra-plate variation of MBU. That is for two replicates on the same plate this value is equal to ((MBU_1 - MEAN_MBU) + (MBU_2 - MEAN_MBU)) / MEAN_MBU. Then such a value is calculated for all the plates and finally averaged. This metric is used to access contribution of setling replicates on different plates.
Spread: variation of MBU accros all the plates. That is for four replicates on two plates plate this value is equal to ((MBU_1 - MEAN_MBU) + (MBU_2 - MEAN_MBU) + (MBU_3 - MEAN_MBU) + (MBU_4 - MEAN_MBU) ) / MEAN_MBU. This metric is used to access contribution of setling replicates on different plates.
SpreadRatio: ratio of Spread to SpreadPlate. Thus, it displays the mutliplier of the variation added by setling replicates on different plates

2.1.2 Filtering metrics and flags

FlagIntensity: Is set to ‘Y’ if sample RLU value is higher than a particular threshold (default 10000). This flag is used as an exclusion flag for the trend analysis.
DistanceToMean: An absolute difference between a sample MBU value and mean MBU (across the replicate group) normalized to the mean MBU (across the replicate group). For example: MBU value for a sample X = 20 and Mean mBU value of all the 4 replicates (that is samples treated exactly the sample as X) = 25, then DistanceToMean = |(20 - 25)|/25 = 0.2.
FlagDistance: Is set to ‘Y’ for a sample if this sample has an acceptable distance to mean MBU (across the replicate group). That is the distance is lower than a particular percentile (default 0.95) of all the samples in this study (except the ones which have FlagIntensity == 'N').
PassedDistances: Number of samples with acceptable distance to mean MBU (across the replicate group)
FlagKeepByDistance: Is set to ‘N’ for the sample with the highest distance to mean (if a replicate group has at least one sample with unacceptable distance) within a replicate group. Set to ‘Y’ otherwise. This flag should serve as an exclusion flag for the trend analysis.
FlagNotSingletonByDistance: Is set to ‘Y’ for the whole replicate group if the whole group has more than 1 sample with acceptable distance to mean.
FlagTrendAnalysis: Is set to ‘Y’, if `FlagIntensity == ‘Y’ & FlagKeepByDistance == ‘Y’ & FlagNotSingletonByDistance == ‘Y’. This flag defines whether a sample can be used in the trend analysis.

2.2 Examples:

Let’s assume that we have two plates with 648 samples in total. The exlusion will then include the following steps:

We exclude all the samples with RLU less than a particular threshold (that is we apply FlagIntensity) flag. In the following step we will work only with the remaining samples.
Then for all the remaining samples we calculate the distribution of their DistanceToMean values. Based on this distribution we take a particular percentile (default 0.95) and use it as a threshold. That is we filter out all the samples with 5% worst (largest) DistanceToMean values. However: there is an option to use a user-defined threshold instead of the distribution based, or even a logic combination of them both.
Assume we set the DistanceToMean threshold to 0.2 at the previous step and now we apply it to a replicate group. That is we assign a FlagDistance to the samples.

2.2.1 2 replicates fitered out

As we have samples with a DistanceToMean above the threshold:

The one with the highest DistanceToMean is excluded via FlagKeepByDistance.
As we have more than one sample passing the threshold, FlagNotSingletonByDistance is set to ‘Y’ for all the samples.
FlagTrendAnalysis is set to ‘N’ only to the sample with the highest DistanceToMean. That is this sample is excluded from the validation analysis

2.2.2 3 replicates fitered out

As we have samples with a DistanceToMean above the threshold:

The one with the highest DistanceToMean is excluded via FlagKeepByDistance.
As we have only one sample passing the threshold, FlagNotSingletonByDistance is set to ‘N’ for all the samples.
FlagTrendAnalysis is set to ‘N’ for all the samples as FlagNotSingletonByDistance is set to ‘N’. That is this replicate group is exlcuded from the validation analysis

3 Validation Strategy

3.1 Validation analyses

Validation aims to detect those compounds which have significant and consistent effect on the intensity of protein-protein interactions. That is, for a given treatment scheme MBU values monotonically increase with the increase of applied concentration. Therefore validation analysis utilizes dose-response analysis, monotonicity tests and means-difference significance tests. Please note, that only the samples with FlagTrendAnalysis == 'Y' are used for the validation analysis. Please note, that validation analysis runs independently for each treatment scheme.

The examples of validation analysis can be found in the end of this chapter

Validation analysis consists of the following parts:

Dose-Response Analysis: The EPA approved tcplfit2 package is used to run dose-response analysis. It provides the following results:
- model function: Function which has the best approximation to the experimental dose-response. We believe that NanoBRET dose response is modelled by exp4 or exp5 functions.
- BMR: Minimum desired response which is by default set to min(MBU control) + 10*sd(MBU control).
- BMD: Minimum concentration at which BMR is reached.
- BMDU: Upper confidence interval (0.95) for the BMD value. This value is a minimum concentration at which we expect the significant change of response.
- AC50: Concentration at which half of the activation (that is difference between minimum negative control MBU value and MBU value of the plateau at the highest concentrations) is reached.
Williams Trend Test: Checks for monotonical increase of MBU along the concentration range for each individual concentration. R implementation of Williams Trend Test does not give p-value(s), it rather provides the statistics values for each concentration along with their critical values (at alpha = 0.05). Therefore in order to get proxies for z-scores of Williams Trend Test we divide the statistics values to their critical values (WTTScore). Original publication, R function.
Jonckheere-Terpstra Test: Checks for monotonical increase of MBU along the concentration range for all the concentrations. Outputs single p-value, which is then directly used in the validation under the name JTTPValue. Original publication, R function
T-test: Checks if mean MBU for each of the concentration is larger than a negative control. It is possible to apply t-test as the NanoBRET MBU data generally follow t-test requirements. Please note, that multiple testing adjustment is also performed via Benjamini & Hochberg method. In validation analysis these adjusted t-test p-values are denoted as TTestPvalue.
Critical fold-change: Is a maximal significant fold-change of MBU between samples with a particular concentration and negative control samples. Significance here means that the p-value (unadjusted t-test, ‘greater’ hypothesis) for the difference between MBU(treated samples) and MBU(control samples) * fold_change is smaller than a particular threshold (0.05 by default).

3.2 Validation Scoring

There are two scores assigned to each treatment scheme:

ConfidenceScore: Displays the confidence that a particular compound enhances protein-protein interactions in coherent dose-response manner. This score is between 0 and 1, where 0 stands for a zero confidence and 1 for a full confidence.
- In order to calculate ConfidenceScore we first apply the thresholds to the aforementioned metrics (WTT scores, JTT p-value, t-test p-values and critical fold changes). The applied thresholds are ‘soft’, that is the result of their application is not only TRUE or FALSE, but rather lie within [0:1] interval. Thus for each of these metrics we get the scores (single value for JTT, and for the other metrics we have a set of values (one per concentration). Please note, that the ‘soft’ thresholding is done via parametrized logistic function.
- Applying a threshold to JTT p-value is rather trivial as it is a single number. For the other metrics we first define the minimum number of concentrations required to have a significant change in response (let’s call it thr_passed and set default to 2). Then for each concentration we take a weighted mean of these metrics and select top thr_passed of these means.
- Finally we take a weighted mean of the threshold-based scores (by default all the metrics have equal contributions) and assign it to ConfidenceScore.
InteractionScore: Defines a strength of a particular compound to enhance protein-protein interactions. As for ConfidenceScore we first define the number of concentrations to consider (thr_passed, default is 2), and take a mean of thr_passed critical fold-change values. This mean is a InteractionScore.

3.3 Validation Parameters

BMR Scale: minimum desired response (BMR) is set to min(MBU control) + bmr_scale*SD(MBU control). BMR defines at which response BMD value is calculated according to the best fitting model.
T-test critical value: maximum significance level (p-value) at which fold-change is calculated. That is fold change 1.5 at a concentration 1, means that MBU of this concentration is significantly higher than those of control with a p-value being set here.
Williams trend test threshold: threshold for WTT scores. The higher it is set, the stricter are requirements for monotonicity (coherent increase) of the dose-response.
Jonckheere-Terpstra test threshold: threshold for Jonckheere-Terpstra Test p-value, which tests montonicity. The lower it is, the stricter are requirements for coherent increase of the dose-response.
T-test threshold: threshold for T-Test p-value adjusted for multiple testing. The lower it is, the stricter are requirements for the stength of the dose-response.
Fold-change threshold: threshold for signinficant fold-change. The higher it is, the stricter are requirements for the stength of the dose-response
Number of Active Concentrations: Number of concentrations, which are supposed to have a substantial increase in MBU regulation compare to control. The higher the number, the stricter is validation process.

3.4 Examples

3.4.1 Clear Validation

This is an example of a strong and coherent dose-response.

We observe a monotonic increase of the MBU values with increase of the concentration, which results high WTT scores for all the concentrations and low JTT p-value. Thus this treatment scheme passes ‘monotonicity’ tests.
We observe a substantial increase of MBU values for all the concentrations compare to the control (concentration 0), which results in low T-test p-values and high significant Fold Change values. Thus this treatment scheme passes ‘regulation strength’ tests.

3.4.2 Only monotonicity

This is an example of a weak but coherent dose-response.

We observe an acceptably monotonic increase of the MBU values with increase of the concentration, which results high WTT scores for all the concentrations and low JTT p-value. Thus this treatment scheme passes ‘monotonicity’ tests.
We observe a small increase of MBU values for all the concentrations compare to the control (concentration 0), which results in rather high T-test p-values and small significant Fold Change values. Thus this treatment scheme barely passes ‘regulation strength’ tests.

3.4.3 Only strength

This is an example of a strong but incoherent dose-response.

We observe an increase of the MBU value only for the highest concentration, which results in generally low WTT scores and rather high JTT p-value. Thus this treatment scheme does not pass ‘monotonicity’ tests.
We observe a strong increase of MBU values for the highest concentration compare to the control (concentration 0), which results in rather high T-test p-value and small significant Fold Change value for the highest concentration. Thus this treatment scheme passes ‘regulation strength’ tests.

3.4.4 High variation

This is an example of a moderate and coherent dose-response which is spoiled by a high variation of MBU values.

Even though it seems that we have a clear dose-response, high variation of MBU values, especially of the control samples, worsens all the metrics for both ‘monotonicity’ and ‘regulation strength’ tests.

3.4.5 Clear Invalidation

This is an example of an absence of response.

This treatment scheme does not display any dose-response, which resluts in failing both ‘monotonicity’ and ‘regulation strength’ tests.