-
Bogomolov, M. and Heller, R.
Discovering findings that replicate from a primary study of high dimension to a follow-up study (2013)
Journal of the American Statistical Association, accepted.
-
abstract: We consider the problem of
identifying whether
findings replicate from one study of high dimension to another, when
the primary study guides the selection of hypotheses to be examined
in the follow-up study as well as when there is no division of roles
into the primary and the follow-up study. We show that existing
meta-analysis methods are not appropriate for this problem, and
suggest novel methods instead. We prove that our multiple testing
procedures control for appropriate error-rates.
The suggested FWER controlling procedure is valid for arbitrary dependence among the test statistics within each study. A more powerful procedure is suggested for FDR control. We prove that this procedure controls the FDR if the test statistics are independent within the primary study, and independent or have dependence of type PRDS in the follow-up study.
For arbitrary dependence within the primary study, and either arbitrary dependence or dependence of type PRDS in the follow-up study, simple conservative modifications of the procedure control the FDR. We demonstrate the usefulness of these
procedures via simulations and real data examples.
-
Paper: PDF file.
-
Heller, R. and Heller, Y. and Gorfine, M.
A consistent multivariate test of association based on ranks of
distances(2013)
Biometrika, Vol. 100, No. 2, Pp. 503-510.
-
abstract: We consider the detection of associations between random vectors of
any dimension. Few tests of independence exist that are consistent
against all dependent alternatives. We propose a powerful test that
is applicable in all dimensions and is consistent against all
alternatives. The test has a simple form, is easy to implement, and
has good power.
-
Paper: PDF file.
-
R Package: HHG
-
Heller, R. and Gorfine, M. and Heller Y.
A class of multivariate distribution-free tests of independence based on
graphs (2012)
Journal of Statistical Planning and Inference, Vol. 142, No. 12, Pp. 3097–3106.
-
abstract: A class of distribution-free tests is proposed for the independence of two
subsets of response coordinates. The tests are based on the pairwise distances
across subjects within each subset of the response. A complete graph is
induced by each subset of response coordinates, with the sample points
as nodes and the pairwise distances as the edge weights. The proposed test
statistic depends only on the rank order of edges in these complete graphs.
The response vector may be of any dimensions. In particular, the number
of samples may be smaller than the dimensions of the response. The test
statistic is shown to have a normal limiting distribution with known expectation
and variance under the null hypothesis of independence. The exact
distribution free null distribution of the test statistic is given for a sample of
size 14, and its Monte-Carlo approximation is considered for larger sample
sizes. We demonstrate in simulations that this new class of tests has good
power properties for very general alternatives.
-
Paper: PDF file.
-
Heller, R.
Discussion of “Multiple Testing for Exploratory Research” by J. J. Goeman and A. Solari (2012)
Statistical Science, Vol. 26, No. 4, Pp. 598-600.
-
abstract: Goeman and Solari [Statist. Sci.26(2011) 584–597] have
addressed the interesting topic of multiple testing for exploratory research, and
provided us with nice suggestions for exploratory analysis. They defined
properties that an inferential procedure should have for exploratory analysis:
the procedure should be mild, flexible and post hoc. Their inferential procedure
gives a lower bound on the number of false hypotheses among the
selected hypotheses, and moreover whenever possible identifies elementary
hypotheses that are false. The need to estimate a lower bound on the number
of false hypotheses arises in various applications, and the partial conjunction
approach was developed for this purpose in Biometrics 64(2008) 1215–1222
(see also Philos. Trans. R. Soc. Lond. Ser. A367(2009) 4255–4271 for more
details). For example, in a combined analysis of several studies that exam-ine the same problem,
it is of interest to give a lower bound on the number of studies in which the finding was reproduced.
I will first address the rela-tion between the method of Goeman and Solari and the partial conjunction
approach. Then I will discuss possible extensions and address the issue of ex-ploration in more
general settings, where the local test may not be defined in advance or where the candidate
hypotheses may not be known to begin with.
-
Paper: PDF file.
-
Heller, R.
Comment:Correlated z-values and the accuracy of large scale statistical
estimates(2010)
Journal of the American Statistical Association, Vol. 105, No. 491, Pp. 1057-1059.
-
abstract: Professor Efron has given us an interesting article on how to quantify the uncertainty
in summary statistics of interest in large scale problems, when the summary statistics
are based on correlated normal variates. It is shown that the inflation in the accuracy
estimate due to correlation among the normal variates cannot be ignored (except
possibly at the very far tails of distributions).
Using a series of simplifications of the covariance formula, a simple formula is derived
and it is shown in a numerical example that the approximation is indeed very close to
the truth. In particular it is shown that the entire correlation structure is captured
by one parameter ?, the rms correlation. Several methods of estimating ?, as well as
the other unknown parameters, are suggested.
In what follows I will discuss several topics in large scale significance testing that are
related to the results of this paper.
-
Paper: PDF file.
-
Heller, R. and Rosenbaum, P.R. and Small, D.S.
Using the cross-match test to appraise covariate balance in matched
pairs(2010)
The American Statistician, Vol. 64, No. 4, Pp. 299-309
-
abstract: Having created a tentative matched design for an observa-tional study,
diagnostic checks are performed to see whether
observed covariates exhibit reasonable balance, or alternatively
whether further effort is required to improve the match. We
illustrate the use of the cross-match test as an aid to appraising
balance on high-dimensional covariates, and we discuss its
close logical connections to the techniques used to construct
matched samples. In particular, in addition to a significance
level, the cross-match test provides an interpretable measure
of high-dimensional covariate balance, specifically a measure
defined in terms of the propensity score. An example from the
economics of education is used to illustrate. In the example,
imbalances in an initial match guide the construction of a better
match. The better match uses a recently proposed technique,
optimal tapered matching, that leaves certain possibly innocuous
covariates imbalanced in one match but not in another, and
yields a test of whether the imbalances are actually innocuous.
-
Paper: PDF file.
-
R Package: Crossmatch
-
Heller, R. and Jensen, S.T. and Rosenbaum, P.R. and Small, D.S.
Sensitivity Analysis for the Cross-Match Test, With Applications in
Genomics(2010)
Journal of the American Statistical Association, Vol. 105, No. 491, Pp. 1005-1013.
-
abstract: The cross-match test is an exact, distribution free test of no treatment e§ect
on a high dimensional outcome in a randomized experiment. The test uses optimal
nonbipartite matching to pair 2I subjects into I pairs based on similar outcomes, and
the cross-match statistic A is the number of times a treated subject was paired with a
control, rejecting for small values of A. If the test is applied in an observational study
in which treatments are not randomly assigned, it may be comparing treated and control
subjects who are not comparable, and may therefore falsely reject a true null hypothesis
of no treatment e§ect. We develop a sensitivity analysis for the cross-match test, and
apply it in an observational study of the e§ects of smoking on gene expression levels. In
addition, we develop a sensitivity analysis for several multiple testing procedures using the
cross-match test and apply it to 1627 molecular function categories in Gene Ontology.
-
Paper: PDF file.
-
R Package: Crossmatch
-
Benjamini, Y. and Heller, R. and Yekutieli, D.
Selective Inference in Complex Research(2009)
Philosophical Transactions of the Royal Society A, Vol. 367, No. 1906, Pp. 4255-4271
.
-
abstract: We explain the problem of selective inference in complex research using a recently
published study: a replicability study of the associations in order to reveal and establish
risk loci for type 2 diabetes. The false discovery rate approach to such problems will be
reviewed, and we further address two problems: (i) setting confidence intervals on the
size of the risk at the selected locations and (ii) selecting the replicable results.
-
Paper: PDF file.
-
Heller, R. and Manduchi, E. and Grant, G.R. and Ewens, W.J.
A flexible two-stage procedure for identifying gene sets that are
differentially expressed(2009)
Bioinformatics, Vol. 25, No. 8, Pp. 1019-1025.
-
abstract: Motivation: Microarray data analysis has expanded from testing
individual genes for differential expression to testing gene sets for
differential expression. The tests at the gene set level may focus on
multivariate expression changes or on the differential expression of
at least one gene in the gene set. These tests may be powerful at
detecting subtle changes in expression, but findings at the gene set
level need to be examined further to understand whether they are
informative and if so how.
Results: We propose to first test for differential expression at the
gene set level but then proceed to test for differential expression
of individual genes within discovered gene sets. We introduce the
overall FDR (OFDR) as an appropriate error rate to control when
testing multiple gene sets and genes. We illustrate the advantage of
this procedure over procedures that only test gene sets or individual
genes.
Availability: R code (www.r-project.org) for implementing our
approach is included as supplementary material.
-
Paper: PDF file.
-
R code: in Software section.
-
Heller, R. and Rosenbaum, P.R. and Small, D.S.
Split samples and design sensitivity in observational studies(2009)
Journal of the American Statistical Association, Vol. 104, No. 487, Pp. 1090-1101.
-
abstract: An observational or nonrandomized study of treatment effects may be biased
by failure to control for some relevant covariate that was not measured. The design of
an observational study is known to strongly affect its sensitivity to biases from covariates
that were not observed. For instance, the choice of an outcome to study, or the decision
to combine several outcomes in a test for coherence can materially affect the sensitivity
to unobserved biases. Decisions that shape the design are, therefore, critically important,
but they are also difficult decisions to make in the absence of data. We consider the
possibility of randomly splitting the data from an observational study into a smaller
planning sample and a larger analysis sample, where the planning sample is used to guide
decisions about design. After reviewing the concept of design sensitivity, we evaluate
sample splitting in theory, by numerical computation, and by simulation, comparing it to
several methods that use all of the data. Sample splitting is remarkably effective, much
more so in observational studies than in randomized experiments: splitting 1000 matched
pairs into 100 planning pairs and 900 analysis pairs often materially improves the design
sensitivity. An example from genetic toxicology is used to illustrate the method.
-
Paper: PDF file.
-
Heller, R. and Manduchi, E. and Small, D.S.
Matching methods for observational
microarray studies(2009)
Bioinformatics, Vol. 25, No. 7, Pp. 904-909.
-
abstract: Motivation:We address the problem of identifying differentially
expressed genes between two conditions in the scenario where the
data arise from anobservational study, in which confounding factors
are likely to be present.
Results:We suggest to use matching methods to balance two
groups of observed cases on measured covariates, and to identify
differentially expressed genes using a test suited to matched data.
We illustrate this approach on 2 microarray studies: the first study
consists of data from patients with two cancer subtypes, and the
second study consists of data from AMKL patients with and without
Down syndrome.
Availability: R code (www.r-project.org) for implementing our
approach is included as supplementary material.
-
Paper: PDF file.
-
R code: in Software section.
-
Benjamini, Y. and Heller, R.
Screening for partial conjunction hypotheses(2008)
Biometrics, Vol. 64, No. 4, Pp. 1215-1222.
-
abstract: We consider the problem of testing for partial conjunction of hypothesis, that
argues that at least u out of n tested hypotheses are false. It offers an in-between approach
to the testing of the conjunction of null hypotheses against the alternative that at least
one is not, and the testing of the disjunction of null hypotheses against the alternative that
all hypotheses are not null. We suggest powerful test statistics for testing such a partial
conjunction hypothesis that are valid under dependence between the test statistics as well
as under independence. We then address the problem of testing many partial conjunction
hypotheses simultaneously using the false discovery rate (FDR) approach. We prove that if
the FDR controlling procedure in Benjamini and Hochberg (1995) is used for this purpose
the FDR is controlled under various dependency structures. Moreover, we can screen at all
levels simultaneously in order to display the findings on a superimposed map and still control
an appropriate FDR measure. We apply the method to examples from Microarray analysis
and functional Magnetic Resonance Imaging (fMRI), two application areas where the need
for partial conjunction analysis has been identified.
-
Paper: PDF file.
-
Supplementary Material: PDF file.
-
Matlab code: in Software section.
-
Benjamini, Y. and Heller, R.
False Discovery Rates for Spatial Signals(2007)
Journal of the American Statistical Association, Vol. 102, No. 480, Pp. 1272-1281.
-
abstract: The problem of multiple testing for the presence of signal in spatial data
can involve a large number of locations. Traditionally, each location is tested
separately for signal presence but then the findings are reported in terms of
clusters of nearby locations. This is an indication that the units of interests
for testing are clusters rather than individual locations. The investigator may
know a-priori these more natural units or an approximation to them. We
suggest testing these cluster units rather than individual locations, thus increasing
the signal to noise ratio within the unit tested as well as reducing
the number of hypotheses tests conducted. Since the signal may be absent
from part of each cluster, we define a cluster as containing signal if the signal
is present somewhere within the cluster. We suggest controlling the false
discovery rate (FDR) on clusters, i.e. the expected proportion of clusters
rejected erroneously out of all clusters rejected, or its extension to general
weights (WFDR). We introduce a powerful two-stage testing procedure and
show that it controls the WFDR. Once the cluster discoveries have been made,
we suggest ’cleaning’ locations in which the signal is absent. For this purpose
we develop a hierarchical testing procedure that tests clusters first, then locations
within rejected clusters. We show formally that this procedure controls
the desired location error rate asymptotically, and conjecture that this is so
also for realistic settings by extensive simulations. We discuss an application
to functional neuroimaging which motivated this research and demonstrate
the advantages of the proposed methodology on an example.
-
Paper: PDF file.
-
Heller, R. and Golland, Y. and Malach, R. and Benjamini, Y.
Conjunction group analysis: An alternative to mixed/random effect
analysis(2007)
Neuroimage, Vol. 37, No. 4, Pp. 1178-1185.
-
abstract: We address the problem of testing in every brain voxelv whether at
least uout of n conditions (or subjects) considered shows a real effect.
The only statistic suggested so far, the maximump-value method, fails
under dependency (unless u=n) and in particular under positive
dependency that arises if all stimuli are compared to the same control
stimulus. Moreover, it tends to have low power under independence.
For testing that at leastuout ofnconditions shows a real effect, we
suggest powerful test statistics that are valid under dependence between
the individual conditionp-values as well as under independence
and other test statistics that are valid under independence. We use the
above approach, replacing conditions by subjects, to produce informative
group maps and thereby offer an alternative to mixed/random
effect analysis.
-
Paper: PDF file.
-
Heller, R. and Stanley, D. and Yekutieli, D. and Rubin, N. and Benjamini, Y.
Cluster-based analysis of FMRI data(2006)
NeuroImage, Vol. 33, No. 2, Pp. 599-608.
-
abstract: We propose a method for the statistical analysis of fMRIdata that
tests cluster units rather than voxel units for activation. The advantages of this analysis over previous
ones are both conceptual and statistical. Recognizing that the fundamental units of interest are the spatially
contiguous clusters of voxels that are activated together, we set out to approximate these cluster units
from the data by a clustering algorithm especially tailored for fMRIdata. Testing the cluster units has
a two-fold statistical advantage over testing each voxel separately: the signal to noise ratio within
the unit tested is higher, and the number of hypotheses tests compared is smaller. We suggest controlling
FDR on clusters, i.e., the proportion of clusters rejected erroneously out of all clusters rejected and
explain the meaning of controlling this error rate. We introduce the powerful adaptive procedure to control
the FDR on clusters. We apply our cluster-basedanalysis (CBA) to both an event-related and a block design
fMRI vision experiment and demonstrate its increased power over voxel-by-voxel analysis in these examples
as well as in simulations.
-
Paper: PDF file.
-
Matlab code: in Software section.