Large-scale inference with block structure
Abstract/Contents
- Abstract
- The detection of weak and rare effects in large amounts of data arises in a number of modern data analysis problems. Known results show that in this situation the potential of statistical inference is severely limited by the large-scale multiple testing that is inherent in these problems. Here we show that fundamentally more powerful statistical inference is possible when there is some structure in the signal that can be exploited, e.g. if the signal is clustered in many small blocks, as is the case in relevant applications. We derive the detection boundary in such a situation where we allow both the number of blocks and the block length to grow polynomially with sample size. This result recovers as special cases the heterogeneous mixture detection problem (1) where there is no structure in the signal, as well as scan problem (2) where the signal comprises a single interval. We develop methodology that allows optimal adaptive detection in the general setting, thus exploiting the structure if it is present without incurring a penalty in the case where there is no structure. The advantage of this methodology can be considerable, as in the latter case the means need to increase at the rate of square root of the log of the problem size to ensure detection, while in the former case the means may decrease at a polynomial rate. The identification version of this problem is also considered in this thesis, where the length of the block is allowed to grow polynomially with the sample size while the number of blocks is assumed to grow at most logarithmically with the sample size. This setting greatly generalizes previous results. The multivariate version of this problem is also considered, in which we try to identify the support of the rectangular signal(s) in the hyper-rectangle. A lower bound below which the identification is impossible is presented and an asymptotically optimal and computationally efficient procedure is proposed under Gaussian white noise. This signal identification problem is shown to have the same statistical difficulty as the corresponding detection problem, in the sense that whenever we can detect the signal, we can identify the support of the signal. We also discuss about the signal identification problem under the exponential family setting and the robust identification problem where the noise distribution is unspecified.
Description
| Type of resource | text |
|---|---|
| Form | electronic; electronic resource; remote |
| Extent | 1 online resource. |
| Publication date | 2017 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Associated with | Kou, Jiyao |
|---|---|
| Associated with | Stanford University, Department of Statistics. |
| Primary advisor | Walther, Guenther |
| Thesis advisor | Walther, Guenther |
| Thesis advisor | Lai, T. L |
| Thesis advisor | Siegmund, David |
| Advisor | Lai, T. L |
| Advisor | Siegmund, David |
Subjects
| Genre | Theses |
|---|
Bibliographic information
| Statement of responsibility | Jiyao Kou. |
|---|---|
| Note | Submitted to the Department of Statistics. |
| Thesis | Thesis (Ph.D.)--Stanford University, 2017. |
| Location | electronic resource |
Access conditions
- Copyright
- © 2017 by Jiyao Kou
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...