|
@@ -1,13 +1,13 @@
|
|
|
---
|
|
|
title: Supplementary Materials to Establishing the reliability and validity of measures extracted from long-form recordings
|
|
|
output:
|
|
|
- pdf_document:
|
|
|
- toc: yes
|
|
|
- toc_depth: 3
|
|
|
html_document:
|
|
|
toc: yes
|
|
|
toc_depth: '3'
|
|
|
df_print: paged
|
|
|
+ pdf_document:
|
|
|
+ toc: yes
|
|
|
+ toc_depth: 3
|
|
|
---
|
|
|
|
|
|
```{r setup, include=FALSE, eval=TRUE}
|
|
@@ -266,6 +266,7 @@ df.icc.mixed<-read.csv("../output/df.icc.mixed.csv")
|
|
|
|
|
|
#df.icc.mixed[df.icc.mixed$formula=="no_exp",c("data_set","metric")]
|
|
|
|
|
|
+df.icc.mixed$Type<-get_type(df.icc.mixed)
|
|
|
|
|
|
|
|
|
```
|
|
@@ -278,7 +279,6 @@ Out of the `r dim(df.icc.mixed)[1]` fitted models, `r table(df.icc.mixed$formula
|
|
|
|
|
|
```{r icc-allexp-fig3, echo=F,fig.width=4, fig.height=3,fig.cap="Distribution of ICC attributed to corpus (a) and children (b), when combining data from all corpora."}
|
|
|
|
|
|
-df.icc.mixed$Type<-get_type(df.icc.mixed)
|
|
|
|
|
|
|
|
|
ggplot(df.icc.mixed, aes(y = icc_child_id, x = toupper(data_set))) +
|
|
@@ -298,7 +298,7 @@ panel.background = element_blank(), legend.key=element_blank(), axis.line = elem
|
|
|
|
|
|
The majority of measures had ICCs between .3 and .5. `r sum(df.icc.mixed$icc_child_id > .5)` measures had higher ICCs, and surprisingly, `r sum(df.icc.mixed$icc_child_id[grep("och",df.icc.mixed$metric)] > .5)` of them corresponded to the "other child" category, known to have the worst accuracy according to previous analyses (Cristia et al., 2020).
|
|
|
|
|
|
-### Checking whether high ICC for other child measures are due to presence of other siblings
|
|
|
+### Checking whether high ICC for other child measures are due to number of siblings
|
|
|
|
|
|
```{r explo-och-sibn}
|
|
|
|
|
@@ -349,13 +349,30 @@ names(icc.result.split)=c("icc_child_id")
|
|
|
```
|
|
|
|
|
|
|
|
|
-We reasoned this may be because children in our corpora vary in terms of the number of siblings they have, and that siblings' presence may be stable across recordings. To address this possibility, we fit the full model again to predict number of vocalizations from other children, but this time including sibling number as a fixed effect $lmer(metric~ age + sibling_number + (1|corpus/child))$, so that individual variation that was actually due to sibling number was captured by that fixed effect instead of the random effect for child. We had sibling number data for `r sum(has_n_of_sib[,"TRUE"])` recordings from `r length(levels(factor(mydat2$child_id[!is.na(mydat2$n_of_siblings)])))` in `r length(levels(factor(mydat2$experiment[!is.na(mydat2$n_of_siblings)])))` corpora (`r corp_w_sib_clean`). We fit this model for the metric with the highest Child ICC, ACLEW's total vocalization duration by other children. Results indicated the full model was singular, so we fitted a No Corpus model to be able to extract a Child ICC. In fact, there was no difference in Child ICC in our original analysis (`r round(df.icc.mixed[df.icc.mixed$metric=="voc_dur_och_ph" & df.icc.mixed$data_set=="aclew","icc_child_id"],2)`) versus this re-analysis including the number of siblings (`r round(icc.result.split["icc_child_id"],2)`).
|
|
|
+We reasoned this may be because children in our corpora vary in terms of the number of siblings they have, that siblings' presence may be stable across recordings, and that a greater number of siblings would lead to more other child vocalizations. As a result, any measure based on other child vocalizations would result in stable relative ranking of children due to the number of siblings present. To test this hypothesis, we selected the metric with the highest Child ICC, namely ACLEW's total vocalization duration by other children. We fit the full model again to predict this metric, but this time, in addition to controlling for age, we included sibling number as a fixed effect $lmer(metric~ age + sibling_number + (1|corpus/child))$, so that individual variation that was actually due to sibling number was captured by that fixed effect instead of the random effect for child. We had sibling number data for `r sum(has_n_of_sib[,"TRUE"])` recordings from `r length(levels(factor(mydat2$child_id[!is.na(mydat2$n_of_siblings)])))` in `r length(levels(factor(mydat2$experiment[!is.na(mydat2$n_of_siblings)])))` corpora (`r corp_w_sib_clean`). The number of siblings varied from `r min(mydat2$n_of_siblings,na.rm=T)` to `r max(mydat2$n_of_siblings,na.rm=T)`, with a mean of `r round(mean(mydat2$n_of_siblings,na.rm=T),1)` and a median of `r round(median(mydat2$n_of_siblings,na.rm=T),1)`. Results indicated the full model was singular, so we fitted a No Corpus model to be able to extract a Child ICC. As a sanity check, we verified that the number of siblings predicted the outcome, total vocalization duration by other children -- and found that it did: ß = `r round(summary(model)$coefficients["n_of_siblings","Estimate"],2)`, t = `r round(summary(model)$coefficients["n_of_siblings","t value"],2)`, p < .001. This effect is relatively small: It means that per additional sibling, there is a .2 standard deviation increase in this variable. Turning now to how much variance is allocated to the random factor of Child, there was no difference in Child ICC in our original analysis (`r round(df.icc.mixed[df.icc.mixed$metric=="voc_dur_och_ph" & df.icc.mixed$data_set=="aclew","icc_child_id"],2)`) versus this re-analysis including the number of siblings (`r round(icc.result.split["icc_child_id"],2)`).
|
|
|
+
|
|
|
+### Checking whether high ICC for other child measures are due to presence of siblings
|
|
|
+
|
|
|
+```{r sib-presence}
|
|
|
+mydat2$sib_presence=ifelse(mydat2$n_of_siblings!=0,"present","absent")
|
|
|
+model_sib_presence<-lmer(voc_dur_och_ph~ age_s + sib_presence + (1|experiment/child_id),data=mydat2)
|
|
|
+
|
|
|
+#is sing
|
|
|
+model_sib_presence<-lmer(voc_dur_och_ph~ age_s +sib_presence + (1|child_id),data=mydat2)
|
|
|
+icc.result.split<- t(as.data.frame(icc(model_sib_presence, by_group=TRUE))$ICC)
|
|
|
+#names(icc.result.split)=c("icc_child_id", "icc_corpus")
|
|
|
+names(icc.result.split)=c("icc_child_id")
|
|
|
+
|
|
|
+
|
|
|
+```
|
|
|
|
|
|
|
|
|
-### Code to reproduce text before Table 3
|
|
|
+Perhaps it is not so much the sheer number of siblings that explains variance, but the sheer presence versus absence. After all, we can imagine that the effect of the number of siblings is not monotonic. We therefore repeated the analysis above but rather than adding the actual number of siblings, we had a binary variable that was "present" if the child had any siblings, and "absent" otherwise. As in the sibling number analysis, the full model was singular, so we fitted a No Corpus model to be able to extract a Child ICC. We again verified that sibling presence predicted the outcome, total vocalization duration by other children -- and found that it did: ß = `r round(summary(model_sib_presence)$coefficients["sib_presencepresent","Estimate"],2)`, t = `r round(summary(model_sib_presence)$coefficients["sib_presencepresent","t value"],2)`, p < .001. This effect is, as expected, sizable: It means that there is nearly one whole standard deviation increase in this variable when there are any siblings. In addition to being a better predictor, in this model, the amount of variance allocated to individual children as measured by Child ICC was considerably higher in our original analysis (`r round(df.icc.mixed[df.icc.mixed$metric=="voc_dur_och_ph" & df.icc.mixed$data_set=="aclew","icc_child_id"],2)`) than in this re-analysis including sibling presence (`r round(icc.result.split["icc_child_id"],2)`).
|
|
|
+
|
|
|
+
|
|
|
+## Code to reproduce text before Table 3
|
|
|
|
|
|
```{r reg model icc}
|
|
|
-#I moved this chunk here -- check that nothing is broken by it
|
|
|
|
|
|
lr_icc_chi <- lm(icc_child_id ~ Type * data_set, data=df.icc.mixed)
|
|
|
#binomial could be used, but diagnostic plots look pretty good (other than some outliers)
|
|
@@ -363,10 +380,16 @@ reg_sum=summary(lr_icc_chi)
|
|
|
|
|
|
reg_anova=Anova(lr_icc_chi)
|
|
|
|
|
|
+get_msd=function(x) paste0("M = ", round(mean(x),2),", SD = ", round(sd(x),2))
|
|
|
+
|
|
|
+msds = aggregate(df.icc.mixed$icc_child_id,by=list(df.icc.mixed$Type),get_msd)
|
|
|
+msds$x=gsub("0.",".",msds$x,fixed=T)
|
|
|
+rownames(msds)<-msds$Group.1
|
|
|
+
|
|
|
```
|
|
|
|
|
|
|
|
|
-Going back to our overarching analyses, we explored how similar Child ICCs were across different talker types and pipelines. We fit a linear model with the formula $lm(icc_child_id ~ type * pipeline)$, where type indicates whether the measure pertained to the key child, (female/male) adults, other children; and pipeline LENA or ACLEW. We found an adjusted R-squared of `r round(reg_sum$adj.r.squared*100)`%, suggesting much of the variance across Child ICCs was explained by this model. A Type 3 ANOVA on this model revealed only type was a signficant (F(`r reg_anova["Type","Df"]`)=`r round(reg_anova["Type","F value"],1)`, p<.001), whereas neither pipeline nor the interaction between type and pipeline were significant.
|
|
|
+Next we explored how similar Child ICCs were across different talker types and pipelines. We fit a linear model with the formula $lm(icc\_child\_id ~ type * pipeline)$, where type indicates whether the measure pertained to the key child, (female/male) adults, other children; and pipeline LENA or ACLEW. We found an adjusted R-squared of `r round(reg_sum$adj.r.squared*100)`%, suggesting much of the variance across Child ICCs was explained by these factors. A Type 3 ANOVA on this model revealed type was a signficant predictor (F(`r reg_anova["Type","Df"]`) = `r round(reg_anova["Type","F value"],1)`, p<.001), whereas as was pipeline (F(`r reg_anova["data_set","Df"]`) = `r round(reg_anova["data_set","F value"],1)`, p = `r round(reg_anova["data_set","Pr(>F)"],3)`); the interaction between type and pipeline was not significant. The main effect of type emerged because output metrics tended to have higher Child ICC (`r msds["Output","x"]`) than those associated to adults in general (`r msds["Adults","x"]`), females (`r msds["Female","x"]`), and males (`r msds["Male","x"]`); whereas those associated with other children had even higher Child ICCs (`r msds["Other children","x"]`).
|
|
|
|
|
|
|
|
|
|
|
@@ -392,7 +415,7 @@ kable(x,row.names = F,digits=2,caption="Most commonly used metrics.")
|
|
|
## Code to reproduce "Exploratory analyses: Correlations based on paired analyses of recordings done within two months"
|
|
|
|
|
|
|
|
|
-```{r correlation analysis for close recs, warning = F, echo=FALSE}
|
|
|
+```{r compute correlation analysis for close recs, warning = F, echo=FALSE}
|
|
|
#to compare with Gilkerson, we just need to focus on AWC, CVC, CTC (in LENA, but for comparison purposes, we also include aclew)
|
|
|
|
|
|
|
|
@@ -476,6 +499,12 @@ for(i in 1:nsamples){#i=1
|
|
|
}
|
|
|
|
|
|
|
|
|
+
|
|
|
+```
|
|
|
+
|
|
|
+```{r}
|
|
|
+
|
|
|
+
|
|
|
#summary(all_rs)
|
|
|
rval_tab=cbind(apply(all_rs[,1:20],1,mean),apply(all_rs[,1:20],1,sd),all_rs[,21:22])
|
|
|
colnames(rval_tab) <-c("m","sd","p","met")
|
|
@@ -493,15 +522,15 @@ colnames(mytab)<-c("aclew","lena")
|
|
|
rownames(mytab)<-c("AWC","CVC","CTC","Chi vocs")
|
|
|
print(mytab)
|
|
|
|
|
|
+rval_tab$Type<-get_type(rval_tab)
|
|
|
+
|
|
|
```
|
|
|
+
|
|
|
Out of our `r length(levels(factor(mydat_aclew$experiment)))` corpora and `r length(levels(factor(mydat_aclew$child_id)))` children, `r length(levels(factor(dist_contig_lena$child_id)))` children (belonging to `r length(levels(factor(gsub(" .*","",dist_contig_lena$child_id))))` corpora) could be included in this analysis, as some children did not have recordings less than two months apart (in particular, no child from the Warlaumont corpus did).
|
|
|
|
|
|
## Code to reproduce Fig. 4
|
|
|
|
|
|
```{r r-fig4, echo=F,fig.width=4, fig.height=3,fig.cap="Distribution of correlation coefficients."}
|
|
|
-#bug here probably inherited from above
|
|
|
-
|
|
|
-rval_tab$Type<-get_type(rval_tab)
|
|
|
|
|
|
|
|
|
ggplot(rval_tab, aes(y = m, x = toupper(p))) +
|
|
@@ -525,11 +554,14 @@ reg_sum_cor=summary(lr_cor)
|
|
|
|
|
|
reg_anova_cor=Anova(lr_icc_chi)
|
|
|
|
|
|
+r_msds=aggregate(rval_tab$m,by=list(rval_tab$p),get_msd)
|
|
|
+rownames(r_msds)<-r_msds$Group.1
|
|
|
+
|
|
|
cor_t=t.test(rval_tab$m ~ rval_tab$p)
|
|
|
|
|
|
```
|
|
|
|
|
|
-To see whether correlations in this analysis differed by talker types and pipelines, we fit a linear model with the formula $lm(cor ~ type * pipeline)$, where type indicates whether the measure pertained to the key child, (female/male) adults, other children; and pipeline LENA or ACLEW. We found an adjusted R-squared of `r round(reg_sum_cor$adj.r.squared*100)`%, suggesting this model did not explain a great deal of variance in correlation coefficients. A Type 3 ANOVA on this model revealed a significant effect of pipeline (F = `r round(reg_anova_cor["data_set","F value"],2)`, p = `r round(reg_anova_cor["data_set","Pr(>F)"],2)`), due to higher correlations for ACLEW (m = `r round(cor_t$estimate["mean in group aclew"],2)`) than for LENA metrics (m = `r round(cor_t$estimate["mean in group lena"],2)`). See below for fuller results.
|
|
|
+To see whether correlations in this analysis differed by talker types and pipelines, we fit a linear model with the formula $lm(cor ~ type * pipeline)$, where type indicates whether the measure pertained to the key child, (female/male) adults, other children; and pipeline LENA or ACLEW. We found an adjusted R-squared of `r round(reg_sum_cor$adj.r.squared*100)`%, suggesting this model did not explain a great deal of variance in correlation coefficients. A Type 3 ANOVA on this model revealed a significant effect of pipeline (F = `r round(reg_anova_cor["data_set","F value"],2)`, p = `r round(reg_anova_cor["data_set","Pr(>F)"],2)`), due to higher correlations for ACLEW (`r r_msds["aclew","x"]`) than for LENA metrics (m = `r r_msds["lena","x"]`). See below for fuller results.
|
|
|
|
|
|
```{r print out anova results rec on cor}
|
|
|
|