Questions regarding reviewer comments on my systematic review regarding cognition & working memory..

[Doctor_Strange]

Research question: Evaluate and synthesize randomized controlled trial (RCT) study designs that investigated the effects of physical activity on working memory performance in healthy individuals.

Main inclusion criteria stipulated (1) a healthy sample population, (2) a physical activity intervention, (3) a working memory outcome measured, and (4) a randomized control trial design.

-------------------------------------------------------------------------------------------------------------------------------
(I graduated since submitting this journal and my ability to contact my research adviser is intermittent as she is on sabbatical, so I was hoping to get some additional insight from this thread--hope that is permissible)

Questions:

1) One reviewer commented that including acute and chronic exercise interventions must be separately analyzed to which I now realize and fully agree. My question is that I only have 11 studies and 5 are acute exercise interventions. Should I simply exclude them (per reviewer's recommendation), or perhaps attempt to conduct a separate meta-analysis of the 5 acute exercise-based studies? Doing so, however, may be problematic as each of the 5 studies used different working memory measurement tools (my initial meta-analysis has subgroups where identical tests are analyzed together).

2) This question is also directly relates to my #1: by removing the 5 studies from my initial meta-analyses, some subgroups, which initially had 4 studies (3 of which were acute, for instance) now become 1 study in said subgroup. So, is there a way to resolve this new problem? Looking at previous reviews, it seems one review in particular employed a Hedges' g test that seemingly allowed for the mixing of different working memory tests (again my understanding of what Hedges' g actually is is poor to say the least).

3) Does having just 6 studies in my review (if I in fact simply exclude acute exercise intervention) limit its appeal or worthiness to the scientific community (my search strategy was very specific to begin with as I only found 11 articles over a six year span to begin with)? Currently, I do not want to exclude them, rather try to perform separate analyses.

4) Difference between moderator and mediator variables? Online sources are confusing me. I want to measure the effect of age on my variables (which is defined as a mediator) and also duration (which is defined by some as a moderator).

5) Do I utilize a sensitivity analysis for #4 or a correlational analysis or even a regression meta analysis? I used RevMan, and such additional tests would require further reading on my end.

---

Members don't see this ad.

 

[temppsych123]

If you're working with 11 studies, I would keep your current analyses as they are and then run additional analyses comparing acute and chronic interventions as two groups to meta-analyze. Either they'll show similar patterns in which case, awesome, quick addition, you addressed the reviewer's comments or they show different patterns and it's something you focus on in the discussion as areas to expand for future research.

Hedges g and Cohen's d are just different measures of effect size, I don't think any effect size measure will be more or less robust to inclusion of different types of measures within a meta-analysis, since that's a substantive question rather than a statistical one.

6 studies is preeeeeetty small for a meta-analysis, I guess it would depend on what journal you're aiming for? But personally if I were working with that small of a sample I'd do more of a narrative review than a meta-analysis, because the variables you'd want to consider as confounds (varying sample characteristics, varying WM measures, etc.) you won't really have the power to control for statistically, but you can talk about them in more detail in a systematic review format.

Moderator = something that changes the relationship between two variables. If A and B are related overall, and C is a moderator, then the relationship between A and B will be different at high/low levels of C or presence/absence of C. For example, maybe the relationship between A and B is significantly positive at high levels of C, but non-significant at low levels of C, or positive at high levels of C and negative at low levels of C. This would be your classic statistical interaction. A mediator is something that explains the relationship between 2 variables such that, when you control for it, the relationship between those variables either goes away or is significantly diminished (partial versus full mediation). So let's say gender is correlated with weight, but if you control for height, this correlation goes away; the relationship between gender and weight is explained (partially or fully depending on the statistical results) by gender differences in height.

Not sure about #5.

Hope that helps, OP!

 

[Justanothergrad]

I can speak a little to the meta-analysis issues:

Hedge's g offers a unbiased estimate of Cohen's d. It actually uses Cohen's d in its calculation and it is possible to translate between the two fairly easily. The difference is how it values sample size of the included studies as a consideration in effect calculation (Cohen's D does not, G does using the inverse of the weight). As temp said, neither will differ in how measures impact it since the difference is focused on sample size.

Instead of excluding studies as the reviewer suggests, I would encourage you to conduct a Random-Effects Model meta-analysis. I'm unsure your methods obviously, but I suspect it's a fixed effect model that you have used. If you are using a Random Effects model then you can code a moderator (0=Chronic, 1= acute) to evaluate differences between the studies. In other words, based on your coding of the studies, is there patterns of difference in effect size between the studies? As a result of this, it will report estimated effect sizes for each of those groups and this will satisfy your reviewers request. It also tells you if the effects are significantly different which offers a strong point of reference for future studies. In general, random-effects models are the preferred approach to meta-analysis and if there is no evidence of heterogeneity to suggest a random effects model (i.e., significant Q), you have calculated the Fixed-effects as part of the process and can report that. A sample of 5 or 6 is insufficient to conduct a meta-analysis on in its own right; 11 is on the small side but might still be of value (I'm unfamiliar with the area of research enough to say). This approach of using a random-effects model addresses both the reviewers desire to exclude that data, the statistical reality of meta-analysis, and best practices (PRISMA group)

As a side note for using a moderator analysis: If you are using subgroup/moderator comparison (e.g., one of the main advantages of a Random effects model and what would happen if you took the approach described above) for your meta-analysis and you have less than 5 studies per group/class, use the overall Tau-squared for the calculation instead of the moderator based Tau-squared. Although the purpose of the moderator specific Tau-estimate is to generate a sample specific estimation of variance, its likely that the restricted sample size will lead to more inaccuracy than using the overall meta's estimate of variance (e.g., Borenstein et al., 2009)

 

[Doctor_Strange]

If you're working with 11 studies, I would keep your current analyses as they are and then run additional analyses comparing acute and chronic interventions as two groups to meta-analyze. Either they'll show similar patterns in which case, awesome, quick addition, you addressed the reviewer's comments or they show different patterns and it's something you focus on in the discussion as areas to expand for future research.

Hedges g and Cohen's d are just different measures of effect size, I don't think any effect size measure will be more or less robust to inclusion of different types of measures within a meta-analysis, since that's a substantive question rather than a statistical one.

6 studies is preeeeeetty small for a meta-analysis, I guess it would depend on what journal you're aiming for? But personally if I were working with that small of a sample I'd do more of a narrative review than a meta-analysis, because the variables you'd want to consider as confounds (varying sample characteristics, varying WM measures, etc.) you won't really have the power to control for statistically, but you can talk about them in more detail in a systematic review format.

Moderator = something that changes the relationship between two variables. If A and B are related overall, and C is a moderator, then the relationship between A and B will be different at high/low levels of C or presence/absence of C. For example, maybe the relationship between A and B is significantly positive at high levels of C, but non-significant at low levels of C, or positive at high levels of C and negative at low levels of C. This would be your classic statistical interaction. A mediator is something that explains the relationship between 2 variables such that, when you control for it, the relationship between those variables either goes away or is significantly diminished (partial versus full mediation). So let's say gender is correlated with weight, but if you control for height, this correlation goes away; the relationship between gender and weight is explained (partially or fully depending on the statistical results) by gender differences in height.

Not sure about #5.

Hope that helps, OP!

Thank you very much for your reply and insight--absolutely it does help.

First, I ended up running two separate analyses showing that the chronic physical activity significantly affected working memory outcomes in contrast to acute physical activity which had a non-significant result.

To your first point about my sample size of studies, the reviewer's at least did not comment on the fact I had 11 to begin with. I only looked from 2009 to 2015 and for such a stringent search strategy I am hopefully the reviewers and editorial board deem it appropriate (I submitted to Systematic Reviews an open-access journal). But to your underlying point, I may need to further express the caution that must be recognized when interpreting these new results in my discussion section. I will make a point to do that during the revisions.

Thank for the distinction between moderator and mediator--very well explained and easy to distinguish. At the request of one of the reviewer's, testing age as a mediator was recommended. I will have to look into this as I am not sure how exactly to run a mediation analysis.

 

[Doctor_Strange]

I can speak a little to the meta-analysis issues:

Hedge's g offers a unbiased estimate of Cohen's d. It actually uses Cohen's d in its calculation and it is possible to translate between the two fairly easily. The difference is how it values sample size of the included studies as a consideration in effect calculation (Cohen's D does not, G does using the inverse of the weight). As temp said, neither will differ in how measures impact it since the difference is focused on sample size.

Instead of excluding studies as the reviewer suggests, I would encourage you to conduct a Random-Effects Model meta-analysis. I'm unsure your methods obviously, but I suspect it's a fixed effect model that you have used. If you are using a Random Effects model then you can code a moderator (0=Chronic, 1= acute) to evaluate differences between the studies. In other words, based on your coding of the studies, is there patterns of difference in effect size between the studies? As a result of this, it will report estimated effect sizes for each of those groups and this will satisfy your reviewers request. It also tells you if the effects are significantly different which offers a strong point of reference for future studies. In general, random-effects models are the preferred approach to meta-analysis and if there is no evidence of heterogeneity to suggest a random effects model (i.e., significant Q), you have calculated the Fixed-effects as part of the process and can report that. A sample of 5 or 6 is insufficient to conduct a meta-analysis on in its own right; 11 is on the small side but might still be of value (I'm unfamiliar with the area of research enough to say). This approach of using a random-effects model addresses both the reviewers desire to exclude that data, the statistical reality of meta-analysis, and best practices (PRISMA group)

As a side note for using a moderator analysis: If you are using subgroup/moderator comparison (e.g., one of the main advantages of a Random effects model and what would happen if you took the approach described above) for your meta-analysis and you have less than 5 studies per group/class, use the overall Tau-squared for the calculation instead of the moderator based Tau-squared. Although the purpose of the moderator specific Tau-estimate is to generate a sample specific estimation of variance, its likely that the restricted sample size will lead to more inaccuracy than using the overall meta's estimate of variance (e.g., Borenstein et al., 2009)

Thank you very much for your insight and reply! Of my 11 studies, none are smaller than 20 (the range is 30 to 120 between all studies). To your first point, my meta-analysis was conducted using a random-effects model in fact (how I exactly code in moderators is something I will need to read up on regarding Rev Man).

As I responded to temp, I hope my sample size does not becoming an immovable obstacle and my intuition says it may not since there was no comment on it in the initial review. Having two meta analyses of 5 and 6 studies is another story, and I will leave it up to the reviewers. Suffice it to say, I will be stating clearly and explicitly to interpret such analyses with caution--I still believe they are of value to include in my SR.

To your last paragraph, much of what you say will require further reading since I am a novice regarding the information you provided.

I think, and if you and @temppsych123 would be available and willing, I am going to post shortly my initial meta-analysis and two analyses of where chronic and acute physical activity have been divided. Perhaps then any further commentary can be more focused. Again, I cannot overstate how appreciative I am of both of your responses and help already given--it is making my first endeavor in peer-review less stressful!

 

[Doctor_Strange]

Initial plot submitted with manuscript

edit: graph deleted

 

[Doctor_Strange]

Chronic physical activity plot and acute physical activity plots now separately analyzed

edit: graph deleted

 

[Doctor_Strange]

And update, RevMan (the program I use to conduct my meta-analyses) does in fact use Hedges g per the handbook: "The one implemented in RevMan is Hedges’ adjusted g, which is very similar to Cohen's d, but includes an adjustment for small sample bias"

 

[DynamicDidactic]

And update, RevMan (the program I use to conduct my meta-analyses) does in fact use Hedges g per the handbook: "The one implemented in RevMan is Hedges’ adjusted g, which is very similar to Cohen's d, but includes an adjustment for small sample bias"

I am not sure what the question is but Hedges g is indeed a measure of effect size controlling for small sample bias. And you should indeed be using it since you seem to have small sample size studies.

Sometimes reviewers driver me bonkers, you definitely should not exclude studies or run two separate meta-analyses. As mentioned above, a moderation analysis would be most efficient. The subplots look stunning.

Also, did you do this as an undergrad? If so, kudos to you, this is very much graduate student caliber. I hope you continue to stay involved in research through your medical school training.

 

[Justanothergrad]

Agreed with DynamicDidactic. This is an impressive undertaking, both as a undergrad (VERY impressive) and a grad student because, even then, meta-analysis is a lot of work. I highly recommend the Borenstein book as a resource for learning about meta-analysis. Its very digestible and readable. It will help bring to life a lot of the processes underlying meta-analysis. I use excel for meta (its a pain, but it helps me know where the numbers come from) but I see the appeal of RevMan for sure.

As I said before, your total sample size (k, or the number of included studies) is insufficient to divide it into a separate meta-analysis. Typically you will want at least 14-17 (I've seen people vary, but in that range) to conduct a meta-analysis just to ensure you are getting a broad enough sample of the random effects. It appears from your table, and I would need to see the methods to provide a better insight, that your analysis of differences were conducted across several measures (digit forward, digit backwards, etc.) and that each of those analyses contained only 4-5 studies. I do see where the reviewer would want them separate because it appears some studies only evaluated certain measures (Nback) in one of the groups. However, I don't see that as needing separate measures, you know by virtue of the studies selected that the effect size will relate only to that topic.

As for moderators, conducting a random effects model without including moderators excludes the greatest strength of such an approach.If you look at the Random effects results you'll see a Total Q (homogeneity) and this will tell you that there is variability between the studies. This variability is what the moderators attempt to resolve. Given the limitations of your sample size and the apparent structure of the studies (such that chronic tests certain outcomes and acute tests other outcomes), this might be hard to evaluate.

whoop. gotta run. I'll get back to this later.

 

[Doctor_Strange]

I am not sure what the question is but Hedges g is indeed a measure of effect size controlling for small sample bias. And you should indeed be using it since you seem to have small sample size studies.

Sometimes reviewers driver me bonkers, you definitely should not exclude studies or run two separate meta-analyses. As mentioned above, a moderation analysis would be most efficient. The subplots look stunning.

Also, did you do this as an undergrad? If so, kudos to you, this is very much graduate student caliber. I hope you continue to stay involved in research through your medical school training.

Well, reviewers have the keys to the pearly gates it would seem! But the two who reviewed mine actually had very keen eyes as they pointed out some important mistakes which has greatly helped me. As you mentioned, and I may not have said this already, a moderation analysis was highly recommended by one reviewer and I was just reading about how to conduct one this morning so I am interested to see what happens when I perform one later this weekend.

I did this research project in my last semester as an undergrad which was an independent study with my professor. I hope and plan on continuing writing such reviews in medical school. I think SRs and meta-analyses are critical to the support of evidence-based medicine and science in general!

 

[Doctor_Strange]

Agreed with DynamicDidactic. This is an impressive undertaking, both as a undergrad (VERY impressive) and a grad student because, even then, meta-analysis is a lot of work. I highly recommend the Borenstein book as a resource for learning about meta-analysis. Its very digestible and readable. It will help bring to life a lot of the processes underlying meta-analysis. I use excel for meta (its a pain, but it helps me know where the numbers come from) but I see the appeal of RevMan for sure.

As I said before, your total sample size (k, or the number of included studies) is insufficient to divide it into a separate meta-analysis. Typically you will want at least 14-17 (I've seen people vary, but in that range) to conduct a meta-analysis just to ensure you are getting a broad enough sample of the random effects. It appears from your table, and I would need to see the methods to provide a better insight, that your analysis of differences were conducted across several measures (digit forward, digit backwards, etc.) and that each of those analyses contained only 4-5 studies. I do see where the reviewer would want them separate because it appears some studies only evaluated certain measures (Nback) in one of the groups. However, I don't see that as needing separate measures, you know by virtue of the studies selected that the effect size will relate only to that topic.

As for moderators, conducting a random effects model without including moderators excludes the greatest strength of such an approach.If you look at the Random effects results you'll see a Total Q (homogeneity) and this will tell you that there is variability between the studies. This variability is what the moderators attempt to resolve. Given the limitations of your sample size and the apparent structure of the studies (such that chronic tests certain outcomes and acute tests other outcomes), this might be hard to evaluate.

whoop. gotta run. I'll get back to this later.

I will look into the Borenstein book, I think my local hospital library has a copy in fact.

Regarding my sample size, yes I think it is quite small. However, the only hope I have as since neither of the reviewers were strongly critical of it, I think I will end up writing a paragraph telling readers to look at the meta-analysis not in absolute terms, but to understand the direction the research currently is heading and perhaps prompt more to follow suit (i.e. chronic PA interventions seem to be more impactful on WM). I did just get off the phone with my adviser and she said if the reviewers said to keep them separate we should do so and explain why they will be.

Also, the reviewers did not mention or ask for a moderator analysis only mediators, I just thought doing both is better than doing only one, but judging from your comments in the last paragraph I should likely stick to what the reviewer asked of (which was just to run a mediation analysis of age, nothing else).

I may be the odd medical student student in that I actually enjoy the density that comes with SRs and meta-analyses and will look to continue to build on this in some way next year.

 

[Doctor_Strange]

So long story short, here is a meta regression of all my studies regressed against age. Some studies used multiple WM tests, and I combined them to give said study one unit of analysis. I did this because I felt then the regression would be easier to interpret with 15 distinct study point. Can I get some help interpreting this data, I have literally several days trying to find guidance on the internet, and am still quite unsure.

This is just to show what I mean by unit of analysis: some tests had multiple effect sizes from different tests that were measuring WM, so I just combined them.

edit: graphs deleted

 

[Doctor_Strange]

Just a quick update--the paper changed majorly from my initial post, i.e. able to include more studies, better descriptive and analytical analyses, slightly altered objective, but it was finally accepted today!

Thanks to those in this thread for your help--it was crucial at a time when I was super stressed.

 

[Justanothergrad]

Congrats on the acceptance.

 

[DynamicDidactic]

post the article once its published.

Congrats.

 

[Doctor_Strange]

For anyone interested, here's the link: The effects of chronic and acute physical activity on working memory performance in healthy participants: a systematic review with meta-analysis of randomized controlled trials