A. Get published Review Protocols from Campbell Collaboration.
On campbellcollaboration.org webpage, click “Campbell systematic Reviews journal”The link takes you to the Wiley Online Library and you will need a library account to browse the contents. Click “Campbell Article Types”Select “Protocol” from the list of types.Total 248 results which can be further narrowed by selecting “Campbell subject Categories”.
B. Get published review protocols from Cochrane Reviews.
https://www.cochranelibrary.com/Search ‘mathematical’ in the Title Abstract Keyword and get 1 result under the Cochrance Protocols tab.
C. Layout of the Protocol
Background
The problem, condition or issue
Intervention
How the intervention might work
Why it is important to do the review
Products of this systematic review
Objectives
Methodology
Criteria for including and excluding studies
Types of study designs
Time and language
Types of participants
Types of interventions
Duration of follow-up
Types of settings
Search strategy
Search terms
Description of methods used in primary research
Criteria for determination of independent findings
A systematic review study is planned with the purpose of investigating whether current educational programs are effective for developing problem solving in early childhood education.
Same platform has same interface, nut the subject headings are different. Same database on different platforms has the same keywords for titles and abstracts, but different heading and different truncation and proximity syntactic rules.
Group means, SD, N, estimated effect size, appropriate SD, F, t test, significance, inverse variance weight
D. Statistics
Cohen’s kappa
Cohen’s d
effect size
aggregate/weighted mean effect size
95% confidence interval: upper and lower
homogeneity of variance (Q statistic): Test if the mean effect size of the studies are significantly heterogeneous (p<.05), which means that there is more variability in the effect sizes than would be expected from sampling error and that the effect sized did not estimate common population mean (Lipsey & Wilson, 2001)
df: degrees of freedom
I square (%): the percentage of variability of the effect size that is attributable to true heterogeneity, that is, over and above the sampling error.
Outlier detection
mixed-effects model (consider studies as random effects): moderator analysis for heterogeneity (allow for population parameters to vary across studies, reducing the probability of committing a Type I error)
Proc GLM/ANOVA (consider studies as fixed effects): moderator analysis for heterogeneity
Region
Socioeconomic status
Geographical location
Education level
Setting
Language
sampling method
Statistical difference in the mean effect size of methodological feature of the study
confidence in effect size derivation (medium, high)
reliability (not reported, reported)
validity (not reported vs. reported
classic fail-safe N/Orwin’s fail-safe N: The number of missing null studies needed to bring the current mean effect size of the meta-analysis to .04. Threshhold is 5k+10, k is number of studies for the meta-analysis. If the N is greater than the 5k+10 limit then it is unlikely that publication bias poses a significant threat to the validity of findings of the meta-analysis.
Used to assess publication bias. eg. control for bias in studies (tightly controlled, loosely controlled, not controlled)
E. Purpose/Research Questions
Whether the treatment is associated with single effect or multiple effects?
Understand the variability of studies on the association of treatment with single or multiple effects, and explain the variable effects potentially through the study features (moderators). How do the effects of the treatment vary different study features?
F. Reference
Odesope et al, 2010: A Systematic Review and Meta-Analysis of the Cognitive Correlates of Bilingualism
SUGI27 paper: (Hamer and Simpson, 2002) SAS Tools for Meta-Analysis* *The methodology in this paper is ok, but the example was not interpreted correctly. I have corrected the example in this post.
A meta-analysis is a statistical analysis that combines the results of multiple scientific studies. Meta-analysis can be performed when there are multiple scientific studies addressing the same question, with each individual study reporting measurements that are expected to have some degree of error. The aim then is to use approaches from statistics to derive a pooled estimate closest to the unknown common truth based on how this error is perceived.
Research collect data for meta-analysis by systematic review of the literature in the field, and compile data directly from the summary statistics in the publication.
C. Problem with simply lumping the data from different studies together
Not consider treatment-by-study interaction
Assume response rates are the same in all studies.
D. SAS Solution (follow Hamer and Simpson’s paper, but corrected the output from the paper)
Create data set with the results of 2 studies. B: Remitted; N:Not remitted; P: Placebo; D: Drug.
I have used B (Better) to indicate Remitted cases because Proc Freq test is based on column 1 and row 1 of the 2 by 2 table, so if we code R for Remitted cases then the remitted case will be in column 2 because the table is by alphabetical order and R is after N.
The Hamer and Simpson paper actually tested the null hypothesis for the non-effective cases rather than the effective cases.
data chm;
input study $ response $ trt $ cellfreq @@;
datalines;
study1 B P 24 study1 N P 3
study1 B D 58 study1 N D 30
study2 B P 16 study2 N P 57
study2 B D 2 study2 N D 10
;
run;
Run Cochran-Mantel-Haenszel Statistics using Proc Freq procedure with cmh option.
The Mantel-Haenszel estimator of the common odds ratio assumed the estimation to be homogeneous among both studies.
The Mentel-Haenszel statistics tests the null hypothesis that the response rate is the same for the two treatments, after adjusting for possible differences in study response rates.
For Proc Freq testing options, make sure the group that you want to tested are in row 1 and column 1. It is also important to crosstab treatment as row and response as column, so the interpretation of the relative risk for the risk of improvement make sense. In Hamper and Simpon’s paper the crosstab has been transposed, therefore the relative risk output doesn’t make sense.
G. Interpretation
The CMH test statistics is 4.65 with a p-value of 0.03, therefore, we can reject the null hypothesis that there is no association between treatment and response. P-value lower than 0.05 indicates that the association between treatment and response remains strong after adjusting for study.
Relative Risk (Column 1) equals to 0.74 which means the probability of the improvement with the drug is 0.74 time the probability of the improvement with the placebo.
Relative Risk (Column 2) equals to 1.51 which means the probability of no improvement in the symptoms with the drug is 1.51 times the probability of no improvement with the placebo.
The Breslow-Day test has a large p-value of 0.295 which indicates there is no significant difference in the odds ratios among the studies.
* I will show the odds ratio and relative risk calculation in Excel in another post.
BI Practice 