Knowledge for Change Program

Guidelines for Reviewers1

TTL: Samuel Fraiberger

Proposal Title: Data-driven Analysis and Visualization of Entrepreneurial Ecosystems

The Knowledge for Change Program’s Internal Management Committee (IMC) seeks expert review
of all proposals. The contributions of expert reviewers are essential to foster high quality research,
and careful reviews of research proposals carry great weight in IMC’s funding decisions.

We would like to draw your attention to the attached KCP III activities document which outlines the
priority research questions under each theme. Please take it into account in your review. Please
provide detailed comments, as appropriate.

Reviewers are requested to respond to the following:

1. Objectives. Are the goals or objectives of the proposed study well defined? Are the issues
important for policy in developing (or transition) countries?

Comments:
The goals of the proposed study are well-defined and the raised issues important for policy in
developing countries. However, I have some concerns regarding the possibility of achieving the
proposed goals within this study (see point 10. for a detailed elaboration of this comment).

2. Analytic Design and Data.: Are the analytical approach adequate and methodology rigorous to
deal with the research questions? Is data attainable or do statistical techniques adequately
compensate for data limitation?

Comments:
In the proposal, the description of the methodology is limited. The TTLs state that they will use
(multi-layer) networks and previously developed methods for the analysis of individual careers to
study data from crunchbase, pitchbook, linkedin. However, methods are scantly described and
weakly put in relation to the data and its limitations. The methodology description refers only to
works by two of the TTLs, but only one of them is an academic paper covering social network
analysis. My concern is that the TTLs have not given enough thought to the details of the
methodology, which is worrisome for an 18 months study (see also comments on point 10).

3. Literature. Does the proposal reflect the current knowledge and literature? Does the proposal
provide convincing rationale for the selection of issues within an overarching topic ? – through an

1
Confidentiality of the Review and Reviewer: All reasonable care is taken to maintain reviewer confidentiality.
The applicant will have the opportunity to read the (anonymous) reviews. Reviewers are asked to omit any obvious
references to their own work, if any.
 Guidelines for Reviewers
overview of the existing body of literature on the subject, identification of priorities for deeper
understanding, and discussion of analytical problems constraining operations and policy.

Comments:
References about methodology are limited to success studies published by two of the TTLs experts.
Two of the referenced papers (Sinatra et al., Wang et al.) do not contain any network analysis, but
are only about career developments. The other referenced paper (Fraiberger et al.) uses social
network analysis applied to careers, but no real geographic analysis. The fourth reference (Barabasi
(2018)) is a popular audience book probably without deep methodological insights. Hence, I have
serious concerns that enough thought has been put into how to manage this “comprehensive
network analysis of entrepreneurial ecosystems across geographies.”
Two reference papers, Bonaventura et al. and Yin et al., do exactly a network analysis of
entrepreneurial ecosystems. The proposed project only briefly (and not convincingly) mentions
that it will perform a comprehensive analysis in respect to these studies.
An important reference “Where is Silicon Valley? Science (2015)” is missing.

References about the role of social network for success and valuation are also very limited. One
for all, the Granovetter’s paper about the strength of weak ties (Social Networks, 1977) is highly
relevant, but missing. A clear elaboration of the role of social capital for success is also missing
(see for example Seibert et al. Academy of Management Journal (2017)).

4. Program Implementation. Is the implementation arrangement appropriate to the demands of the

Comments:
The 18 months frame sounds like a limited timeframe to achieve the states goal of “develop an
innovative data-driven approach to map entrepreneurial ecosystems and analyze the role they play
in shaping entrepreneurial success”. However, it is probably enough to provide the promised
deliverables.

The team has enough experience to undertake the project. However, it is unclear who will actually
conduct the analysis, as the team mentions “to hire two additional data scientists to help with the
data analysis and the visualization tool”. I do not see budget allocated for these two data scientists,
unless it is allocated in the line about the consultancy company (USD 130k). More information
about the alignment of team, activities and budget would have been useful to comment on program
implementation.

5. Will the proposed study have potential policy impact? Please rate the potential policy impact in
developing (or transition) countries or in the World Bank, assuming the study is undertaken as
planned.

Your rating___3______(on a scale of 1 to 5, with 1 being no policy impact and 5 being the
greatest potential to influence policy)

Comments:

The proposed study has certainly potential policy impact. However, the results will be developed
using data biased towards the western world and I have concerns that they can be applied to
developing countries.
2
 Guidelines for Reviewers

6. Local Capacity Building. Does the project incorporate developing country/local research
capacity building? Please rate the capacity building potential of the project.

Your rating____4_____(on a scale of 1 to 5, with 1 being no local capacity building and 5 being
with significant local capacity building)

Comments:
I believe the project will have good capacity building potential through its workshops, especially
the one planned in Japan.

7. Operational Linkage. Is the consultation with related research and operations (client facing
lending, advisory services) units as documented in the proposal adequate? Will the study
promote collaboration between research and operations? Please rate the operational linkage
of the project.

Your rating:__3_____ (on a scale of 1-5, with 1 being no linkage, and 5 being the strongest
potential to promote collaboration between research and operations)

Comments:
The proposal offers some elements about the collaboration between research and operation. For
example, the TTLs plan to present the project findings in two workshops, one of them in Japan.
There are also concrete thoughts about the countries to focus on promote startup ecosystems.

8. Budget. Is the budget appropriate? Is the cost commensurate with the possible value of the
findings? If not, what is your suggested budget for this project if it’s worth funding.

Comments:

I had a hard time understanding the alignment between proposed project and budget. For
example, the project mentions that two data scientists will be hired for the visualization and
data analysis, but I cannot see this item on the budget. A substantial portion of the budget is
allocated for consultancy firms, who supposedly will do most of the work since there is an
allocation of only 8 weeks for staff time. More clarity about the team arrangement and who
will carry on this novel and therefore intellectually challenging work would have been useful
to comment on budget and feasibility. Without looking into details, the overall budget (but not
the timeframe) looks appropriate for the project goals.

9. Suggestions. Where you identify shortcomings, please suggest alternatives to alleviate those
problems. This might include reference to literature using more appropriate methods,
reduced scope, a revised team, phased funding or re-sequenced activities. Where possible,
please give rough estimates of financial and schedule implications.

3
 Guidelines for Reviewers
Comments:
Methods and literature: if the TTLs intend to analyze and visualize multiplex networks of
startup networks, they should look into the literature of multiplex network measures (for
example Dickison, Mark E., Matteo Magnani, and Luca Rossi. Multilayer social networks.
Cambridge University Press, 2016, or Kivelä, Mikko, et al. "Multilayer networks." Journal of
complex networks 2.3 (2014): 203-271.). There is also already available software to visualize
multi-layer networks, like MuxViz. They should also use measures that take into consideration
the volatility of data, for example by looking into the temporal network literature (e.g. Holme,
Petter, and Jari Saramäki. "Temporal networks." Physics reports 519.3 (2012): 97-125.).

Please also see comments on point 3.

Team: more TTLs time should be allocated on the project, or people with ad-hoc expertise
should be hired (not some vaguely defined consultant data scientists).

18 months look also like a short time frame for the stated goal of the project. If the goal of the
project is downgraded to the creation of a visualization platform of the multi-layer networks
present in the data, then a budget of around 100k should be sufficient.

10. Overall comments and rating.

Your overall rating of the proposal______3_____ (on a scale of 1-5, with 1 being the worst, and
5 being the best)

Comments:

This project proposes to apply a data-driven approach, based on social network analysis and studies
of individual careers, to datasets about entrepreneurial ecosystems, with the of goal of quantifying
the likelihood of entrepreneurial success. I find the idea of the project novel and interesting, but I
have a number of major concerns at the conceptual level, about data and methodology, and about
project feasibility. I list these concerns below.

Conceptual: how is entrepreneurial success defined? How does the proposed data capture the
multifaceted dimensions of entrepreneurial success (e.g. economic success, financial success,
growth, etc.)? Why are networks important to study entrepreneurial success? What is the necessary
social network theory underlying the proposed project? All these fundamental questions are not
tackled, or only poorly.

Data: The project plans to use three data sets, from pitchbook, crunchbase, and linkedin. The first
two data sets are already available to the TTLs, the third is still object of negotiation. While I
appreciate that the study is planned to be conducted on three different independent data sources,
these three datasets are strongly biased towards western countries. Therefore, I am seriously
concerned whether the findings can be applied to other ecosystems, especially in developing
countries. The TTLs have provided no thoughts or risk limitations about this crucial “study
transportability” caveat.
Another concern regards the ability of the proposed data to “capture the activity, performance,
accomplishments, and impact of individual entrepreneurs and the businesses they develop”.

4
Guidelines for Reviewers
Among these datasets, for example, I cannot see how accomplishments and impact can be
measured in the linkedin dataset.

Methods: The use of social network analysis is certainly relevant to study entrepreneurial
ecosystems. However, no details are provided about the social network theory and assumptions
that will be used, and how they will be operationalized. The study mentions only briefly that a
multi-layer network approach will be used, but does not offer details. Even the units and links at
the base of the network approach are unclear. Also, how does the study deal with the volatility and
longitudinality of data? Success of an entrepreneurial ecosystem is certainly a time-dependent
concept. The description of the methods is missing important details, mainly refers to works by
two TTLs but with references that seem misplaces (see my comments on point 3). Taken together,
it seems that not much thought has been dedicated to the methodological part of the proposal which
is worrisome for an 18 months study, supposed to be completed by June 2021.

Feasibility: I have concerns about the possibility of transport the obtained findings in developing
countries, because of biased data (see above). Also, the goal of “developing an innovative data-
driven approach to map entrepreneurial ecosystems and analyzing the role they play in shaping
entrepreneurial success” does not look commensurate with a length of 18 months, especially given
that only 8 weeks of TTLs time have been allocated for the overall project.

All in all, this study is ambitious and its overarching goal novel. However, it rests on somewhat
poorly described assumptions, possibly biased data, and weak methodology. I would recommend
the goal (and therefore the budget) to be downgraded to the mere creation of a visualization
platform of the datasets, and that in all the workshops and presentations to clients the issues about
the definition of success and data biases are clearly presented.