If you’ve familiarized yourself with general network concepts and our past work, you may be wondering what it takes to evaluate your network. Conducting a network analysis entails four major stages:
- Decide who is in the network
- Decide on network measurements
- Collect the data
- Analyze, visualize, & interpret
Each stage requires extensive communication between the stakeholder (you) and the evaluator (us). You know the players in your network better than an external evaluator does, and are therefore in the position to provide rich context for how best to measure it. Armed with the knowledge of how network analysis can describe your group, you can help us help you describe what is happening in the most informative way possible.
Who is in your network?
The first step is to decide who should appear in your network maps. Should the nodes be individual people, organizations, or a mix of both? What kinds of organizations/people are you interested in? Are they project grantees? Government agencies? Health care providers? You will likely be interested in a variety of different groups, and thinking about the kinds of groups that should be represented will help you decide who, specifically, to include.
Laumann, Mardsen, & Prensky (1983) suggest four different ways to determine network membership. Proper delineation often involves some mix of these:
- Positional: If the group has a formal membership (i.e. “All of the faculty at a particular school” or “All of the fellows and faculty selected for an institute”), then a list of members likely exists. Usually it’s not that easy though, and we have to resort to other methods.
- Reputational: We often need to find people who are knowledgeable about the network and can name important partners. For example, we once asked policy directors at local health departments who the local policy experts were, and then asked those people for additional names. This method still requires a starting list of knowledgeable individuals, and this usually starts with the stakeholders themselves.
- Event: Sometimes a specific activity or event is the appropriate boundary. Some examples include: all of the grantees, schools, and hospitals working on a particular project; everyone who published a research article about dissemination and implementation practices in public health in 2017; or all of the politicians, stakeholders, constituents, and lobbyists responsible for a piece of legislation.
- Relational: Network partners should have some sort of contact or relationship with each other, and thinking about how partners are linked can help us decide who to include. Who do foundations give money to? Who do politicians take money from? Who do students talk to when they have questions about an academic issue? These questions force us to think about how network members are connected, and thus who should be included.
Below is an example of how we collected network partners for local health departments:
We put this in an online survey to policy heads at local health departments. We got their names from NACCHO, who funded the project. Note the three-step nature; 1) NACCHO was knowledgeable about who in each department we should approach, 2) those people were then presumably knowledgeable about who in their jurisdiction worked on tobacco control policy & advocacy, and then 3) they nominated a few other people who were also presumably knowledgeable and could name some additional people.
Note how specific we were about what we were interested in – not just health policy in general, but tobacco control. Not just anyone who worked in tobacco control, but policy and advocacy. (We also provided a clear definition of what we meant by policy & advocacy earlier in the process.)
The final product of the delineation will depend on the type of analysis you would like to do and how the data will be collected. If the data will be collected with a survey, our target is a full list of key partners and their contact information (usually email). Even if the analysis will be at the organizational level, we still need to contact the appropriate individual(s) from each organization who can answer about their working relationships with other organizations. This list will serve as both a participant list and the list of partners that the participants should have relationships with.
What relationships would you like to measure?
Anything that two partners can do together or share is fair game for a network variable and depend highly on the project. Here are just a few examples of what we’ve done in the past.
Are you aware of the following organizations’ tobacco control work in [state]?
This question can also be used as a filter for subsequent questions on an interactive web-based survey. Any organizations that a participant indicates not being aware of will not appear in subsequent questions. This can reduce participant burden by a great deal if they are not familiar with many of the network partners.
On average, how often have you had direct contact (e.g., meetings, phone calls, emails, faxes, or letters) with each of the following partners within the past year? (Do not count listservs or mass emails)
Which response best describes the current level of collaboration between your organization and the following?
|None||Share Info||Informal||Formal||Multiple Projects|
What types of activities have you worked with each of your partners on [topic of interest during time frame of interest]? (Check all that apply.)
|Activity 1||Activity 2||Activity 3|
Note that a question allowing participants to select all that apply will provide a separate network for each activity, plus the possibility of a valued network that sums the number of activities between partners.
Other options include satisfaction with collaboration, barriers experienced with partners, resource sharing, etc. Again, whatever people or organizations are doing together is fair game for a network relationship.
One note of caution is to be mindful of participant burden. The actual number of questions you are asking of them is the number of network questions multiplied by the number of partners in each question. If the network is quite large, this can be a lot of work and will reduce the response rate on a survey. We generally try to keep the entire survey under 30 minutes. This usually means a limit of three network questions, so be sure to prioritize as appropriate.
Sometimes it’s possible to collect network data from archival resources such as Scopus for co-authorship networks or classroom rosters for who has had classes together. Many evaluations require asking network members themselves about their relationships, in which case a survey is the most appropriate tool.
CPHSS typically uses Qualtrics, an online web-based survey platform, to administer network surveys. Once the network partners have been identified and survey design is finalized, data collection typically follows these steps:
- An authority recognized by the network partners sends a heads-up email to participants letting them know that a survey will be on its way and encouraging them to take it. This email serves several purposes:
- Pilot test for contact emails – if any of them bounce, the sender can notify CPHSS and work to get the correct address.
- Participants will understand that the survey is important to the recognized authority.
- Despite best efforts, Qualtrics invitation emails sometimes end up in spam folders. This message can alert participants to check their spam folders.
- The invitations for the survey are sent from the Qualtrics system. The invitations contain links that are personalized for each participant, enabling automated reminders for any non-completions at specified time intervals (usually once a week).
- CPHSS tracks who has participated. Depending on the situation, follow-ups with non-completions may be conducted by CPHSS or the stakeholder.
High response rates are critical for the data collection to be informative. While traditional surveys can be informative so long as a representative sample is collected, a network survey is more demanding. Imagine that the two lightly-shaded organizations did not participate. If the relationship pictured is non-directional, we can infer relationships with the participating organizations from their responses. However, we would know only half of the story with a directed relationship, and we would know nothing about the relationships between the non-participating organizations. For these reasons, we aim for a response rate of at least 80%.
Analysis: What does it all mean?
Two challenges with analysis of network data are 1) the amount of information available is often overwhelming, and 2) stakeholders are much more familiar with the context that the networks occur in than we as external evaluators are. To that end, CPHSS has developed an online Network Navigator application, a customizable tool that allows stakeholders to explore their networks interactively and decide what is most important to focus on. Click here to see a demonstration example.
Key features of the Network Navigator include:
- A page describing the project, including information about how to interpret the statistics and graphics displayed.
- A tab for each network relationship.
- Within each network, users are able to select different levels of the relationship and size the nodes according to a variety of node-level statistics. Available options are discussed in Network Analysis 101.
- An Excel workbook containing both node-level and network-level statistics for the selected network can be downloaded. Both levels of statistics change depending on the relationship and level selected, and the downloaded file is automatically named as appropriate.
- Nodes can be selected based on group enabling the examination of within-group links.
- Hovering over an individual node will reveal a pop-up with additional information about that node. Clicking on the node will highlight its “neighborhood” (the nodes it is directly linked to) and display how its connection statistics compare with the network average.
- Clicking on a node also enables the isolation of the node and its neighborhood, producing an “ego” network for that node.
- Right-click anywhere on a network graphic to export a static .png image.
The interactive nature of the application allows stakeholders to engage directly with the data, bringing their own knowledge of the network partners to the table, and facilitates evidence-based decision making. Are there any disconnected partners, and if so, who are the most appropriate partners to link them to? Are the right partners connected to each other? Does a hub partner in a highly centralized network have the capacity to handle all of its connections? Are the biggest providers who we expected? Are there partners that should step up or pull back? All of these questions require the stakeholder to interact with the data. As an external evaluator, CPHSS can shine the light, but the stakeholder must know where to point it. The Network Navigator application facilitates the dialog that makes the required coordination possible.
The application is meant to be a stand-alone self-explanatory product. The available options are not limited to those presented in the demonstration application and we are always looking for ways to enhance it. Some of the most interesting options have been developed at the request of stakeholders, so we welcome the opportunity to add more functionality to fit the needs of a particular project.
The application can also be password-protected if desired. In the case of a protected application, the primary stakeholder decides who should have access, and CPHSS administers invitations to authorized individuals to create an account.