Goal 1 Mini Group - Survey Library Carpentry Members

[arieldeardorff]

I thought I would get the ball rolling since goal 1 doesn't have a chair. As a reminder, this group is looking into surveying the Library carpentry membership to assess impact and gather stories.

Members:
@jcoliver, @chodacki, @gitti1, @Jaredabera, @konrad

Useful links from the etherpad:

• Issue w/ discussion: Review Pre & Post Workshop Surveys #4 (comment) - @kariljordan from the Carpentries will be very helpful here
• Materials from the long term assessment survey: https://carpentries.org/assessment/
• Long term assessment survey questions on the etherpad: https://pad.carpentries.org/lc-advisory-group. Are there library related questions the survey doesn't ask that we would like to ask attendees after a workshop?

Here are some questions to consider:

• Do we want to create a new survey of library carpentry attendees?
• Do we want to add some additional questions to the already existing Carpentries surveys?
• Does anyone volunteer to be chair of this group to make sure things are accomplished?

[libcce]

Thanks @arieldeardorff for creating this issue to start. I'm not on this group but I will add that I believe this goal comes from the question, do we know if Library Carpentry workshops have been successful? Have attendees actually integrated anything from the lessons into their workflows? And/or do they come away with a better understanding of the overall context? If so/not, let us know. Many of the workshops do a roll call in the Etherpad. We can think about reaching out to workshops which have been run within the last year? The only thing I would be interested in is the breakdown of library roles that have attended the workshops. The current survey doesn't allow us to understand which library roles tend to attend the workshops.

[kariljordan]

This quarter we are collecting feedback for our Long-Term Impact survey. We will collect feedback from 2-20 December. Questions related to impact are included in this survey, and we ask which Carpentry (Library, Data, Software) workshop participants last attended.

You can see the survey questions here.

Once we've collected this data I'm happy to pull out all of the responses for LC.

[konrad]

Thanks, everybody. I can take over to chair this mini group as I am quite interested in the feedback. I had a discussion with @tracykteal regarding available data of the workshop impact two days ago as she presented some of the results of a previous assessment as part of a talk that I was not aware of. This can be found here. Most important for this question is the section "Workshop Impact" and "Figure 12" in there. The responses were rather positive but for sure biased as people with negative might not have answered.

One pointed we also discussed was if it might be better if the invitation to the survey is send by instructors as they have (hopefully) a closer connection to the learners. I am not sure if we are allowed to do this for past workshops without having the consent of the learners - would that be GDPR compliant?

Regarding the questions - I think the survey that @kariljordan linked is quite complete. My biggest concerns are that those surveys getting too long so I would rather not add further LC specific questions unless there would a significant gain.

[arieldeardorff]

Thanks for volunteering to lead @konrad!

[konrad]

Minor update as preparation for our call today: The pull request to update the report is under review right now.

[konrad]

The document "Analysis of The Carpentries Long-Term Surveys" can be now found on Github pages as HTML and on Zenodo as PDF file. I would suggest we read this and have a call to discuss it. I will take care of the scheduling.

[libcce]

Hi @konrad the turn out from librarians looks low. I've seen this in previous surveys as well. In addition, how the roles are listed can lead to maybe some confusion. I can see a directed survey towards the library community regarding Library Carpentry possibly improving the response rate. If we could look into the roles more granularly I think that would be interesting. Are there certain groups/roles from libraries attending? What we also seem to be digging for always are the stories regarding how people have applied what they have learned (and the stories seem to really resonate). An LC directed survey might also help inform the curriculum (at the moment this guided more by the instructors). Just some quick thoughts but I'd be open to a meeting. Thanks for organizing!

[gitti1]

Hi @konrad and all, thanks for organising today's call. All mini group members may have a look at our notes here - feel free to add your thoughts/comments. Talk again soon!

[arieldeardorff]

Thanks for drafting the survey @konrad @gitti1 and @Jaredabera! I added some comments but I think it would be great to see it again (and maybe share with @kariljordan) once you have a survey mockup we can test out.

[kariljordan]

Yes! I'd be happy to review once it's ready. I'd also like @klbarnes20 (Carpentries Deputy Director of Instructor Training) to review as she's going to be working on learner assessment as well.

[ragamouf]

Hi Birgit and Konrad,
Great to see this ready to distribute! I have a few suggestions, please disregard if you've already hashed these out!

Survey preamble: It might be a good idea to be more explicit about the purpose of the survey eg:  
"The aim of this survey is to develop our understanding of Librarians as community members of The Carpentries, in terms of technical skill development, application and impact of Library Carpentry workshops and future curriculum development. We will use the results of this survey to guide the Library Carpentry Advisory Group's advocacy work in The Carpentries, and foster greater participation in the Library Carpentry community.
Please forward this survey to colleagues and others who you know have been involved with or attended any Library Carpentry workshops."

**And maybe include a note about privacy or survey results? Do you intend to publish the data on zenodo for example, or upload it here in the LC repo?

Q2 Perhaps change: What is the role of your library in The Carpentries to
Q2 "What activities describe your Library's relationship to The Carpentries? Select all that apply"

Q6 Perhaps move 'I don't remember' to the bottom of the list so that respondents don't default to the first option :)

Q7 This seems to ask two different questions. Perhaps split into 2 questions:

7a) In which areas have you applied the skills you gained through Library Carpentry workshops
(Checkboxes list the same answers as Q3)

7b) How have you applied the skills you learnt in previous Library Carpentry workshops?

• I have adopted tools to support my daily work activities
• I have suggested tools to library colleagues and explained how they work.
• I have suggested tools to clients and explained how they work
• I have applied tools in a new work area or project

Q8 Perhaps reorder the answers to:

• Daily
• Weekly
• Monthly
• Several times a year
• Less than once a year

Q9 Perhaps change 'processable' to 'reproducible' (This sounds like a FAIR in disguise question. Would it be worth adding a response like " Enabling FAIR research data"?

Q11 If this question is to solicit how librarians typically pursue training and other professional development, perhaps include some online options also, eg

• attend webinars or community discussions
• join a mooc
• join an online community of practice
• form a local community of practice with colleagues
• (Perhaps reword this answer: Participated in a semester-long course) Undertook tertiary study (eg semester long course or university subject for credit)
• Pursued certification or further technical qualifications

Q12 Add answers to include

• Joined a lesson as a Maintainer
• Participated in other Carpentries community activities (sprints, lesson development, community discussions, attended CarpentryCon, CarpentryConnect)

Other questions:
Are you interested, or does it matter where respondent's libraries are based, geographically? This could help us target specific communities.
Are you planning a report or summary of these results? Would you like to ask respondents to include their email address if they would like to receive a summary of the survey results after analysis? Thinking about opportunities for further engagement after the survey, although I appreciate this makes it harder to assure anonymised responses.
Are you interested in responses from library-adjacent professional/research staff or students, such as data stewards or people involved in organisational units outside of Libraries such as repositories or eResearch or Research Offices? Implication to reword Q3 if you want to broaden the scope.

Whoa. I think I'd better stop there! I'm really looking forward to the responses!

[annajiat]

Feedback on https://carpentries.typeform.com/to/zCkEJrkX

Thank you @gitti1 , @konrad, and everyone else for the hard work. It looks wonderful.

Should we think about the following changes?

In the first screen:

• updating to use oxford comma
• updating "10 minutes (18 questions overall)" to "15 minutes (27 questions overall)" or specify a range of 10-15 mins
• updating "Takes 7+ min" to "Takes ~15 mins" or specify a range of 10-15 mins
• using justified alignment instead of the center alignment
• updating "Sharing of results" to "Sharing of Results"

Q5 When did you last attend a Carpentries Workshop?

• using sentence case for the options
• updating "6-12 Months Ago" to "7-12 months ago" to avoid fencepost error

Q6 What content was covered?

• re-ordering options alphabetically or perceived popularity based on workshop offerings?
• considering inclusion of "Introduction to Data for Archivists" even though it may be covered by "regular expressions"

Q16 Creating processable reports (e.g. R Markdown, Jupyter notebook)

• updating to "e.g., R Markdown, Jupyter Notebook"

Q17 In this series of 8 questions, you will be asked to rate which of the following behaviours/routines you have adopted as a result of completing a Carpentries workshop. Please select the option most consistent with your practices.

Other

• I was confused about how to answer this question. May be omitting "Please select the option most consistent with your practices."?

Q19 In which of the following learning activities have you engaged since completing a Carpentries workshop?

• adding "an": Contributed as "an" instructor/helper to a Carpentry workshop
• adding another option for 4 days or change 2 days to 4 days: "Participated in a short course or workshop (up to 4 days)"

Q20

• adding a question mark (?)

Q23, 24

• adding a full stop (.)

Q28?

• may be adding Q28 with open response, "Anything else that you would like to share?"

Please feel free to correct if you find any errors above.

[chennesy]

Also offering feedback on https://carpentries.typeform.com/to/zCkEJrkX and with kudos for putting this together! Looks great. A few small things, in addition to what Annajiat shared above:

Q2: I'm not clear what "We participate in workshops" means here. Does this mean teaching workshops? Hosting them? Helping out with them?

Q17: I was also confused about what is being asked here. The previous questions included a Likert scale to rate behaviors/routines the user has adopted after attending a workshop. This Q has the same prompt (asking us to rate behaviors) but then includes a free-text box with the prompt "Other" and "Type your answer here." If the idea is for users to write in with other behaviors that have changed I would simply ask: "What other behaviours/routines have you adopted as a result of completing a Carpentries workshop?" and then include a free-text field.

[emcaulay]

A survey was developed and shared in Summer 2021. I will come back and post a link and the deadline for closing the survey. We have circa 90 responses so far.