Endless Sky Community Survey

[Hecter94]

Hey everyone!

As we've recently been able to start pushing out regular updates, we've seen a large growth in our community across multiple platforms. We now have several thousand members across Steam, Reddit, Discord, Github, and elsewhere. With our community continuing to grow, we want to ensure we keep tabs on everything across the various platforms and are still effectively meeting our members' needs.

As such, we have put together an open-ended and completely anonymous community survey to give everyone an opportunity to share their thoughts and opinions on how everything is being managed, share any concerns you may have, and make any suggestions you feel could improve our community.

The survey is available here: https://forms.gle/Yi2f11ww1TtmEV2m7

It is completely anonymous, and all questions are optional. You may give as detailed feedback as you wish, fill out every section, or just answer a single question. The results of this survey will help to determine the future of the Endless Sky community. They will be instrumental in determining how our community continues to grow and evolve into the future.

Your feedback is invaluable in helping us better understand the needs and preferences of our community, and we appreciate any participation. Thank you to everyone for being a part of this project!

[Zitchas]

Note: This is a smart survey, meaning there are a variety of sections that will be hidden invisibly based on your answers. As best I can tell, every "required" question (indicated by the subtle red asterisk) is a control question that, if you pick no, will completely hide everything even vaguely related to that subject, with no opportunity to indicate that you used to use it, or have tried it out and stopped, or anything else.

So, for instance, if you've ditched using discord for development for whatever reason, and tick the "No" box that asks if you use discord, then you won't see any of the questions regarding discord or have any opportunity to elaborate on why you don't use discord. So if you did use a particular medium and dropped it, you will need to answer "yes" to the question of "Do you use X medium" in order to provide any feedback.

As such, if you have opinions on development work on Discord, then you must check "Yes" on both the "Do you use Discord" question, as well as the "Are you currently engaged with Endless Sky Development on Discord?" question, otherwise it will just skip the section and you won't get the chance to provide any feedback on development activity that happens there.

Which is a fairly serious problem, as it biases the results in any particular section, as all the people that have used one of the various mediums listed but have stopped and no longer use it for whatever reason will automatically be excluded, leaving all the feedback in that section to be purely from the perspective of those who have continued to use it, and presumably have at least a passingly positive opinion on it. By its very nature, this means that those with the most negative opinions of a particular communications medium will be filtered out from providing any feedback, or even being told that questions were omitted.

In short, if you're going to filter out large chunks of areas to respond in based on if someone currently uses that medium, it'd be much more honest and useful if it went with a couple more questions such as "Have you used it in the past?" and "Why did you stop using it?", along with perhaps a "Do you have any other opinions on X medium?", and answering yes to the first or last one should result in getting the full question set for that medium. As it stands, the survey results have a built in bias, and will need to be treated accordingly.

[MasterOfGrey]

I have discussed with Zitchas that this sort of smart survey is very much a standard survey design these days.

However, this draws attention to the fact that people are not necessarily as familiar with that as the survey team expected, and so the first 3 paragraphs here are a good note for others to pay attention to if you're less/unfamiliar with modern smart surveys.

[Zitchas]

Agreed. Just for the record, I've dealt with quite a few of these in the past. This is why I made use of the back and next buttons to go back and forth and experiment with the answers to see what gets revealed. They're not novel at this point, it's primarily the language, specifically the grammatical tense, that I'm not fond of. This survey reads like something interested exclusively in the opinions of people actively using that medium right now. To be fair, this is perfectly valid if you want to find out people's attitudes about the thing exc;lusively from the people involved in using that thing right now. My own bias shows through on this, as I view finding out why people stop or avoid using a medium to be more useful.

For future reference, questions need to be a bit more inclusive or provide secondary ares for feedback.

For instance, "Do you currently participate with the ES Community on GitHub?" emphasizes present tense usage, giving the sense of "do you use it right now."

Two options to improve it:

1. Change that to "Do you currently participate or have you ever participated with the ES Community on GitHub?" (Which allows us to see how many people have used GitHub)
2. Alternatively, for people that respond "No", have the follow-up ask "If they ever have participated" and "Do you have any thoughts on it?", responding yes to either of these would then have them given the GitHub section. Same thing goes for pretty much all the determinator questions.

It would be far more useful to get paired values so we can see both how many people have used GitHub to follow things alongside how many currently do, rather than just seeing how many currently do.