Hey Guido - Thanks for checking out Symbol. We do agree that arbitrarily using percentiles or star systems on resumes isn't useful, since these self-grades aren't in relation to anything (and so don't mean anything).
As an experiment, we thought "What if we could create a system where you could see your percentile IN RELATION TO everyone else on the platform?". In other words, if you receive a score of 72 percentile in graphic design skills, this means you demonstrate more expertise in graphic design than 72% of those with that skill on the platform.
If the platform gets larger enough, you can imagine that the percentile score represents your general ranking in the population (or at least a large enough pool for the scores to be useful).
Not sure this is the right approach, but we do think it's interesting to consider how skills might be accurately measured...
@guidooo percentiles are used to rank people's skills, not measure the quality or "usefulness". This ranking is just a statistical measure, which can be both accurate and dynamic, e.g. it can scale up with the increased community talent level.
What would be an alternative? 5 golden stars ⭐? A thumbsup 👍? A number of seconds you keep mousedown on a 👏 button (medium)?
There's been a substantial research revealing mental biases in people when rating something in e.g. 1-10 points or 1-5⭐, which wouldn't allow for fair assessment (sorry can't find the direct source, but it is mentioned in Criticism of NPS: https://en.wikipedia.org/wiki/Ne...). Basically when asked to rank something from 1 to 10, people would most of the time say "7".
Another problem with fixed scores like 5⭐ is that you have to know your population to be able to assign a "fair" number of stars to each rank, i.e. 4⭐ when the rank is over 70%, but below 90%. But your population can change, and rarely you would have all the data beforehand (definitely, Symbol doesn't have it all from start).
So leaving percentiles as they are (e.g. say 87 instead if 4⭐, say 69 or 58 instead of 5⭐) works the best in my opinion.
@guidooo@viktor_cherkaskyi I think his point is not about the accuracy of the system but measuring skills itself. You can't measure skills, that's the thing.
Just because A made X and B don't, doesn't mean in any way that A is better or is more skilled than B. Even worse, if B made Y how they score that the case study X is better than Y? Why? Even if X were better than Y, does that really mean that A is more skilled than B? Absolutely not.
Again, you can't measure skills. Therefore, you can't rank people's skills.
@guidooo@viktor_cherkaskyi@inthe0n Exactly. One of the things we were taught in my design school was to never rank your skills by using an sort of % or grading method.
Because of the simple fact that what if I rate my photoshop skills at 60% because I believe I have a lot of things left to learn (3d modeling in PS, gif creation in PS, etc) and then someone else comes along and puts a 90% on their resume and they actually have less technical knowledge than I do but they just believe that they understand the concepts that they need to know.
Because everyone is going to rate their skills differently. It's hard to measure exactly what skills your looking for because skills can vary so much depending on a project. If I put 90% skill rating for Adobe photoshop on my resume then my new boss gives me a task that I fail at because I didn't do it well enough because it was something that was out of my "skill zone that I was rating myself at".
Using % is a very tricky thing to do because everyone is going to make themselves look good even if the skills they are evaluating themselves on isn't the same as what someone is looking for in someone.
--
Another example,
I rate myself a 80% at photoshop. Someone comes to me and asks me to do a digital painting. I tell them I don't do paintings. Does that mean that I was lying about my 80% skill in PS. How do I evaluate myself. Its very hard because there are so many variances to look at in terms of trying to put an overall % on many skills.
@guidooo@viktor_cherkaskyi@mastemine 💯. Dunning–Kruger effect.
Quoting Dunning: "If you're incompetent, you can't know you're incompetent ... The skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is."
@guidooo@viktor_cherkaskyi@inthe0n@mastemine This is a fascinating discussion. I briefly mentioned the Awwwards system in another comment, would a weighted voting system where reviewers with more gravitas are given stronger weighting be a fairer assessment? Or is the concept of objectively rating one's skills too flawed as a concept?
Hey Product Hunt!
I'm excited to share Symbol with you.
We've been thinking a lot about the question: "How can anyone credibly and universally prove their skills without relying on traditional credentials (i.e. degrees, job titles, years of experience)?".
Symbol is one of our experiments to create a new and free "universal credential".
On Symbol, you can publish a beautiful portfolio of your work and then see where you rank globally for any of your skills (you're given a percentile score from 1-99 for each skill). These rankings are free to receive and over 95% accurate (thanks to the Law of Large Numbers, explained more below).
We'd love to hear what you think and see if you have any ideas to help 1. Improve the idea, 2. Create the necessary incentives to kickstart the network, 3. Other ideas to create a free universal credentialing system.
________________________________
Here's how it works, briefly:
1. Use our Medium-like editor to create a beautiful portfolio of your best work (here's an example case study: https://withsymbol.com/u/max/191...).
2. Tag each case study with the skills it demonstrates and publish the case study. (This can include tangible skills like graphic design and coding or less tangible skills like creativity and leadership).
3. Once published, your case study is anonymized and reviewed by others on the network. Reviews are done by pairwise comparisons (we ask a reviewer which of the two anonymous case studies better demonstrates the skill under review).
4. One review like this is very subjective, but once each case study is reviewed about 30-40 times directly (and thousands of times indirectly), your score converges to your "true score" with about 95% accuracy. For this to work, the "better" case study needs to be selected only about 55-60% of the time (where 50% is random), so there's a lot of baked in room for noise.
*Scores are calculated using a modified version of the Glicko rating system, which is also used to compute chess rankings on Chess.com and your "hotness ranking" on Tinder, for example. We've made a number of modifications to make the algo work much better for this kind of ranking system (i.e. reviews are weighted in correspondence with the score of the reviewer for that particular skill).
5. Once your get your scores (a percentile score from 1-99), you can manage them how you want, either keeping them private or public.
We have some additional info on our homepage: https://withsymbol.com/.
________________________________
Obviously, this is pretty out there, but we believe creating a free, credible, universal way to credential anybody in the world is one of the most compelling opportunities currently. At scale, it would drastically change the ways the education and job markets work.
We would love if you could help us explore this idea further. Let us know if you have any feedback or thoughts, and looking forward to seeing your portfolios on Symbol!
I feel like there's not really a need for a tool like this.
Pros:Nice interface... i guess?
Cons:Can we all stop using percentiles to measure skills?
Monthly
Monthly