"Never accept results out of hand, always question wether your analysis may have lead you to a faulty solution"
Old methods, recent syntoms
Reading the material you find around the web about evaluation,
you will notice that it boils down
to some common 'basic' rules. The problem is that, like all generic rules, they are
-by a long shot- not valid
enough. Therefore, even if they were all valid, the capability
to 'feel' if a site is bogus (or not) will still require
some 'Fingerspitzengefühl' which amounts to a combination of experience
and subtle intuition, to acquire the uncanny insight of
accurately perceiving fake constructions and attitudes.
I believe that
looking for information on the Internet
you will need to use the same critical
evaluative skills that
you would use when choosing or reading
a book, when choosing or using a paper index, when choosing or evaluating a musical
when deciding about the value of a database.
For this reason you should by all means and aforemost
Let's have a closer look, evaluating critically the data
you'll find is more and more important within the morass of networked
data, where the few
valuable nuggets are submerged by an incredible amount of junk.
Internal (Intrinsic) clues to the 'credibility' of the target
Author's own qualifications
Of paramount importance, yet watch it! You must not just believe what HE HIMSELF says!
Stalking and combing techniques may come handy to check his
Watch it! Many studies LOOK professional, but scanning the references you'll
quickly realise they are not even at ordinary universitary level (besides NOT ALL
universities are equal: is the university where the study / author 'dwells' an university,
or study institute,
with a certified track of competence or is it just "some kind" of university?
You would not
believe how many universities nowadays are fighting hard to keep at levels
any good secundary school had thirty years ago :-(
Style patterns and Language correctness Careful, very careful with this on a multilinguistical web! Like the previous
one, a difficult clue to use. Take for instance
a good messageboard, with significative multinational partecipation, like
one of these. You may (may underlined)
find snippets of incredibly valuable knowledge in pidgin english, full of
mistakes, dotted with
unscientific references. On the other hand, often enough,
you'll find preposterous university-thesis
with lengthy wording, perfectly correct, and exact scientific layout, yet
with no knowledge value whatsoever.
Quite important. You may reasonably expect better value and competence on a site that is
continuously updated vis-a-vis a site that 'does not move' for months (or years). The
usual caveats apply, though: updates must regard CONTENT, not just frills.
Timestamps (chronological indications)
VERY important: veritatem dies aperit (Time discovers truth).
You'll quickly notice how many sites DO NOT DATE their documents, and even
go to great lenghts in order to avoid the possibility that you could infer
their real publication date. The
simple reason for this is that those sites FEAR 'LOOKING' OBSOLETE.
Our perception (skewed by
propaganda and advertisement) is that 'new' research
means 'better' research. This is absolutely not true (the contrary is quite often the
case, see [Sielaff's] teachings)
but this untruth, being perceived as such, is worshipped on lazy sites
('Readers should not know that
this stuff is rather old').
Bias Most 'apparently' academic sites avoid polemic and
prefer an "equilibrated essay style". This is
often due to the 'mimicking' of the scientific style as perceived by the broad
public. In the reality valuable scientific research does not
avoid confrontation, never. To establish a new theory
(not just trivial addendums) means
mostly to DESTROY established (and 'cherished' and 'bread-gaining') theories. This is
even more true on the Web, where you can "publish and be damned" with an ease that
would be impossible in the academic world.
Yet bias can indeed constitute a clue in 'negative' sense as well: excessive obsession
theories (taking them very seriously) is a classical sign of boiled air
External (also non-intrinsic) clues to the 'credibility' of the target site
HTML Style of the site (VERY careful with this!)
art and kind of sites linking to the target site
art and kind of sites your target site
'Librarian' clues to the 'credibility' of the target site
is there any feedback provision?
how is the indexing of the target site performed?
is the target site easy to use and navigate?
does its content addresses 'the real needs' of his users?
Some of these rules are (at least in part)
correct, yet some of them are QUITE obsolete. This being the case -for
instance- for rule 1.3 (style and language). this kind of
approach does not have much
sense on a more and more
pidging-english and multi-cultural web, where a very valuable
apport can be expressed
with a very poor english. You should not care about the
spelling, or grammar of the written work that
you evaluate except when the wording is so unclear that the
results unclear, imprecise, or ambiguous. I am personally
unconvinced by rule
1.6: the apparent lack of bias and political correctness
of the textes being often just a simple rethorical cover for very
partisan positions ("look!
It looks really like I'm really neutral and unbiased, my dear
friend and future client"). Note that the viewpoint of the site may be
explicit -say included in a
scope statement- or you may be able to confirm your suspicions
only analyzing the point of view that 'transpares' through
the contents of the site, which you should by all means always do,
scepticism is a very sound attitude when perusing the web.
Rule 2.1, on the other hand, is patently absurd only if it tries
identify a 'positive' style for evaluation purposes, but I
found that it is after all a useful rule for judging immediately
the 'negativeness' of the sites you found, as I will
But the rules above need to be implemented:
A common problem is due to the many 'hidden' forms and techniques to smuggle
advertising inside the pages you will find,
seem definitely more difficult to differentiate - on the net than
on the streets or in
mass media & television channels. The various techniques used for
[reality cracking] purposes can come
- I believe - very handy in such contexts.
Note that very often the commercial label of a given font is
blurred: a growing number of sites may have started out
because some people felt that the content belonged on the web, but
now many of these sites wish to take advantage of their 'position'. I doubt that
you'll be able to find nowadays
many "important" sites without any advertisement whatsoever (bar mine :-)
Your own maturity is required: the important thing to pay attention
to is whether a
site has valuable content or not and whether its presentation or
biases make any difference in terms of what you need to get out of
Yet another problem is due to the incredible "individual" possibilities
and to the half-anonymous character of the media:
it is definitely possible on the web for individuals to
share working papers or
information they have been working on very seriously, but
as you can imagine, there is no guarantee that this work
has undergone a rigorous review
process, in fact it hasn't probably
been through the
peer review processes that are intrinsic to scholarship (even if those
same processes are very often far from being perfect, as anyone that
works in an university knows). This opens the doors to all sort of scams: you
could easily (with a little phantasy and a little research) put up a beautiful
'University of Cordonshire' and start distributing interesting
courses for distance learning MBAs that would have more or less the same chances of success
that all the other scam schemes that already exist (have a look at the last dozen
pages of each issue of 'The Economist' if you want to see a nice mix of pseudo
and bogus "universities" offering MBAs aplenty and targeting ignorant CEOs that need
some kind of university
papers on-the-fly :-)
Anyway the fact that the work you must evaluate was not
examinated and did not went through 'the trade
publishing industry' has
some disadvantages and
not only advantages :-)
Note moreover that the 'free' characteristics of the Web
stimulates the experimentation, by all types of publishers, of various forms of
with -say- a printed magazine and the supplementary publishing of
"some" information on the Internet, mostly holding back
some important stuff in order to sell the print publication.
Since in most cases there is somewhere a 'correspondent'
database on-line (which requires payment to be accessed), power seekers will
probably - ahem - have to learn to
find their way to the 'unrestricted' publications as well. It is nevertheless
important to realize that most of the results you will get from a search may be
(for profit reasons) in their 'crippled' form, and you will have to 'cut deeper'
to get the meat, should you fancy to do it.
So, once more, I will not deny that some of the rules listed above are (at
least in part) correct, but
I would like to add some rules of mine, that I found VERY valuable
in order to quickly
evaluate results in our more and more commercially infested
Negative evaluation rules
Fravia's "negative" evaluation rules are in fieri and such will remain until I find some time
to update them...
"Slides" style crap
More and more sites have adopted a zombizied "slide-presentation"
model. You have to click
a dozen times to scrap together the (meager) content that could
have been presented - and
more easily printed - on a
single unique page. The point is exactly that: minimum data, maximum spamming.
Such crap-sites are designed,
built and 'ameliorated' on the "visual"
assumption that zombies
will never print any info anyway, being too busy 'butterflying' from
one site to another without construct whatsoever. This slide-sliding is -obviously- not only limited to the
net. But on the web nowadays
many sites present scarce info in a "crumble" way that
bores searchers to death. See also point 7 (Powerpoint) below.
Publishing - advertising frenzy
statistical tricks: tables
Every now and then you can find some tables on the web. There are some rules that
apply to ALL tables, no matter what are they specifically related to.
Have a good look at them: tables are
very easy to reverse. As a first rule: the more 'grid lines' and -even more important- the
more 'white space'
you see and find in those tables the harder it will be to compare data...
(see on these same lines also 'the slide
problem' described above). This dictates a very handy rule: lotta unused grids & space... hey! Throw that
Some useful additional 'parameters':
Are the LABELS (of those tables) clear?
Are the LABELS (of those tables) complete?
Are those tables ordered in columns? (a sequence of numbers is easier to follow down a
than across a row, duh)
And, this being the case... are the most important columns EVIDENT or are they buried in the middle of the tables?
statistical tricks: charts & graphs
When faced with charts, always check the vertical scale to make sure that it has
not been expanded to make a small change look large or worse, inverted, to make a fall look like a rise.
Remember that SCALES are extremely important when analyzing graphs:
scales that do not start form zero will exagerate modest movements
logarithmic scales will flatten out every increase.
The moment you see a scale that do not start from zero or is logarithmic, you have VERY GOOD REASONS
to suspect why
they did present it so :-)
See also point 1 about the "crumbles" of information on some sites.
Powerpoint is THE distinctive tract of idiocy. If someone uses powerpoint, you can be SURE it is not worth remaining on
the side (or in the room): go away IMMEDIATELY. Any PowerPoint presentation elevates format over content. It's commercialism
stupidity at its worst, turning everything into a sales pitch. The anti-knowledge tool
par excellence, loved by idiots all over the world. With powerpoint even the
"ideas" hidden inside the bambinesque noise, are simplified to the point that they become
redundant and unclear.
Well, the two first essays should give the reader a strong enough WARNING
about possible manipolations trough the web. The first one is a 'slight fake', the
second one is totally bogus (but only because it was written in that way, of course
all sorts of more subtle alterations would be possible.
Notice also, in the second essay, how the very LINK to the bogus pages
can be effectively obscured.
"Alterating historical data on and through the Web"
~ Ideated: 2 December (of course) 2001 :-)
~ Thought as an evaluation lore and as a present for GoW working on Xmas
~ Maybe too easy :-(
Learning to transform questions into effective queries by fravia+, february 2003
part of the [Essays],
of the [evaluation] section
and of my London workshop preparation.
7 questions: quis, quid, ubi, quibus
auxiliis, cur, quomodo, quando?
Only part of this paper is about evaluation.
A discussion about the utility of the seekers' library (January 2004): "The major problem I see with these papers is that they
are made to sound pompous in order to impress the other empty 'academic' heads. For a
person that just wants to make the damn thing work they are tedious to read, hard to quickly
evaluate and can probably be replaced with an hour or so serious thinking on the problem. Now,
with the more avant-garde problems they may be the only source of reliable knowledge, which sadly
means that one has to swallow the tons of crap to find the important bits in there, but hey - nobody
said it should be easy :)"
Searching the Internet For Historical Information by Stevieo Part of the searching essays & evaluating results
sections. ~ July 2004
"Stevieo: Historical information can be hard to find on the internet.
This paper starts by listing a few of these difficulties. The rest is a live experiment where I
try to figure out how to do historical research using the internet. I hope to provide some tips
to help others do research on the internet. I also hope anyone reading this will offer their ideas and solutions"
Searching among Internet Blogs for Information by Fravia+ Part of the searching essays & evaluating results
sections. ~ July 2004 "By allowing anyone to get his hands on the means of production - to write,
produce and publish his own content without needing an editor or publisher, blogs may threaten the traditional
media's hold over the spread of information and ideas.
Unfortunately enough, many bloggers are just bitchy individuals with some axes to grind, unable of rising above their
immediate concerns and personal prejudices and not capable of weighing up the facts. Moreover, alas, there are many more
bigot right-oriented bloggers as one would deem possible
in a medium, like the Internet where even idiots can find real (and professional) information pretty easily."
Powerbrowsing! by +Mala What every seeker should know PowerBrowsing means browsing the Web seeing only
what you chose to see. Even if this might seem an easy thing to do, it is not
and it will become harder and harder in the future... unless, of course, you